File size limits when importing large csv data files via bench using its import-csv command

Frappe Framework
1

Hi guys,

Just wanted to consult on the upper cap with the number of records per a csv file to be imported with bench and its import-csv command (https://frappe.github.io/frappe/user/en/guides/data/import-large-csv-file).

The documentation for Data Import Tool states the maximal number of records per file is 5000 (handling batch import files of such size will require some additional system configuration fine-tuning if running ERP Next in Production mode though - see Data Import Tool hangs on ERP Next instance running in Production Mode - #6 by gvyshnya).

So we would like to know what gain we get if using bench import-csv method instead of Data Import Tool. What is the largest number of records still feasible to process with bench import-csv per a file?

Thank you.

2

Not long time ago I’ve read about a .csv tool called xsv. It is supposed to be pretty efficient and fast operating .csv files, there are several rough benchmarks of it.
Is there any applicable use for xsv in this case or it’s out of scope?

3

@strixaluco: thank you for the suggestion on xsv.

In this case specifically, it could be out of scope as I looked for specific limitations of tools available for ERP Next data import.

4

Yes, I meant what if xsv was used in the ERPNext Frappe? Is it feasible?

5

@strixaluco http://discuss.frappe.io/users/strixaluco: I am not aware
of xsv being applicable to Frappe/ERP Next …