Data import times out after about 1500 records (V13.beta.6)

Frappe Framework
1

I am using the new Data Import page of Version 13 beta 6

I am obliged to break up an 8100 row file into 8 pieces for the importation to work.

The importation is of Payment Entries from a legacy database with dirty data. When one of those partial files completes I see roughly 2% of the records throw the error:

Row #1: Allocated Amount cannot be greater than outstanding amount.

However if I try to import pieces larger than 1500 rows, the only error information thrown is…

Time out error

… with no further explanation.

Is this a known problem?

Is there a better workaround than manually breaking up files?

2

what is your http_timeout setting?

3

This timeout happens even if I close the page of the specific Data Import and return to the Data Import List (or any other page).

Ultimately, the process fails and the status of the importation is left as “Not started”, with no other information.

Because of this, I assumed that the timeout is internal to the server, and that communications delays have nothing to do with it.

Am I mistaken?

4

I have used Bench CLI to import large amount of data. Our 600,000+ journals took 24+ hours to import. I have also used tmux to run the process without keeping the session on.

You can also import using:
bench --site sitename data-import --doctype Item --file /path/to/file.csv

7 Likes
5

It seems to be working.

Q: If there were errors, how would it tell me?

6

Hi,
The system primarily checks the file row by row before beginning the process. During the process, if errors found it will create a file with failed row and will save at frappe-bench/sites

1 Like
7

Really Cool!

Thank you for the great tip!!!

1 Like
8

For future creatures looking into this …

… the command, as shown by @hiimkhaled, leaves data in draft mode but, you can use the switch …

  --submit-after-import   Submit document after importing it

All options:

erpdev@derpl:~/frappe-bench-DENH$ bench data-import --help
Usage: bench  data-import [OPTIONS]

  Import documents in bulk from CSV or XLSX using data import

Options:
  --file PATH             Path to import file (.csv, .xlsx)  [required]
  --doctype TEXT          [required]
  --type [Insert|Update]  Insert New Records or Update Existing Records
  --submit-after-import   Submit document after importing it
  --mute-emails           Mute emails during import
  --help                  Show this message and exit.
erpdev@derpl:~/frappe-bench-DENH$
2 Likes