Validation Error: Too many writes in one request. Please send smaller requests

Frappe Framework App Development
1

Hi all,

i’m syncing items from csv file using my own logic and code. We’ve about 20k items, including:

  • Item Price
  • Supplier Item detail
  • Barcodes (in a Item Barcode Child table)

After about 6k entries i got the error:

Validation Error: Too many writes in one request. Please send smaller requests

Any hint?

2

break it up into 5k imports. works fine.
Also can delete any empty columns.

3

What u mean break it up?

Reading on other posts it could be needed to send a frappe.db.commit() every 100 - 200 items

4

The best way to import that many is to use the frappe client. Way better than the import within the app.

5

Hi @JoEz,

I had the same error the other day importing customers to the system and this was the solution I came up with:

In [1]: import csv
   ...: from frappe.utils import cstr
   ...:
   ...: with open("/home/frappe/frappe-bench/customers.csv", 'rU') as csvfile:
   ...:     file_content = csv.reader(csvfile, delimiter=str('|'), quotechar=str('|'))
   ...:
   ...:     for idx, row in enumerate(file_content):
   ...:         if cstr(row[9]).strip() == "ACTIVE":
   ...:             doc = frappe.new_doc("Customer")
   ...:
   ...:             doc.customer_name = row[1],
   ...:             doc.tax_id = row[0].strip(),
   ...:             doc.party_group = "Comercial"
   ...:
   ...:             doc.insert()
   ...:
   ...:             if (idx % 5000) == 0:
   ...:                 frappe.db.commit()

Notice that the file was not comma separated, it was pipe | delimited.

Good luck!

1 Like
6

Thx for the hint! frappe.db.commit() is the key …actually i’m more conservative and running it every 500 doc insert :grimacing::grimacing:

7

@yefritavarez answer looks to be the best solution.
In my case 8k of Customers with Addresses, Multiple Contact details and Multiple Visit History. I made one big .CSV with all the Customers, Customer0001 on Row 1, Customer0002 on Row 2 etc etc. and then broke that CSV up into 4 .CSV files each containing 2k of customers. Customer0001 to Customer1999 on .CSV1 etc etc. Each Visit History was exported out of old DB Via SQL and I just inserted Customer0001 etc prior to actual visit data on each row so as to keep them correctly linked.