Due to select for update statement , it takes time to insert 20k+ rows in frappe ryt??
so has anyone found out a way around?
1 Like
Hi @AaronShenny
Can you let us know from where are you running the update statement?
1 Like
Python script mostly.
Or api.
1 Like
Why not using data import tool? It is specifically designed for long queue and insert data using background jobs.
1 Like
Yes, you’re right, the SELECT FOR UPDATE locking in Frappe can become a real bottleneck when inserting large batches like 20k+ records one by one.
A few approaches that work well:
1. Use frappe.db.bulk_insert() (Frappe v14+)
This bypasses the ORM and uses a raw SQL INSERT for multiple rows at once, much faster for bulk data.
frappe.db.bulk_insert('DocType Name', fields=['field1', 'field2'], values=rows)
2. Direct SQL with frappe.db.sql()
If bulk_insert isn’t available or you need more control:
frappe.db.sql('INSERT INTO `tabYourDocType` (field1, field2) VALUES (%s, %s)', values=rows, as_dict=False)
Use commit=True at the end or call frappe.db.commit() after the batch.
3. Chunk your inserts
Even with bulk methods, inserting 20k rows in one transaction can strain memory. Break into chunks of 500-1000 rows.
4. Disable real-time events during import
If you’re using doc.insert() in a loop, frappe.flags.in_import = True can skip some hooks and speed things up.
What’s your use case, importing from CSV, syncing from another system, or something else? That’ll help narrow down the best approach.
1 Like
If i use that, other person using the app in another device, tries to insert, name could get duplicate ryt.. but name is unique so errors would come ryt
Mainly importing from csv or excel