Hi,
after i discovered the frappe.db.bulk_insert function (below) that helped me A LOT for improving performance inserting tens of thoundsound of recording faster than with the standard DB API. i am now willing to implement a frappe.db.bulk_update function that will bulk update records.
Below the bulk insert function
def bulk_insert(self, doctype, fields, values, ignore_duplicates=False, *, chunk_size=10_000):
"""
Insert multiple records at a time
:param doctype: Doctype name
:param fields: list of fields
:params values: list of list of values
"""
values = list(values)
table = frappe.qb.DocType(doctype)
for start_index in range(0, len(values), chunk_size):
query = frappe.qb.into(table)
if ignore_duplicates:
# Pypika does not have same api for ignoring duplicates
if self.db_type == "mariadb":
query = query.ignore()
elif self.db_type == "postgres":
query = query.on_conflict().do_nothing()
values_to_insert = values[start_index : start_index + chunk_size]
query.columns(fields).insert(*values_to_insert).run()
i would like the signature of the bulk update function to be def bulk_update(doctype, names, fields, values):
i am not an expert using the query builder and in SQL in general so if someone could help please.
General question:
How are you handle large insert and update in general within frappe apps ? using the doc API is farrrr too slow, the frappe.db API too.
Thanks !