Reports: Handing Big Data Sets

Mohammad_Ahmad_Zulfi · July 1, 2020, 1:53pm

Hi,
In real scenarios we have large data sets where as ERPNext frontend is capable of handling limited data sets as they mentioned in the document Scaling ERPNext. Has anyone overcome this problem. Please share your ideas and possible solutions. I am willing to contribute towards a standard solution to merge it with the core.

While convincing customers that it is an alternative to SAP and yet we have this issue is a downside. I think its fixable or at least we can have a minimum viable solution for it.

According to my findings. The delay in receiving the data by the server is around 10 seconds for a data set of 450,000 rows. Below is the code from qurey_report.js where the frontend timeouts rendering big data sets.

I temporary disabled report rendering to see the reponse.
Machine Specs:
4 Core
16GB Ram
SSD

Regards,
Muhammad AHmad

large-data-set-report

github.com

frappe/frappe/blob/63232f4867b861f98699b4a54b2e2c6e383ef377/frappe/public/js/frappe/views/reports/query_report.js#L659


      
          				showTotalRow: this.raw_data.add_total_row,
          				direction: frappe.utils.is_rtl() ? 'rtl' : 'ltr',
          				hooks: {
          					columnTotal: frappe.utils.report_column_total
          				}
          			};
          
          
			if (this.report_settings.get_datatable_options) {
          				datatable_options = this.report_settings.get_datatable_options(datatable_options);
          			}
          			this.datatable = new DataTable(this.$report[0], datatable_options);
          		}
          
          
		if (typeof this.report_settings.initial_depth == "number") {
          			this.datatable.rowmanager.setTreeDepth(this.report_settings.initial_depth);
          		}
          		if (this.report_settings.after_datatable_render) {
          			this.report_settings.after_datatable_render(this.datatable);
          		}
          		this.$report.show();
          	}