Does ERPNext support high volume of data (more than 2 millions transactions in a year)?

ERPNext
#1

Dear ERPNext team and users,

We are planning to implement ERPNext for a retail business having high volume of transactions. Below is details:

Branches : 15
Application Users : 50-100

Daily Transactions :
a. Sales and Purchase : 3000
b. Account : 5000

Monthly : 200K

Yearly : 2.5 millions

Total expected transactions in next 5 years : 16.25 millions (with 30% increase)

From our initial analysis, we came to know that MySql can handle millions of transactions.

Our queries regarding ERPNext are:

  • Does ERPNext support such a high volume of data processing?
  • Does ERPNext support 50-100 concurrent users and how good ERPNext is in concurrent processing?
  • Would ERPNext be able to process and generate various transaction reports?
  • Is it possible to split data year-wise?
  • Has anyone implemented ERPNext for such a high volume business? What challenges did you face and what are changes did you make?
  • If yes then,
    a. what infrastructure would be required to run application seamlessly?
    b. what are the points we have to keep in mind while doing implementation?

We would highly appreciate your views and input on this.

1 Like
#2

This is a very interesting question. We have a use case too in which we are estimating similar numbers for a different custom DocType which is a price database for electronic components. While your numbers seem high to you, this is really not high volume of data, that would be terabytes in a MariaDB instance.

Does ERPNext support such a high volume of data processing?

One problem that we have seen throughout the system is indexes missing which causes some DocTypes to load very slowly. These are because some of these DocTypes just haven’t been designed for big data. With Sales and Purchase Orders as well as the transactions this should not be the case. In any way these are easily fixable as the system scales. I hope the foundation or Frappe will dedicate some resource into optimizing the database design.

Having said that, these are really pretty minor and easily fixable. MariaDB can absolutely handle data like this and the framework can either handle it or can be adjusted if needed.

Does ERPNext support 50-100 concurrent users and how good ERPNext is in concurrent processing?

50-100 users isn’t much. The system handles that easily. Tasks are properly queued in the system.

Would ERPNext be able to process and generate various transaction reports?

There are plenty of reports already available out of the box, you can WYSIWYG create custom reports yourself or if they are really tricky you can code them in Python.

Is it possible to split data year-wise?

What does that mean?

Has anyone implemented ERPNext for such a high volume business? What challenges did you face and what are changes did you make?

We are on track of implementing it for a similar load and we are not worried.

What infrastructure would be required to run application seamlessly?

Go with the ERPNext Pricing 2022 800 USD / Month medium plan and have Frappe deal with the Database and help you when performance issues arrise.

If you insist on self-hosting, have an average gaming PC like a I7 or Ryzen with Ubuntu Server installed.16GB Ram on a RAID-1 SSD with 128GB for Frappe and consider an internal HDD for backups.

What are the points we have to keep in mind while doing implementation?

Fine-tuning your server and tracking the performance of your MariaDB server will be essential to success here. You are probably best to get professional help with this by having Frappe manage that for you, otherwise you will need to have someone manage that for you since you are moving big data.

Hope this helps!

2 Likes
#3

Dear @dominik, thank you so much for your detailed input on this. We highly appreciate your help.
We have few more queries:

  • would it make any sense if we split App and Database Server?
  • will it make any performance improvement if we do on-premise installation?

We are planning to do testing using upto 10 millions transactions.

Split data year-wise mean, if we can segregate and move data after year closing to other database.

with warm regards,

#4

I agree mostly with Dominik, you will have to keep optimizing as you hit bottlenecks. This is a constant process we do on a regular basis, where we keep optimizing based on what we see on our cloud.

All I would strongly recommend is that if you clear bottlenecks, do keep posting your fixes as pull-requests.

Plug: If you want we can help you to release bottlenecks at the backend as long as you believe in the “contribution first” philosophy! Feel free to drop in a mail at hello@erpnext.com

1 Like
#5

You are welcome @sanjay we are very interested in moving ERPNext towards high availability and eventually even load balancing and high availability if one of the nodes goes down. Frappe is not designed for that but maybe it will be moving towards that.

would it make any sense if we split App and Database Server?

Possibly, but only if they are in a vlan (cloud) or a dedicated lan network (local bare metal) that has high throughput. You will never be as fast going through a network connection as you will be locally. I don’t think that the database entries and the load will be so big that you will need a very specialized server. Typically you would have something like a high CPU server with a SSD for your workers and a high RAM server for your DB server. What I would do is get a big SSD mirrored and have both in one.

I would try to keep the complexity low and have it on the same server.

will it make any performance improvement if we do on-premise installation?

You mainly cut out the internet as a performance factor. We have decided for a local instance because all of our machines are plugged into erpnext and our internet connection is not 100% reliable despite 3way failover. The thing is, if you have a local server you have to also think about the physical safety of the server from fire, power-outage, theft etc.

2x240GB SSD, 8 Core 3.8 GHz, 32 GB Ram could be a great setup for what you are trying to do and is available for 120 USD on vultr or can be configured for under 2000 USD on premise.

Maybe you would be well advised to consider ERPNext.com custom dedicated server and discuss pricing directly with them. Frappe are the experts and they could help you clear bottlenecks if you are their customer.

1 Like
#6

@sanjay in any case please keep me updated on issues you face this is of highest interest to us and I believe to the framework as we are moving from SME to enterprise customers.

#7

Thank you @dominik and @rmehta for your help. Its really very helpful. We will follow up your advice and keep posted about our progress on this.

@rmehta we will contact you and your team soon on this.

1 Like
#8

@sanjay how the usage of the big data set that you have ?.. im also facing the same question now

thanks