Fastest way to migrate large Old ERP Data

ERPNext
, ,
1

Hello Beautiful People,

We have planned to migrate our existing system to ERPNext. But we’ve noticed that importing large amount of data takes a long time, importing and submitting 80,000 Delivery notes took 3 days (Linked with Sales Order and Sales Invoice).

I am wondering if there is a better way to import large data. We are using AWS EC2 (c5n.xlarge vCPU 4, RAM 10.5 GiB), suggestions for increasing server capacity and configurations also encouraged.

Can we use ERPNext Data Migration Tool? I have tried to connect with another MariaDB Server, no luck. I don’t know how it works.

Thanks in advance for your help

2

How is your EC2 storage provisioned?

3

Hi
You can use the macros as per ERPNext excel template. Once your file is ready import the record with submit option then you don’t need to submit the records again.

4

Try below (assuming there is no core code change)

  1. Get backup of your old erp site.
  2. Create a new site in newer version of erp
  3. Restore old erp site db back into new site
  4. Run bench update requirements, bench migrate, bench build
5

~3 seconds per Delivery Note sounds reasonable, don’t you think? Keep in mind that it’s not just storing the data but also validating it. This takes some time, but it makes sure that you won’t have any garbage in your new system that would cause problems down the line.

6

I also would advise this. Takes 30min to have everything up and running Esp for large data files.

7

If old system is not ERPNext. Just an alternative approach if it fits your use case,

  • Do not migrate 1:1 data.
  • If previous data needs to exist for archival of records, create a Custom DocType and migrate un-validated data for archival and searching from old system into custom doctype.
  • Assuming data was validated in your previous system anyway, custom doctype can just reference it.
  • Make opening journal entries and opening stock entries and try to match the state of balances of accounts and warehouses.
  • Start making new entries into ERPNext.
  • For referencing old entries use the custom created doctype.
  • May not work “as is” if serials, batch expiry, or stock functionality is used.
  • Think of workaround entries that help in reflect the state of reports for company’s needs in ERPNext.
3 Likes
8

Thanks @rmeyer!
It is better for us to be able to shorten our migration time, because you know that a company is like a moving ship. We need to change its surface.

9

Thanks for your time, @revant_one!
We are moving from a fairly straightforward system and well-structured database. During the test, ERPNext was able to import and validate 100% of the requested data. Just wondering if there is any other way to speed up the process.

10

If you know data is absolutely correct? then Just insert it as SQL query without any validations.
Challenge here is to generate the query which we get after validations are done and data is in ERPNext. Hack up some scripts and with trial and error it can be achieved.