Hi @kkulloters ,
I’m not sure if I understood your question.
My problem is to do a restore without losing the existing data on my target environment.
Hi @TNHD_Pro,
If I understand your requirements correctly:
- You have data in both Development and Production.
- You want to take any additional data that exists in Development, and -append- that data to what is already in Production.
- Without losing or overwriting your data in Production. Just adding to what is already there.
Is this a correct summary?
bench restore will wipe out all existing data and add new data.
You will have to manually add (via data import) either production or development information. I don’t recall that there is a straight forward update to existing information.
Agree with @Muzzy: the only way to accomplish this is manually.
For a tool to accomplish this automatically, there would be huge complexity and risk. How would it know which records to copy, and which to leave behind? How will it understand merge scenarios?
For example, assume the following:
-
Development has a
Suppliernamed “Acme Anvil Manufacturing, Inc.”. -
Production has a
Suppliernamed “Acme Anvil Mfg, Inc.”. - Is this the same
Supplier, or two different ones? How does the migration tool know the rules? - If they are the same, but certain attributes are different (e.g. Payment Terms, Address), which attribute is the correct one for Production?
There are hundreds of other small challenges: Customizations, DocTypes that are Singles, Versions, User defaults, environment-specific defaults like Keys and Secrets, and more. Also, what if the MySQL table schemas are different between Development and Production?
I’ve never seen any ERP accomplish something like this. Instead, people manually design and develop custom scripts for their particular database + requirements.
Even an automated tool for copying the other direction (Production —> Development) is not always easy. I’ve been asked to build something like this for a client. Great care must be taken to ensure that your Development environment’s database is reconfigured, to behave correctly (e.g. disable automation so the Development environment does not send emails to real customers, employees, APIs, etc.)
So what is the deal with this situation?!
How do you guys manage the development and production environments with ERPnext?
I’m searching for a while about the exact subject, I’m a business user, not a developer. I just consulted some of my close developers that don’t know what ERPnext is, they just working on many other frameworks and they explain to me the approach of having a development and production environment.
They said you should create a development environment in the beginning, and work on it, once that is ready for production, you should convert it to production, and then you will have a copy of that production environment to consider it as a development, then work on it, and once you are ready with your new features and updates, you just migrate the development (changes) to the production, and repeat the cycle.
And in regards to the data, all of these migrations are for the application not for the data.
This is how I understand it.
But it seems there is another approach that ERPnext community is using, which I don’t understand yet!
So, could someone explain the mystery behind that subject?
Okay, that’s great!
How to do it?
I have two containers (srv1 & db1) and they are both active and running, how to get a copy from these containers for development?
Then how to migrate the development containers to the production?
@adelalmajed using custom apps of course . create a custom app . save all your customization or new features into the app . push it to github . and then you can install it on other servers .
check this topic about custom apps .
if you want to copy the database itself . use backup and restore .
Thank you @bahaou for the answer.
Unfortunately, it’s not what I’m looking for, perhaps I didn’t explain it well.
I’m talking about cloning a production environment where I have two exact running instances, and the main idea is to modify some documents and test the workflows before migrating them to the production.
I found the following question which is exactly what I’m looking for, but unfortunately, no one answers his question! 
@adelalmajed I did understood your question and my answer still the same . you have two running instances , one is dev and the otehr is production . you want to do your work in the dev instances , test it and then migrate it to the production server . that’s exactly what custom apps do
Thank you @bahaou
While reading the link you provided, I don’t know why I felt it is a method of creating a separate app! It seems like creating a new module or something needs to be coded, while my need is to implement the business workflows and permissions on the existing ERPnext modules.
However, after reading your confirmation, I felt it was my fault for judging something I didn’t try yet.
Therefore, I will start learning the method you have suggested, and I hope it is easy to learn.
If you have more examples or tutorials for that methods, please share them with us.
Many thanks
@adelalmajed you can indeed add new modules in the app . but it also provide the possibility to save customization and everything you have changed. example you changed some permissions and workflows for sales order . you export them to the app . after installing the app to a new server . all custom changes will appear .
Very nice, that’s exactly what I need.
But what about creating a clone of the production? the link you provide seems to dive into the customization method directly.
From the beginning, I thought there is a bench command that makes everything, for example:
bench --site production_site.local create development_site.local
and after that, a new site will be created alongside the existing one. And when completing the customization of development_site.local, we just run:
bench --site development_site.local migrate-to production_site.local
Isn’t that would be great?
For a moment, I’m going to set aside things like containers.
Fundamentally, every ERPNext environment consists of a few things:
- A database server, with 1 SQL database per Site.
- Python and JS code, stored (mostly) in plain text files. These files are located within
'../frappe-bench/apps/<app_name>/' - Some configuration data (e.g.
'common_site_config.json', ‘site_config.json’) - Some public and private files (usually attachments like .pdf, .doc, .xlsx, .txt)
- The prerequisite software (MariaDB, Python, Node.js, Redis, Nginx, wkhtmltopdf, and more)
How to copy a Database from one environment to another:
- Create a database backup file on the source device. (e.g. using the
'mysqldump'tool) - Copy that backup file to your target device.
- Restore the backup file on your target device’s database.
- Run the command
'bench migrate'to synchronize the newly-restored database with your target system’s schema requirements.
NOTE: The SQL database contains -mostly- business data. But there are important exceptions:
- It might contain configurations that are specific to an environment (e.g. 3rd Party integration API keys)
- It might have rows that contain code scripts (Python, JavaScript), that might only be compatible with certain environments.
Because of the above, you may require extra steps after restoring your database.
For example, one of my clients has a custom Mailchimp integration. There are different DocType values (configurations) for Production versus non-Production. I don’t want a non-Production environment to accidentally transmit emails to customers or suppliers! 
So after restoring a database (but before starting ERPNext) it may be important to review certain configurations or scripts, and alter them. Frappe has some features that help with this (Fixtures). You can also write your own scripts to help automate this, as part of your database deployment process.
How to copy Code from one environment to another:
ERPNext “code” can exist in 2 places:
- In a plain-text file, on disk.
- Inside a SQL table, as mentioned in the section above.
To copy the former, the most common method is by using 'git' commands. Every ERPNext ‘App’ has its own git repository. When we make code changes in an environment, we ‘commit’ and ‘push’ those changes to GitHub/GitLab. On another device, we can ‘pull’ those changes downward.
The Frappe and ERPNext “code” is really nothing more than plain-text Python and JavaScript. While you’d have to be extremely careful and diligent, an experienced administrator “could” just copy from one environment to another. Or use a tool like ‘rsync’ to help.
There are very important considerations. Each App has it’s own 3rd party package dependencies (Python requires PyPi packages, JavaScript requires NPM packages). These must be correct per environment.
However, most people use a combination of ‘git’ and ‘Bench’ commands to simplify deployment.
How to copy Configuration or Files from one environment to another:
These are ordinary files. So you can just copy and paste. Use a tool like ‘rsync’ to handle the copying. Whatever you want.
If you don’t actually need the latest files and attachments in an environment, then skip this step.
How I Do It
I write a bunch of Bash, Python, and SQL scripts to help me automate this. For example:
- Pulling the latest code into Production.
- Moving Production data into my Staging environment.
- Clean up cache files
- Rebuild certain tables from scratch.
Whether you’re on bare metal, a cloud VPS, or inside a Docker container, the fundamentals are all the same: text files and SQL data. There aren’t (yet) any compiled languages in the workflow.
@adelalmajed if you are talking about migrating existed customizations you can actually do that by creating the app then , exporting all customizations to this app using (bench export-fixtures --site sitename) , you have to write some code in hooks , and the changes must be throught customizations not editing doctypes , new doctypes should be changed to the new app module. just keep learning about apps and you will find out that all you need could be done .
Thank you @brian_pond , your reply seems full of details that I will need later, and you have provided me with a hint to start learning about a tool called ‘rsync’ which seems will help me somehow. Therefore I bookmarked this page.
Thank you @bahaou for your help.
I don’t have any existing customization nor a real active instance running ERPnext yet.
I’m just planning and preparing to start implementing ERPnext for a start-up business and trying to cover all expected issues that I might get into before officially working with ERPnext. And I feel that I’m almost there.
There is still something I don’t understand, and I can’t ask about anything before learning and applying the method you suggest.
Therefore, I can’t say anything now, but many thanks for your help.