What are the steps required to upgrade from v10.x.x (production) to v11?

[bkm]

@kolate_sambhaji

Instead of new server you can also install fresh erpnext on Test Server or local machine using multi bench setup

https://github.com/frappe/bench/wiki/Multi-Bench-Setup

Once your second bench instance setup done, you can simply copy old bench site folder to new bench site folder and run bench migrate command.

Thanks @kolate_sambhaji , but this looks a bit more complicated than your first idea, and I really need to put my ongoing support behind the idea of @Fred1 to try to shine a light on the fact that bench update needs to be fixed so it can handle the process like it used to in previous versions. Your first idea works with very good predictability so I flagged it as the answer for now.

There really needs to be a simple way to “upgrade in place” without having to generate a whole new instance in order to get the job done. In previous versions we had that with the bench update command.

BKM

[bkm]

@flexy2ky

As usual, @bkm to the rescue! But you can skip steps 6 and 7 if you have a single tenant and you’re restoring to a clean server. Using the easy install script, if you add the --site [your.site.name] tag and ensure you use the url for the live server, you don’t need to delete the default site and create a new one, as the site created during the install process will be exactly the one you need and all you need do after the initial setup wizard is to restore your backup and bench migrate.

Thanks @flexy2ky , I kind of suspected this to be true, but just wanted to prove @kolate_sambhaji instructions first to make sure we had a working path to the migration. Since I have some time today, I will try to strip down the server and try it again WITHOUT the second tenant site on the new server. I will post back how it works later this afternoon.

BKM

[flexy2ky]

@bkm While you’re planning to migrate, be aware of a nascent issue with V11, specifically ERPNext: v11.1.42 (master) and Frappe Framework: v11.1.37 (master). As of this version, the Dropbox access token field in Dropbox Settings doctype is gone. And the ability to customize the Dropbox Settings doctype has been disabled. So you cannot re-authenticate dropbox for backup.

So if you rely on Dropbox for backups then I suggest you figure a way out before you move your instances to V11.

[Fred1]

@kolate_sambhaji your method of backup in Dropbox is commonly used by all.Since the Dropbox setting is no longer available in the current versions of Erpnext,any way out?.Any suggestion from others are highly welcomed

[bkm]

@flexy2ky

So if you rely on Dropbox for backups then I suggest you figure a way out before you move your instances to V11.

Fortunately for me, I have always had a huge distrust issue with paid online backup space like Dropbox, Google Drive, Sync, etc. So, I never bother trying to adopt them and have actively discouraged my clients from doing so.

Instead I have always relied on systems that I can control completely myself. Protecting yourself is ALWAYS your responsibility and if you entrust it to some other party, then you have no recourse for when the 3rd party does not save you. Would you depend on a police officer to save you from an attacker trying to hurt you in your home? After all the police department is responsible for such things, and besides they are only 15 minutes away when you need them to save your life… right? NEVER place your full trust in the service of a 3rd party. Always have a plan of your own that YOU can depend on!!

Personally and for all of my clients, I use The Poor Man’s Backup System (v2) in addition to anything my clients “think” is a safe backup plan. I spread the copies of the backups across several servers in several locations around the country to minimize the potential for failure.

So, anyway… I have no worries about the Dropbox issues, but I will start probing for anything else that might be problematic. Thanks for the tip.

BKM

[flexy2ky]

@bkm

Personally and for all of my clients, I use The Poor Man’s Backup System (v2) in addition to anything my clients “think” is a safe backup plan. I spread the copies of the backups across several servers in several locations around the country to minimize the potential for failure.

Unfortunately i do not have paying clients. all my deployments so far have been for friends and acquaintances and setting up an extra server just for backups would come at a cost they may not be prepared to handle at this time While i understand the need to control your destiny as far as critical data is concerned, sometimes, cost considerations outweigh every other reason.

Dropbox has worked for me since i started using it and it is quite sad that development progression has seen it being disabled (inadvertently i hope). More so, it was a simple method of backup and it would help if the functionality can be restored.

[bkm]

Ok, I can confirm that this shorter version of the remedy also works if you are updating a server that only has one site. I originally suspected this might work, but needed to prove it to myself first.

@flexy2ky

As usual, @bkm to the rescue! But you can skip steps 6 and 7 if you have a single tenant and you’re restoring to a clean server. Using the easy install script, if you add the --site [your.site.name] tag and ensure you use the url for the live server, you don’t need to delete the default site and create a new one, as the site created during the install process will be exactly the one you need and all you need do after the initial setup wizard is to restore your backup and bench migrate.

In some cases you may need to run bench update --patch after migrating to ensure that all the tables are migrated and patched properly (i have encountered a case where migrating didn’t reconcile all doctype changes in V11 and running bench update --patch eventually fixed it).

So, I will put the shortened procedure here in case you do NOT need to update a multi-tenant server.

1.) Create a fresh and updated server (I used Ubuntu 18.04 LTS)

2.) Use the instructions for the “Easy Install” to install default latest v11

3.) Login to the new v11 as Administrator and complete the setup wizard to create a dummy site.

4.) After setup wizard is done, logout and close the browser.

5.) Open a ssh session to the new server using your default user account

6.) Copy the latest backup of your live site to your new server (it will be restored to the update site later)

7.) Use scp to move all of the files in the /public/files and /private/files directories from your live site to the same locations on the new server in the update site.

8.) Use the following command to restore the live backup to your update site:

sudo bench --force --site [your.site.name] restore /path/to/your/backup.sql

9.) In the ssh session for your update site, run the following commands:

bench --site [your.site.name] migrate
cd sites/[your.site.name]
nano site_config.json  (use the "encryption_key" line from live server to replace this one)
cd ~/frappe-bench   
bench setup nginx
sudo service nginx restart

10.) At this point you should open a fresh browser and log into the update site to test your migration to v11.

Again… This has been tested on Ubuntu 18.04 LTS and you will have to verify if it works on your distribution yourself.

Also note that I did NOT yet use the ‘bench update --patch’ command. The intent is to verify that everything works first. This can always be run if you find issues.

And as always… Your mileage may vary! Happy Migrating!

BKM

[bkm]

@flexy2ky

Unfortunately i do not have paying clients. all my deployments so far have been for friends and acquaintances and setting up an extra server just for backups would come at a cost they may not be prepared to handle at this time

If you have setup ERPNext for “friends and acquaintances” then those people are using the system to make money at some level. So even if it is just a cheap OpenVZ type pf server with very little memory, at least it would be a safe spot to scp backup file into for safe storage. One server could support all of your friends sites because it would only be for receiving and cataloging a small number of backup files.

You should tell them that if they find any value at all in the ERPNext server they need to give you $100 a year to pay for a safe backup server or else if they have a failure, you will not be able to help them.

BKM

[bkm]

Still waiting to see if anyone here can come up with a way to update/upgrade from v10 to v11 in place without having to create a second server. I am willing to try any suggestions.

Anyone?

BKM

[smino]

Have you tried dropping your site(s) , updating frappe/ERPNext , then reinstalling / restoring them?

I haven’t looked , are there requirements for 10 and 11 , versions of things like Redis, Nginx, NodeJs, Yarn , python libraries etc that can be compared to see if some preliminary environment packages should be upgraded prior to upgrading frappe/ERPNext?

[bkm]

@smino

Have you tried dropping your site(s) , updating frappe/ERPNext , then reinstalling / restoring them?

Hmm… I didn’t drop the sites first, but I did try updating and there was a flurry of activity that seemed to be doing something with the doc types, but then after rebooting the host the system was still at the same level of v10 that it was before the update. SO, apparently no changes actually happened.

Tried this 3 times to make sure I wasn’t missing something. Every time the same result. It didn’t matter if I rebooted the host or not. Cleared the browser cache and logged in to see if the version changed and it had not.

So, I am not sure how dropping the sites would make a difference, but if I can get some time later this week I will try it just to make sure it has been attempted.

BKM

[jodeq]

I have done this migration last month and all was good.
Here how I have done :

• bench update on current v10 bench
• create a new bench in v11 with bench init --frappe-branch master --python python3 my-bench-v11
• in my-bench-v11 I get erpnext with bench get-app --branch master erpnext
• next I uncompress my last backup from v10 bench gunzip ....../bench-v10/sites/my-site/private/backup/****.sql.gz
• create new site in my v11 bench with this backup : bench new-site --db_name mysite_v11 --source_sql full_path_to_backup_unzip.sql mysite
• Manually copy encryption_key from v10 site_config.json to v11 site_config.json to keep your communications
• Manually copy all files from v10 site private files and public files to v11 site
• setup production (some patch script needed a redis running) sudo bench setup production frappe
• and it’s all ready to run bench migrate to run all patch scripts

now you can disable production from your v10 bench sudo bench disable-production and a let’s encrypt setup in your v11 bench

no need to drop site from v10 if you setup a new db_name
no reboot required
no second server necessary, only a new bench folder

[noetico]

This sound super-interesting! Thanks…

[noetico]

To my surprise, I had a simple bench update run from 9 to 11, although it failed towards the end with an interfaceError thing from pymysql during bench migrate (which by the way is annoying)… but it headed to v11. The only thing that makes it different is, this had the bench-repo folder… I wonder if that made a difference.

[bkm]

@jodeq

no need to drop site from v10 if you setup a new db_name
no reboot required
no second server necessary, only a new bench folder

Interesting, but still not quite what I was looking for. In your case everything is created in a “my-bench-v11” folder and the original bench folder is eliminated.

I have many other processes that interact with the bench from outside of ERPNext and it would mean changing all of those processes to understand the new bench name. Further complicating that would be that my outside processes interact with many ERPNext servers and they all have the standard bench directory structure and bench name. So to change the name of the bench would name having to setup several duplicate outside processes to handle the work and merge the resulting data. That would be a significant development effort!!

When I said “update in place” I really meant to update in place without altering the directory structure if at all possible (much like the ‘bench update’ command used to do).

Thanks for the innovative suggestion as I am sure it will be of benefit to some additional users, but it is not quite right for my purposes.

BKM

[jodeq]

I understand and i have done this because to verify migration.
If migrate failed then I don’t lose anything from my current site.

I didn’t try the in-place update but here what can be done (test must be done before)
from your bench repo
bench switch-to-branch master
bench migrate

and why not a bench migrate-env python3 to get ready for 2020.

[bkm]

@jodeq

and why not a bench migrate-env python3 to get ready for 2020.

I agree. Although I will probably try that after the successful migration to v11. I prefer to not add more troubling variables to my efforts and tackle them one at a time.

BKM

[flexy2ky]

@bkm

From root (not the user account), run the install.py like this:

python install.py --production --user [your user] --frappe-branch v10.x.x --erpnext-branch v10.x.x

@bkm as of V12 released today, Frappe has killed the ability to install earlier versions. i just tried to install V10 using the branch tag above and even though the installation completed successfully i got presented with V12. Worse still, it’s impossible to restore a V10 database to V12 now. I attempted to restore a v10 backup and my instance essentially died after running bench migrate (the full trace back is damn too long to even try to post). I ended up killing the instance and reinstalling.

[jkimathi]

@flexy2ky - I also tried v10 using the branch tag on V12 released 4 days ago - but was presented with v12. I urgently need to do a database restore from v10 to v12. Anyone please help especially forks from Frappe technologies might have an idea.

[flexy2ky]

As of right now I don’t know how to upgrade from v10 to v12. restoring v10 backup to v12 also fails.