Help! Moving docker instalation from one server to another

ERPNext
1

I am attempting to move my docker-compose installation from one server to another.

I stopped the docker containers on the source server.
I exactly reproduced the source server directory on the destination server (using rsync).

I started the docker containers on the destination server.

Problem 1: The containers could not connect to the database.
Solution 1: I went into the database container and GRANTed permission. Can’t see why I had to do that?

Problem 2: When I login, and go to the Desk, I get an error pymysql.err.OperationalError: (1054, "Unknown column 'onboarding_status' in 'field list'")
More Info: I went into the frontend container and searched for that string. It was all over the place. But in the original container, it was not found at all.
I can only assume that building the containers on the new server has built a different version, but not bothered to update the database schema.
I tried deleting the entire contents of the app directory in every container, and restoring them from the containers in the original server. But, every time I restart the containers, they come back.

How do I make this work?
How is the database supposed to be updated to contain the fields expected in the latest code? Is this not automatic?

2

I seem to be talking to myself here!

Had time to take a further look at it. It seems that rebuilding the container always loads the latest version of frappe and the apps. But the database was the old version.

So I have incorporated a bench migrate into the create-site container in the docker-compose file.

Not sure why the database permissions disappeared - still investigating this one.

3

Which directory? frappe_docker directory, the repo that was cloned will not have any data.

docker data is in /var/lib/docker, that will have so that volumes including db volumes.

You can use restic which is part of image, How to backup with restic to S3 compatible storage

With restic you can use multiple backends to take and restore snapshots. You need to do is bench --site all backup and create restic snapshot of sites directory, that way you can take backup of updated files.

Example backup commands custom_containers/compose/erpnext-backup.yml at main · castlecraft/custom_containers · GitHub

4

It was the sites folder I reproduced. Note that I am using direct folder mounts, rather then hidden docker volumes, to store the site data.

I am trying again.

The new site was working, so I copied a backup of the old site (taken with bench backup, including all the files) to the new site, and restored it (using bench restore), then upgraded it using bench migrate.

Now the site pages look a mess, because it is trying to load (e.g.) assets/frappe/dist/css/desk.bundle.ZYSAVPXI.css - this returns Not Found.

Loading the same pages on the old site works fine - it is loading /assets/frappe/dist/css/desk.bundle.MDPUJYT3.css instead.

Don’t understand why the difference, or why rebuilding the container didn’t update the assets.

Just rebuilding the containers again - I notice some warnings:

WARN Cannot connect to redis_cache to update assets_json

Is that the problem?

[Later]
Did a build of only front-end with --no-cache. Then a build of everything (so it would use the cached build just created). Suddenly it all seems to work again.

I wish I understood what is going on, and how it all fits together. My current theory is that build --no-cache creates a randomly named “asset” (e.g. assets/frappe/dist/css/desk.bundle.ZYSAVPXI.css), so that, if all the containers are built with --no-cache, the last container built has named all the assets with one random name, but the containers built earlier are expecting a different random name. But it’s a wild guess!

We would be happy to pay for a day’s consultation from someone who really understand how it all fits together, and show us what we are doing wrong.

5

You need to re create the containers instead of using old. Don’t mount assets volume and new volume will be created along with new containers.

Recent post that discuss creating new containers. Best way to deploy new versions of custom app in self hosted docker setup

6

I think it says there the best way to deploy is something called a swarm.
Don’t have such a thing!

We are just deploying using docker-compose (the version which comes with our Debian - not the new one you all seem to use).

We don’t have the option of deploying the latest docker, or a swarm (whatever that is).

It would be really helpful to understand why there are 3 volumes created in your container (sites, sites/assets and logs), and why it is configured in such a way as to cause this problem.

7

If you have 3 volumes you can remove the assets volume. logs is optional.

You need to go through this to understand the change in volumes frappe_docker/docs/migrate-from-multi-image-setup.md at main · frappe/frappe_docker · GitHub

pwd.yml explained frappe_docker/docs/single-compose-setup.md at main · frappe/frappe_docker · GitHub

pwd.yml frappe_docker/pwd.yml at main · frappe/frappe_docker · GitHub

once your volumes for assets are not bind mounted or specified in containers, It’ll auto-create volumes and mount with new created containers.

I think docker-compose up re-creates new containers.

Debian Bookworm was released recently. Check if it gives you newer version of software.
If you’re moving to new server, move to updated debian stable.

1 Like
8

I have looked at all those, and am none the wiser. I guess I’m thicker than I think I am!

In the Containerfile for custom (for that is what I must start with - I want to install apps like notforprofit), there is the following code:

VOLUME [ \
  "/home/frappe/frappe-bench/sites", \
  "/home/frappe/frappe-bench/sites/assets", \
  "/home/frappe/frappe-bench/logs" \
]

If I understand correctly, that will create 3 persistent docker volumes the first time the container is built, and the data will remain there for subsequent builds.

Is there a reason for the assets volume?

9

Also, I decided, as an experiment, to try to use the standard frappe-custom image mentioned in the repo - only to find it no longer exists
pull access denied for frappe-custom, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

10

There was no such image built

If you specify the mount then it does that.
If you don’t specify any type of mount then it’ll create new unnamed volume whenever the container is recreated.

1 Like
11

So how do I follow the instructions (e.g. frappe_docker/docs/list-of-containers.md at main · frappe/frappe_docker · GitHub and Best way to deploy new versions of custom app in self hosted docker setup ) to use it, if it doesn’t exist?

In which file do I “specify the mount”?
In the container build file (Dockerfile, in my version)?
In the docker-compose?
In both?

Sorry to be so ignorant about how docker works. I have read pages and pages of documentation, but seem to have learned nothing - I have no model in my mind of how it works, so most of it is just magic to me!

12

On a related note, is the frontend container necessary? We have nginx running on the host - if we adapted the nginx.conf file from the frontend container into a nginx conf file on the host, could we not bother with the frontend container? It seems inefficient to have 2 versions of nginx running, the host proxying to the one in frontend, which presumably proxies to the one in backend and/or websocket.

14

I didn’t get what you’re trying to follow

If you’re trying to build image with custom apps then follow frappe_docker/docs/custom-apps.md at main · frappe/frappe_docker · GitHub

In compose yaml, under volumes section. Don’t specify anything for assets.

If your nginx can serve static assets from some running container, reverse proxy gunicorn and websocket, serve public and private files per site. Then try it without frontend containers.

15

So I have to build a container using plain docker? I am used to just using docker-compose build.

The instructions say I have to push it to docker hub - but I find (by experiment) that I don’t, it will use the local image,

I don’t understand why ERPNext is so difficult to deploy. Is it just that there haven’t been enough resources to make it easier, or is there some fundamental problem with making a docker image you can just pull and run?

16

But my experiment seems to have failed - I built the new container, and apps.json still only has frappe in it.

Could that be a left-over from having an existing sites volume? Should I delete the sites volume before building? Or before bringing the container up for the first time?

By the way, and very importantly, thank you for all your help.

17

Whatever needs to be there for building images, pushing them or not pushing them, FAQ, docker compose, docs is out there in public.

There were few people who were able to build and use custom images.

I’m out of ideas to help you, may be someone who has successfully setup erpnext with your constraints might help.

18

So most people couldn’t get it to work?

I haven’t yet found an overview of how docker and docker-compose work - I find manuals, and read them, but without a better understanding of how and why, it is difficult.

Will do some more experiments on this tomorrow.

19

Seems like it. Only I use containers. No one else answers.

There are countless working site running on containers.

More than 100 that I can remember I’ve setup. I stopped keeping count. No one will answer here. If no one answers here means no one uses containers. Or containers are so reliable that no one has any problem!

Kubernetes/Helm chart users are separate. There are many setups running on Kubernetes. Many I’ve helped upgrade when they ask on frappe/helm repo. One one will answer here.

In past few days someone faced problem building image with payments, it was reported and fixed, user forked payments repo and used it. Countless people use it and don’t even report back.

I don’t think you’ll find the following posts useful. Or I guess they are not comprehensive enough for users to understand anything.

I really can’t help you much.

1 Like
20

I’m one of those (few?) people who got erpnext working with containers on regular docker-compose. =)

I think lots of people have problems when they think it should be super easy, then find out it isn’t. Docker (and projects that promote using docker to deploy) tend to get that advertising, “It’s so easy, why doesn’t everyone just use docker?” etc.

Real life, that’s not really true. Every kind of technology needs to be learned. That’s true for networking, coding (even “easy” languages), virtualization, containers, you name it. There are no shortcuts. If you want to utilize a technology, you have to put in the time to learn it. Sorry. It’s true.

I started with ERPNext on an Ubuntu VM since that’s what most people use, because I knew already how to deploy virtual machines, and because that’s what the documentation was mostly written for. Then I got bitten by the “I want to learn Docker” bug. I tried deploying erpnext that way, but I had problems since I was shorcutting the process of learning Docker slowly and properly.

Eventually I gave into the concept that I would actually have to put in the time learning how Docker works and experimenting with it before expecting to do something useful with it. Docker/containers are complicated and there’s a lot to learn: images, custom builds, compose files, networking, volumes, etc.

I’m still not an expert, but I know enough to get what I want working now. That “Container Basics” link @revant_one posted would be a great first stop. Then try to deploy a vanilla erpnext install from the “single server example” in the frappe_docker github without customizing anything. Just follow the steps and see if the login page shows up. Like anything worth doing, this takes time!

3 Likes
21

I can build and install plain erpnext using docker-compose at least one time in two (I have found a problem where the assets.json file gets out of step with the randomly generated asset names, usually every other build). Especially now I have realised you need to migrate if you are using an existing database, because the code might have changed (I added that into the create-site container in my docker-compose.yml, and it helped a lot).

But I keep hitting issues caused by my not understanding docker properly (especially volumes).

I started with a plain erpnext install. Then we realised we needed the notforprofit app. So I rebuilt to add that, and it didn’t work. That’s because the rebuild created a new volume with the new apps.json in it, but docker-compose up ignored that, and used the old volume, with the old apps.json in it.

In a plug-and-play dockerised app, there would be some code that recognised the new build was connecting with an old volume, and set about upgrading anything in the old volume that needed it.

So I just deleted my sites volume, and re-deployed. Now the new frappe can’t connect to the database, because it has created a new site_config.json, with a new random database password, but it hasn’t granted access to the frappe user with that password.

All very fine if you fully understand frappe, and are prepared to patch files in your volume after running it, but not exactly plug and play.

So I edited the database password in the sites volume to be the same as the old one. Now it connects, but there is still no sign of the apps in the site. Apps.json is correct.

Looking at the create-site logs, I find ERPNext can only be installed on a fresh site where the setup wizard is not completed.. So it looks as if I have to trash my database volume too. But how would you go about installing a new app to an existing site?

I’m writing this down both to explain my problems, and to enlighten anyone else who comes along later and experiences any of these issues.

1 Like