Help! Moving docker instalation from one server to another

[nikkilocke]

On a related note, is the frontend container necessary? We have nginx running on the host - if we adapted the nginx.conf file from the frontend container into a nginx conf file on the host, could we not bother with the frontend container? It seems inefficient to have 2 versions of nginx running, the host proxying to the one in frontend, which presumably proxies to the one in backend and/or websocket.

[revant_one]

@nikkilocke

So how do I follow the instructions

I didn’t get what you’re trying to follow

If you’re trying to build image with custom apps then follow frappe_docker/docs/custom-apps.md at main · frappe/frappe_docker · GitHub

@nikkilocke

In which file do I “specify the mount”?

In compose yaml, under volumes section. Don’t specify anything for assets.

@nikkilocke

frontend container necessary

If your nginx can serve static assets from some running container, reverse proxy gunicorn and websocket, serve public and private files per site. Then try it without frontend containers.

[nikkilocke]

So I have to build a container using plain docker? I am used to just using docker-compose build.

The instructions say I have to push it to docker hub - but I find (by experiment) that I don’t, it will use the local image,

I don’t understand why ERPNext is so difficult to deploy. Is it just that there haven’t been enough resources to make it easier, or is there some fundamental problem with making a docker image you can just pull and run?

[nikkilocke]

But my experiment seems to have failed - I built the new container, and apps.json still only has frappe in it.

Could that be a left-over from having an existing sites volume? Should I delete the sites volume before building? Or before bringing the container up for the first time?

By the way, and very importantly, thank you for all your help.

[revant_one]

Whatever needs to be there for building images, pushing them or not pushing them, FAQ, docker compose, docs is out there in public.

There were few people who were able to build and use custom images.

I’m out of ideas to help you, may be someone who has successfully setup erpnext with your constraints might help.

[nikkilocke]

@revant_one

There were few people who were able to build and use custom images.

So most people couldn’t get it to work?

@revant_one

Whatever needs to be there for building images, pushing them or not pushing them, FAQ, docker compose, docs is out there in public.

I haven’t yet found an overview of how docker and docker-compose work - I find manuals, and read them, but without a better understanding of how and why, it is difficult.

Will do some more experiments on this tomorrow.

[revant_one]

@nikkilocke

So most people couldn’t get it to work?

Seems like it. Only I use containers. No one else answers.

There are countless working site running on containers.

More than 100 that I can remember I’ve setup. I stopped keeping count. No one will answer here. If no one answers here means no one uses containers. Or containers are so reliable that no one has any problem!

Kubernetes/Helm chart users are separate. There are many setups running on Kubernetes. Many I’ve helped upgrade when they ask on frappe/helm repo. One one will answer here.

In past few days someone faced problem building image with payments, it was reported and fixed, user forked payments repo and used it. Countless people use it and don’t even report back.

@nikkilocke

I haven’t yet found an overview of how docker and docker-compose work

I don’t think you’ll find the following posts useful. Or I guess they are not comprehensive enough for users to understand anything.

Container Basics

Why? People who don’t know containers land up successfully installing ERPNext with containers and get stuck later. Read for container basics. Prerequisites Unix/Linux basics, init system, processes, mount points, bash shell. podman or docker, podman-compose or docker compose is installed. Go for rootless podman if you can. Standard docker will work. Make sure you’re NOT using snap to install docker, it’ll not work as expected through snap. Install, docker docker compose Or alternatively …

Container Builds

Read Container Basics before you begin. Let’s build a simple frappe py server that returns 404 Not Found Create a Dockerfile in current working directory. FROM python:3-alpine # Install required os packages to install frappe RUN apk add -U git build-base linux-headers && \ # Create directory for apps and sites mkdir -p /opt/simple-server/apps /opt/simple-server/sites && \ # Create python env python3 -m venv /opt/simple-server/env && \ # install frappe version-14 /opt/simple-serve…

I really can’t help you much.

[trustedcomputer]

I’m one of those (few?) people who got erpnext working with containers on regular docker-compose. =)

I think lots of people have problems when they think it should be super easy, then find out it isn’t. Docker (and projects that promote using docker to deploy) tend to get that advertising, “It’s so easy, why doesn’t everyone just use docker?” etc.

Real life, that’s not really true. Every kind of technology needs to be learned. That’s true for networking, coding (even “easy” languages), virtualization, containers, you name it. There are no shortcuts. If you want to utilize a technology, you have to put in the time to learn it. Sorry. It’s true.

I started with ERPNext on an Ubuntu VM since that’s what most people use, because I knew already how to deploy virtual machines, and because that’s what the documentation was mostly written for. Then I got bitten by the “I want to learn Docker” bug. I tried deploying erpnext that way, but I had problems since I was shorcutting the process of learning Docker slowly and properly.

Eventually I gave into the concept that I would actually have to put in the time learning how Docker works and experimenting with it before expecting to do something useful with it. Docker/containers are complicated and there’s a lot to learn: images, custom builds, compose files, networking, volumes, etc.

I’m still not an expert, but I know enough to get what I want working now. That “Container Basics” link @revant_one posted would be a great first stop. Then try to deploy a vanilla erpnext install from the “single server example” in the frappe_docker github without customizing anything. Just follow the steps and see if the login page shows up. Like anything worth doing, this takes time!

[nikkilocke]

I can build and install plain erpnext using docker-compose at least one time in two (I have found a problem where the assets.json file gets out of step with the randomly generated asset names, usually every other build). Especially now I have realised you need to migrate if you are using an existing database, because the code might have changed (I added that into the create-site container in my docker-compose.yml, and it helped a lot).

But I keep hitting issues caused by my not understanding docker properly (especially volumes).

I started with a plain erpnext install. Then we realised we needed the notforprofit app. So I rebuilt to add that, and it didn’t work. That’s because the rebuild created a new volume with the new apps.json in it, but docker-compose up ignored that, and used the old volume, with the old apps.json in it.

In a plug-and-play dockerised app, there would be some code that recognised the new build was connecting with an old volume, and set about upgrading anything in the old volume that needed it.

So I just deleted my sites volume, and re-deployed. Now the new frappe can’t connect to the database, because it has created a new site_config.json, with a new random database password, but it hasn’t granted access to the frappe user with that password.

All very fine if you fully understand frappe, and are prepared to patch files in your volume after running it, but not exactly plug and play.

So I edited the database password in the sites volume to be the same as the old one. Now it connects, but there is still no sign of the apps in the site. Apps.json is correct.

Looking at the create-site logs, I find ERPNext can only be installed on a fresh site where the setup wizard is not completed.. So it looks as if I have to trash my database volume too. But how would you go about installing a new app to an existing site?

I’m writing this down both to explain my problems, and to enlighten anyone else who comes along later and experiences any of these issues.

[revant_one]

@nikkilocke

I started with a plain erpnext install. Then we realised we needed the notforprofit app. So I rebuilt to add that, and it didn’t work. That’s because the rebuild created a new volume with the new apps.json in it, but docker-compose up ignored that, and used the old volume, with the old apps.json in it.

apps.txt is important, ignore sites/apps.json

frappe/frappe_docker/blob/ce272a9290742befa8fa4ae1b6c73162fb23c266/pwd.yml#L23-L23

Above command updates the apps.txt with apps from image.

apps.json has git details about apps. In container images apps are not git repos. .git dir is removed

frappe/frappe_docker/blob/ce272a9290742befa8fa4ae1b6c73162fb23c266/images/custom/Containerfile#L122-L122

For updating static assets do not specify assets under volumes in your compose.yaml, refer pwd.yml

1. Stop containers
2. Remove stopped containers
3. remove hash / unnamed docker volume (docker volume ls|rm) if your sites is in bind mount on host directory then you can just do docker volume prune and it’ll remove volumes not attached to containers
4. Restart containers with new image, it’ll create new unnamed/hash volumes for new static assets.

To add or remove apps, run command to update apps.txt and bench install-app uninstall-app. Sequence:

To add app

1. Update images using above steps
2. update apps.txt
3. bench install-app

To remove

1. bench uninstall-app
2. Update image
3. update apps.txt

Other notes:

create-site is run once, it’s there in PWD for demo.
You need configurator and migration, refer custom_containers/compose/erpnext.yml at main · castlecraft/custom_containers · GitHub for migration. Refer PWD for configurator. Configurator needs to run when you change db, redis hosts or any common config or you need to update apps.txt

You can remove them after their job is done. Or the docker-compose exec command can be used to avoid adding run once services

[nikkilocke]

Thanks @revant_one - that’s really helpful.

I hadn’t got the rebuild of apps.txt line in my docker-compose - I guess I must have made mine more than 4 months ago!

Will it do any harm to rerun the build container on every restart? Or do I have to take it out of the docker-compose, or put some test in it so it does nothing for an existing site?

Any offers of wisdom on this question? How to restore a backup from one site name to another - #5 by nikkilocke

[revant_one]

@nikkilocke

Will it do any harm to rerun the build container on every restart? Or do I have to take it out of the docker-compose, or put some test in it so it does nothing for an existing site?

What do we mean by “build”

• if build is the “configurator” and “migration” service ideally keep it separate so even on server restarts it doesn’t trigger them. If you are okay with them running over and over again. Or you managed to script it in for when to run, then keep it in compose yaml
• if build means image build service, then ideally build your image with a name and tag, it will be visible locally, execute docker images to list local images. Then you can use the image in your compose, if the {name}:{tag} exists locally it’ll not complain or try to pull from remote and use the local image. This way you don’t trigger builds on server restart.

[nikkilocke]

I should have been more clear, sorry - I mean the containers in the pwd.yml called configurator and create-site. Will it do any harm to run them on an existing, working setup? e.g. when the server is rebooted, or the site is taken down for some reason and then restarted?

[revant_one]

create site will fail with “site already exists”.

configurator will run and update files, unnecessary io on disk.

migration, will do bench migrate, site will be unavailable for some time.

[nikkilocke]

OK, so it won’t actually break anything. I will leave them in, because without them people have to know how to (and remember to) do these steps manually sometimes.

[revant_one]

the configurator should complete successfully and nothing else should try to read/write to same json.

it causes error if multiple processes try to update json

Service "configurator" didn't completed successfully: exit 1

configurator should be run once. make sure your common_site_config.json or site_config.json are correct and valid json

[nikkilocke]

@revant_one

the configurator should complete successfully and nothing else should try to read/write to same json.

I’m not quite sure what that means. I am running a docker-compose file based very closely on the pwd.yml one you provide.

Is that wrong in some way?

[revant_one]

@nikkilocke

Is that wrong in some way?

PWD is for demo. It includes all services that need to run to have the site running. There are no restarts expected in this environment.

[nikkilocke]

Sorry, @revant_one , I’m still totally unclear what should be in my docker-compose file, then.

What I am looking for is a single docker-compose file that I can start up, and be sure that I will end up with a working system. It should still function if I:

• Rebuild the container, to get the latest code
• Restore a backup from another instance, possibly running a different version
• Build from scratch with empty volumes
• Start or build with existing volumes

I’m trying to make this foolproof (and failing miserably!). Perhaps because I am an inventive fool

[revant_one]

Safest alternative is to remove run once services from compose and run them as single task using docker run

frappe/frappe_docker/blob/main/docs/single-compose-setup.md#run-once-configuration

Refer container basics for docker run command

If you use immutable image tags then once configured the system can restart without any change.

Build image should be separate. May be create a compose for it separately.

If you don’t want immutable image tags, prefer use of branches instead of tags, no plan to scale bench across servers then struggle so much over native bench and it can do everything with single command bench update