Kubernetes: Decoupled Frappe/ERPNext docker images

[revant_one]

Git repo: castlecraft / erpnext_kubernetes · GitLab

Preparing ERPNext for kubernetes:

• Uses vanilla upstream/develop branches
• Single process per container
• Decoupled docker containers
  • ERPNext Python container (image:erpnext-python)
  • ERPNext Default Worker container (image:erpnext-python, command:worker)
  • ERPNext Short Worker container (image:erpnext-python, command:worker)
  • ERPNext Long Worker container (image:erpnext-python, command:worker)
  • ERPNext Assets container (image:erpnext-assets)
  • Frappe SocketIO container (image:frappe-socketio)
• sites folder is shared between all containers
• assets folder is populated by erpnext-assets and shared with erpnext-python
• nginx proxy pass the $http_host, (kubernetes ingress will manage domain names and certificates)
• SocketIO, Chat, private public File upload works. Tell me what doesn’t work.
• WIP: Helm Chart will have all of the above as deployments (independently scalable)
• WIP: one helm release=one frappe-bench
• WIP: ERPNext Assets and ERPNext python will be part of single pod (assets need to be shared)
• WIP: Try out ReadWriteMany volumes and mount shared sites directory.
• WIP: CronJob/Job kubernetes resource to migrate the bench

It does not use bench at all. Instead .py files are dropped in commands directory to do what bench does. Only basic bench functionality is needed in container environment.

Brief steps to get started (docker and docker-compose needed for testing)

git clone https://gitlab.com/revant.one/erpnext_kubernetes.git && erpnext_kubernetes

Copy example Env file

cp example-env .env

Start / Scale services

docker-compose up \
    --scale erpnext-worker-default=4 \
    --scale erpnext-worker-long=4 \
    --scale erpnext-worker-short=4 \
    --scale erpnext-python=4

NOTE: Wait for mariadb to start successfully and listen to connections

create new site with erpnext (default=site1.localhost)

docker exec -it -e INSTALL_ERPNEXT=1 \
    erpnext_kubernetes_erpnext-python_1 docker-entrypoint.sh new

visit http://site1.localhost to proceed with setup wizard

migrate site (migrates the given site with patches from container)

docker exec -it -e SITE_NAME=site1.localhost docker-entrypoint.sh migrate

Many things will be improved

update: 2019-11-25

Frappe/ERPNext Docker Images

Registry: Container Registry · castlecraft / erpnext_kubernetes · GitLab

• All images are built daily at 4.00 AM UTC
• All images are python 3.7

There are 3 tagged images

• egde: This image is baked from develop branch
• v11: This image is baked from version-11 branch. It will deliver the latest v11.x.x. image
• v12: This image is baked from version-12 branch. It will deliver the latest v12.x.x. image

Minor and patch versions are also available with the version specific tag. e.g. v11.1.62

update: 2019-12-31

Docker Images on docker hub

• Docker
• Docker
• Docker

[rohitbanerjee]

I installed docker and docker-compose on a fresh CentOS 7 x64 VPS, following instructions up to Step Two of How To Install and Use Docker on CentOS 7, and and up to Step One of How To Install and Use Docker Compose on CentOS 7.

However, I ran into an error on this command:

@revant_one

git clone https://gitlab.com/revant.one/erpnext_kubernetes && erpnext_kubernetes

$ git clone https://gitlab.com/revant.one/erpnext_kubernetes && cd erpnext_kubernetes
Cloning into 'erpnext_kubernetes'...
error: RPC failed; result=22, HTTP code = 404
fatal: The remote end hung up unexpectedly

[brian_pond]

This is the best ERPNext + Docker work I have seen yet.

Should we put 100% of ERPNext and its dependencies in containers? Maybe. Maybe not.

But @revant_one has demonstrated how it’s done. Everyone can learn from his solution, which is awesome.

[vrms]

@revant_one

Uses vanilla upstream/develop branches

so is this for development only or can you install a production instance (v11, v12) with this?

[rmeyer]

Great work, thanks @revant_one!

[revant_one]

I’m actively working with develop branch.

Once I’ve everything figured out I’ll make images for prod branches (v11, v12)

Anyone can refer the source and build their own custom apps / old versions into images

I’ve a POC on wip-helm branch. I’m experimenting with things there.

Source: https://gitlab.com/revant.one/erpnext_kubernetes/tree/wip-helm

What I was able to achieve (using hardcoded hostnames for trying out things.)

• all pods run (without health probes)
• common ReadWriteMany volume is mounted at location frappe-bench/sites for all containers
• I was successful with using S3 / Digitalocean Spaces as a mounted volume ( used GitHub - ctrox/csi-s3: A Container Storage Interface for S3)

Screenshots

Running pods

Files written in bucket (DO Spaces)

[rohitbanerjee]

@revant_one

(using hardcoded hostnames for trying out things.)

@revant_one, do you thnk this is why I’m getting the error

error: RPC failed; result=22, HTTP code = 404
fatal: The remote end hung up unexpectedly

[rohitbanerjee]

I changed the URL to end with .git and it worked!
so the git clone command that worked for me was:

git clone https://gitlab.com/revant.one/erpnext_kubernetes.git && cd erpnext_kubernetes

[tejpochiraju]

@revant_one

I was successful with using S3 / Digitalocean Spaces as a mounted volume ( used https://github.com/ctrox/csi-s3 )

Thanks @revant_one! I am going to try this out this weekend. I have had pretty poor performance (write latency) with s3fs style solutions previously, did csi-s3 work OK?

[revant_one]

During setup (writing config files) I didn’t feel any difference.

Probably I’ll need to try for few days.

I’m not using s3fs. I’m using goofys GitHub - kahing/goofys: a high-performance, POSIX-ish Amazon S3 file system written in Go

[revant_one]

@vrms

so is this for development only or can you install a production instance (v11, v12) with this?

I have a schedule that builds version tagged images for v11, v12 and edge. Check updated post
It is ready for production deployment, checkout the gitlab repo.

I’ll write a separate post on

• how to setup a dockerized erpnext in production with letsencrypt.
• how to setup erpnext on kubernetes using helm chart

[vrms]

@revant_one

how to setup a dockerized erpnext in production with letsencrypt.

I guess those instructions are what you’ll have to follow:

Production Installation · Wiki · castlecraft / erpnext_kubernetes · GitLab

I’ll check that out in a bit. thanks thus far @revant_one

[revant_one]

Just for reference I posted same on forum

Docker based production installation

Prerequisites: Docker docker-compose List of sites and server ip added to DNS 1. Install docker-compose-letsencrypt-nginx-proxy-companion cd $HOME git clone https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion cd docker-compose-letsencrypt-nginx-proxy-companion cp .env.sample .env ./start.sh 2. Clone repository cd $HOME git clone https://gitlab.com/castlecraft/erpnext_kubernetes.git cd erpnext_kubernetes touch .env 3. Setup .env file MYSQL_ROOT_PASSWORD=admin,…

for helm chart check the directory kubernetes/erpnext from the repo.

Based on that, I’ll write about kubernetes based installation.

[vrms]

I try to run the v12 following these instructions edition but can’t really achieve a running system.

when I am executing the

docker exec -it \
    -e SITE_NAME=test.localhost \
    -e DB_ROOT_USER=root \
    -e DB_ROOT_PASSWORD=admin \
    -e ADMIN_PASSWORD=admin \
    -e INSTALL_ERPNEXT=1 \
    erpnext_kubernetes_erpnext-python_1 docker-entrypoint.sh new

I am getting Site test.localhost already exists
I can, however go through with this when I choose a different site name. However I can not access my http://test2.localhost site afterwards

also in the terminal that is running the docker-compose ... up command I have noted

mariadb_1 | 2019-12-08 23:57:02 50 [Warning] Access denied for user '_fcadc07dbdbc21db'@'172.21.0.10' (using password: YES)

I am wondering what I may be doing wrong

[revant_one]

I fixed it and also added environment variable for FORCE=1, It will force re-installation if required

[bohlian]

@revant_one thanks for this.

Have been trying to experiment with this but could not get it to successfully run. I’m up to a point where I can see the “Sorry! We will be back soon” page.

The erpnext_kubernetes_erpnext-python_1 container seems to have started fine based on the logs:

But I can not get past this. Any idea what I may be doing wrong?

Thanks.

[revant_one]

Did you add the first site? castlecraft / erpnext_kubernetes · GitLab

Nginx will proxy to site available under bench/venv

name the sites with .localhost, e.g erp.localhost for trying out locally. That way you don’t have to edit /etc/hosts and browser treats localhost as secure

[bohlian]

Yes I added the first site. I think it is DNS related (as you have mentioned) I’m quite confused as to how it should be routing.

I tried IP address (container’s IP) and localhost, both did not work. Is there a way to get it working using port based?

I named the site as server1-docker.domain.local which is not the actual hostname of the server but something I made up, I also configured DNS to point this hostname to the actual server itself. Did I do something wrong here?

[revant_one]

@bohlian

server1-docker.domain.local

Try adding this to /etc/hosts of host machine. It should point to 127.0.0.1

For anything other than .localhost to work it needs to be added in /etc/hosts of host machine.

Port 80 of asset container is forwarded on localhost so it will be available without entering container

Easiest is to create site named server1-docker.domain.localhost instead of server1-docker.domain.local.

[bohlian]

Hmm… Didnt seem to work.

Ok lets say I abandon this. Can I follow these instructions and then just name the site server1-docker.domain.localhost?

But from outside the host machine what URL should I type to access the site?

The host machine is server1.domain