Multitenant on docker swarm + traefik

efleurimond · March 25, 2022, 3:13am

Continuing the discussion from Docker Swarm + Portainer: Frappe only, need ERPNext:

@Ysedky, would you mind sharing how you were able to create multiple sites on docker swarm?

Here is what I did so far:

Create two stacks one for traefik and another one for portainer, they both work well. Traefik was able to detect portainer as soon as I deploy the stack
Create a stack for mariadb which also works well
Create a stack for redis, nginx, frappe and erpnext. Traefik detected the site created from that stack successfully.

My problem is how to have traefik detect additional sites. I tried creating additional sites first by modifying and restarting the site creator service from the initial stack, the site was successfully created according to the container log, but I cannot access it because traefik did not detect it (I guess nginx neither). I was also able to create multiple additional sites by manually creating new containers using portainer, creation went well according to the logs… except traefik never detected those sites (only the first one created with the initial stack). I was thinking about duplicating the initial stack, but I guess I would end up filling the server with duplicate containers for all the different services.

My post is very long and I’m sorry for that, but I felt like attaching the codes would help the community better understand my problem. Thank you for your help.

traefik-stack

version: “3.9”

services:
traefik:
depends_on:
- socket-traefik
image: traefik:v2.6.1
command:
- --providers.docker
- --providers.docker.endpoint=tcp://socket-proxy:2375
- --providers.docker.constraints=Label(traefik.constraint-label, cloud-public)
- --providers.docker.exposedByDefault=false
- --providers.docker.swarmMode=true
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.api.address=:8083
- --certificatesresolvers.letsencrypt.acme.email=${EMAIL?Variable not set}
- --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
- --certificatesresolvers.letsencrypt.acme.tlschallenge=true
- --log=true
- --log.filePath=/logs/traefik.log
- --log.level=DEBUG
- --accesslog=true
- --accesslog.filepath=/logs/access.log
- --accesslog.bufferingsize=25
- --api
deploy:
restart_policy:
condition: on-failure
placement:
constraints:
- “node.labels.traefik-node.traefik-certificates == true”
- “node.role == manager”
labels:
- “traefik.enable=true”
- “traefik.docker.network=traefik-int”
- “traefik.constraint-label=cloud-public”
- “traefik.http.routers.http-redirects.entrypoints=web”
- ‘traefik.http.routers.http-redirects.rule=hostregexp({host:(www\.)?.+})’
- “traefik.http.routers.http-redirects.middlewares=traefik-ratelimit,redirect-to-non-www-https”
- “traefik.http.routers.www-redirects.entrypoints=websecure”
- ‘traefik.http.routers.www-redirects.rule=hostregexp({host:(www\.).+})’
- “traefik.http.routers.www-redirects.tls=true”
- “traefik.http.routers.www-redirects.middlewares=traefik-ratelimit,redirect-to-non-www-https”
- ‘traefik.http.middlewares.redirect-to-non-www-https.redirectregex.regex=^https?://(?:www.)?(.+)’
- “traefik.http.middlewares.redirect-to-non-www-https.redirectregex.replacement=https://$${1}”
- “traefik.http.middlewares.redirect-to-non-www-https.redirectregex.permanent=true”
- “traefik.http.routers.api.entrypoints=api”
- “traefik.http.routers.api.rule=Host(traefik.${DOMAIN?Variable not set})”
- “traefik.http.routers.api.service=api@internal”
- “traefik.http.routers.api.tls=true”
- “traefik.http.routers.api.tls.certresolver=letsencrypt”
- “traefik.http.routers.api.middlewares=api-ipwhitelist,traefik-ratelimit,api-auth”
- “traefik.http.middlewares.api-auth.basicauth.users=${TRAEFIK_ADMINS?Variable not set}”
- “traefik.http.middlewares.api-ipwhitelist.ipwhitelist.sourcerange=172.18.0.0/16, x.x.x.x”
- “traefik.http.services.api.loadbalancer.server.port=8083”
- “traefik.http.routers.traefik-secure.tls.domains[0].main=${DOMAIN?Variable not set}”
- “traefik.http.routers.traefik-secure.tls.domains[0].sans=*.${DOMAIN?Variable not set}”
- “traefik.http.middlewares.traefik-ratelimit.ratelimit.average=100”
- “traefik.http.middlewares.traefik-ratelimit.ratelimit.burst=50”
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
- target: 8083
published: 8083
protocol: tcp
mode: host
volumes:
- /opt/traefik-certificates:/letsencrypt
- /opt/traefik-logs:/logs
- /opt/traefik-config:/etc/traefik
networks:
- traefik-int
- socket-int

socket-traefik:
image: tecnativa/docker-socket-proxy:latest
deploy:
restart_policy:
condition: on-failure
placement:
constraints:
- “node.role == manager”
environment:
NETWORKS: 1
SERVICES: 1
TASKS: 1
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
- socket-int

networks:
traefik-int:
external: true
socket-int:
driver: overlay
driver_opts:
encrypted: “true”

erpnext-stack

version: “3.7”

services:
redis-cache:
image: redis:latest
volumes:
- redis_cache_vol:/data
deploy:
restart_policy:
condition: on-failure
networks:
- erpnext-network

redis-queue:
image: redis:latest
volumes:
- redis_queue_vol:/data
deploy:
restart_policy:
condition: on-failure
networks:
- erpnext-network

redis-socketio:
image: redis:latest
volumes:
- redis_socketio_vol:/data
deploy:
restart_policy:
condition: on-failure
networks:
- erpnext-network

erpnext-nginx:
image: frappe/erpnext-nginx:${ERPNEXT_VERSION}
environment:
- FRAPPE_PY=frappe-python
- FRAPPE_PY_PORT=8000
- FRAPPE_SOCKETIO=frappe-socketio
- SOCKETIO_PORT=9000
networks:
- erpnext-network
- traefik-int
volumes:
- sites_vol:/var/www/html/sites:rw
- assets_vol:/assets:rw
deploy:
restart_policy:
condition: on-failure
labels:
- “traefik.enable=true”
- “traefik.docker.network=traefik-int”
- “traefik.constraint-label=cloud-public”
- “traefik.http.routers.erpnext-nginx.entrypoints=websecure”
- “traefik.http.routers.erpnext-nginx.rule=Host(${SITES?Variable not set})”
- “traefik.http.routers.erpnext-nginx-https.rule=Host(${SITES?Variable not set})”
- “traefik.http.routers.erpnext-nginx-https.tls=true”
- “traefik.http.routers.erpnext-nginx-https.tls.certresolver=letsencrypt”
- “traefik.http.services.erpnext-nginx.loadbalancer.server.port=8080”

frappe-python:
image: frappe/erpnext-worker:${ERPNEXT_VERSION}
deploy:
restart_policy:
condition: on-failure
environment:
- MARIADB_HOST=${MARIADB_HOST}
- REDIS_CACHE=redis-cache:6379
- REDIS_QUEUE=redis-queue:6379
- REDIS_SOCKETIO=redis-socketio:6379
- SOCKETIO_PORT=9000
- AUTO_MIGRATE=1
networks:
- erpnext-network
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
- assets_vol:/home/frappe/frappe-bench/sites/assets:rw

frappe-socketio:
image: frappe/frappe-socketio:${FRAPPE_VERSION}
deploy:
restart_policy:
condition: on-failure
networks:
- erpnext-network
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
- logs_vol:/home/frappe/frappe-bench/logs:rw

erpnext-worker-default:
image: frappe/erpnext-worker:${ERPNEXT_VERSION}
deploy:
restart_policy:
condition: on-failure
command: worker
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
- logs_vol:/home/frappe/frappe-bench/logs:rw
networks:
- erpnext-network

erpnext-worker-short:
image: frappe/erpnext-worker:${ERPNEXT_VERSION}
deploy:
restart_policy:
condition: on-failure
command: worker
environment:
- WORKER_TYPE=short
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
- logs_vol:/home/frappe/frappe-bench/logs:rw
networks:
- erpnext-network

erpnext-worker-long:
image: frappe/erpnext-worker:${ERPNEXT_VERSION}
deploy:
restart_policy:
condition: on-failure
command: worker
environment:
- WORKER_TYPE=long
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
networks:
- erpnext-network

erpnext-schedule:
image: frappe/erpnext-worker:${ERPNEXT_VERSION}
deploy:
restart_policy:
condition: on-failure
command: schedule
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
- logs_vol:/home/frappe/frappe-bench/logs:rw
networks:
- erpnext-network

erpnext-site-creator:
image: frappe/erpnext-worker:${ERPNEXT_VERSION}
deploy:
restart_policy:
condition: none
command: new
depends_on:
- frappe-python
environment:
- SITE_NAME=${SITE_NAME}
- DB_ROOT_USER=${DB_ROOT_USER}
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- ADMIN_PASSWORD=${ADMIN_PASSWORD}
- INSTALL_APPS=${INSTALL_APPS}
networks:
- erpnext-network
volumes:
- sites_vol:/home/frappe/frappe-bench/sites:rw
- logs_vol:/home/frappe/frappe-bench/logs:rw

volumes:
redis_cache_vol:
redis_queue_vol:
redis_socketio_vol:
assets_vol:
sites_vol:
logs_vol:

networks:
erpnext-network:
external: true
traefik-int:
external: true

Ysedky · April 3, 2022, 9:00pm

Hi @efleurimond
Here are the steps I followed:

1- Follow dockerswarm.rocks guide to setup Traefik Traefik Proxy with HTTPS - Docker Swarm Rocks

PS: Make sure to add DNS A Record for (e.g) traefik.domain.com to point to your Manager Node IP Address

At this point you shall be able to access the Traefik Web Management UI

Getting the client IP
If you want to read the client IP in your applications/stacks using the X-Forwarded-For or X-Real-IP headers provided by Traefik, you need to make Traefik listen directly, not through Docker Swarm mode, even while being deployed with Docker Swarm mode.

For that, you need to publish the ports using “host” mode.

So we will go through the following: instead of the file used in the single server mode:

root@node1:~# cd frappe_docker/
root@node1:~/frappe_docker# curl -L dockerswarm.rocks/traefik-host.yml -o traefik-host.yml
root@node1:~/frappe_docker# docker stack deploy -c traefik-host.yml traefik

Check it:
root@node1:~/frappe_docker# docker stack ps traefik

ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
756987jzitzm traefik_traefik.1 traefik:v2.2 node1.domain.com Running Running 3 days ago :443->443/tcp,:443->443/tcp,:80->80/tcp,:80->80/tcp

2- Follow dockerswarm.rocks guide to setup Portainer

3- Deploy mariadb:

1- Create Config
Configs > Add Config > frappe-mariadb-config

Paste the following in the config:
[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4

Create Secret
Secret > Add Secret > frappe-mariadb-root-password
<MARIADBPASSWORDHERE>

I used this to deploy mariadb stack:

=======================
version: “3.7”

services:
  mariadb-master:
image: 'bitnami/mariadb:10.3'
deploy:
  restart_policy:
    condition: on-failure
configs:
  - source: frappe-mariadb-config
    target: /opt/bitnami/mariadb/conf/bitnami/my_custom.cnf
networks:
  - frappe-network
secrets:
  - frappe-mariadb-root-password
volumes:
  - 'mariadb_master_data:/bitnami/mariadb'
environment:
  - MARIADB_REPLICATION_MODE=master
  - MARIADB_REPLICATION_USER=repl_user
  - MARIADB_REPLICATION_PASSWORD_FILE=/run/secrets/frappe-mariadb-root-password
  - MARIADB_ROOT_PASSWORD_FILE=/run/secrets/frappe-mariadb-root-password

  mariadb-slave:
image: 'bitnami/mariadb:10.3'
deploy:
  restart_policy:
    condition: on-failure
configs:
  - source: frappe-mariadb-config
    target: /opt/bitnami/mariadb/conf/bitnami/my_custom.cnf
networks:
  - frappe-network
secrets:
  - frappe-mariadb-root-password
volumes:
  - 'mariadb_slave_data:/bitnami/mariadb'
environment:
  - MARIADB_REPLICATION_MODE=slave
  - MARIADB_REPLICATION_USER=repl_user
  - MARIADB_REPLICATION_PASSWORD_FILE=/run/secrets/frappe-mariadb-root-password
  - MARIADB_MASTER_HOST=mariadb-master
  - MARIADB_MASTER_PORT_NUMBER=3306
  - MARIADB_MASTER_ROOT_PASSWORD_FILE=/run/secrets/frappe-mariadb-root-password

volumes:
  mariadb_master_data:
  mariadb_slave_data:

configs:
  frappe-mariadb-config:
external: true

secrets:
  frappe-mariadb-root-password:
external: true 
networks:
  frappe-network:
    name: frappe-network
    attachable: true

=======================

3- Deploy Frappe/ERPNext
Stacks > Add Stacks > frappe-bench-v13

version: "3.7"

services:
  redis-cache:
image: redis:latest
volumes:
  - redis-cache-vol:/data
deploy:
  restart_policy:
    condition: on-failure
networks:
  - frappe-network

  redis-queue:
image: redis:latest
volumes:
  - redis-queue-vol:/data
deploy:
  restart_policy:
    condition: on-failure
networks:
  - frappe-network

  redis-socketio:
image: redis:latest
volumes:
  - redis-socketio-vol:/data
deploy:
  restart_policy:
    condition: on-failure
networks:
  - frappe-network
  
  erpnext-nginx:
image: frappe/erpnext-nginx:${ERPNEXT_VERSION?Variable ERPNEXT_VERSION not set}
environment:
  - UPSTREAM_REAL_IP_ADDRESS=10.0.0.0/8
  - FRAPPE_PY=erpnext-python
  - FRAPPE_PY_PORT=8000
  - FRAPPE_SOCKETIO=frappe-socketio
  - SOCKETIO_PORT=9000
volumes:
  - sites-vol:/var/www/html/sites:rw
  - assets-vol:/assets:rw
networks:
  - frappe-network
  - traefik-public
deploy:
  restart_policy:
    condition: on-failure
  labels:
    - "traefik.docker.network=traefik-public"
    - "traefik.enable=true"
    - "traefik.constraint-label=traefik-public"
    - "traefik.http.routers.erpnext-nginx.rule=Host(${SITES?Variable SITES not set})"
    - "traefik.http.routers.erpnext-nginx.entrypoints=http"
    - "traefik.http.routers.erpnext-nginx.middlewares=https-redirect"
    - "traefik.http.routers.erpnext-nginx-https.rule=Host(${SITES?Variable SITES not set})"
    - "traefik.http.routers.erpnext-nginx-https.entrypoints=https"
    - "traefik.http.routers.erpnext-nginx-https.tls=true"
    - "traefik.http.routers.erpnext-nginx-https.tls.certresolver=le"
    - "traefik.http.services.erpnext-nginx.loadbalancer.server.port=8080"

  erpnext-python:
image: frappe/erpnext-worker:${ERPNEXT_VERSION?Variable ERPNEXT_VERSION not set}
deploy:
  restart_policy:
    condition: on-failure
environment:
  - MARIADB_HOST=${MARIADB_HOST?Variable MARIADB_HOST not set}
  - REDIS_CACHE=redis-cache:6379
  - REDIS_QUEUE=redis-queue:6379
  - REDIS_SOCKETIO=redis-socketio:6379
  - SOCKETIO_PORT=9000
  - AUTO_MIGRATE=1
volumes:
  - sites-vol:/home/frappe/frappe-bench/sites:rw
  - assets-vol:/home/frappe/frappe-bench/sites/assets:rw
networks:
  - frappe-network

  frappe-socketio:
image: frappe/frappe-socketio:${FRAPPE_VERSION?Variable FRAPPE_VERSION not set}
deploy:
  restart_policy:
    condition: on-failure
volumes:
  - sites-vol:/home/frappe/frappe-bench/sites:rw
  - logs-vol:/home/frappe/frappe-bench/logs:rw
networks:
  - frappe-network

  erpnext-worker-default:
image: frappe/erpnext-worker:${ERPNEXT_VERSION?Variable ERPNEXT_VERSION not set}
deploy:
  restart_policy:
    condition: on-failure
command: worker
volumes:
  - sites-vol:/home/frappe/frappe-bench/sites:rw
  - logs-vol:/home/frappe/frappe-bench/logs:rw
networks:
  - frappe-network

  erpnext-worker-short:
image: frappe/erpnext-worker:${ERPNEXT_VERSION?Variable ERPNEXT_VERSION not set}
deploy:
  restart_policy:
    condition: on-failure
command: worker
environment:
  - WORKER_TYPE=short
volumes:
  - sites-vol:/home/frappe/frappe-bench/sites:rw
  - logs-vol:/home/frappe/frappe-bench/logs:rw
networks:
  - frappe-network

  erpnext-worker-long:
image: frappe/erpnext-worker:${ERPNEXT_VERSION?Variable ERPNEXT_VERSION not set}
deploy:
  restart_policy:
    condition: on-failure
command: worker
environment:
  - WORKER_TYPE=long
volumes:
  - sites-vol:/home/frappe/frappe-bench/sites:rw
networks:
  - frappe-network

  frappe-schedule:
image: frappe/erpnext-worker:${ERPNEXT_VERSION?Variable ERPNEXT_VERSION not set}
deploy:
  restart_policy:
    condition: on-failure
command: schedule
volumes:
  - sites-vol:/home/frappe/frappe-bench/sites:rw
  - logs-vol:/home/frappe/frappe-bench/logs:rw
      
networks:
  - frappe-network

#  fix-vol-permissions:
#    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
#    user: root
#    command: chown -R 1000:1000 /sites /assets /logs
#    volumes:
#      - sites-vol:/sites
#      - assets-vol:/assets
#      - logs-vol:/logs

volumes:
  redis-cache-vol:
  redis-queue-vol:
  redis-socketio-vol:
  assets-vol:
  sites-vol:
  logs-vol:
  
networks:
  traefik-public:
external: true
  frappe-network:
external: true

=======================

PS:
Before deploying set the environment variables:

ERPNEXT_VERSION v13
FRAPPE_VERSION v13
MARIADB_HOST frappe-mariadb_mariadb-master
SITES site1.domain.com,site2.domain.com,site1.otherdomain.com

Dont add this here INSTALL_APPS= erpnext

4- Create the sites (one by one)

Containers > Add Container > add-site-domain-com

Always pull the image
Access Control: Disabled
Node: node1
Set command as new
Select Volume frappe-bench-v13_sites_vol and mount in container /home/frappe/frappe-bench/sites
Select network frappe-network

Set Env variables:
MYSQL_ROOT_PASSWORD ROOTPASSWORDHERE
SITE_NAME site1.domain.com
INSTALL_APPS erpnext
ADMIN_PASSWORD ADMINPASSWORDHERE
Choose Restart Policy None

Start container

Repeat this step for each site.

Hope this will help.

efleurimond · April 12, 2022, 10:39pm

Hi @Ysedky, thank you for providing such helpful details.

It took me a while to provide feedback because I had some issues getting everything to work because I kept getting a Node related error message when trying to use versions higher than 13.22.0 (I tried all of the above until to v13.25.2…same error). Eventually I decided to stick with v13.20.1 which works ok for now. I have a configuration with Traefik and Portainer using both basicauth, ipwhitelist and ratelimit middleware; I decided not to use replication on Mariadb. Thanks again!

revant_one · April 12, 2022, 11:12pm

yesterday I installed ERPNext v13.25.2 using this gist ERPNext on Docker Swarm · GitHub

efleurimond · April 12, 2022, 11:17pm

Interesting! Will try it soon. Thanks @revant_one

system · April 26, 2022, 11:17pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.