Frappe docker behind Nginx (not dockerized) Connection reset by peer

[Mehmet_Ali_Anil]

Hi, I had encountered some issues in my 16.04 → 18.04 upgrade, and somehow broke my manual install.

Therefore, I decided that starting fresh with a dockerized install and migrating the files and database might be a good way to mode forward.

I am trying to bring up frappe_docker, without traefik, and using my native nginx as a proxy to serve erp on my erp subdomain.

I went through the installation with these:

.env

LETSENCRYPT_EMAIL=email@domain.com
ERPNEXT_VERSION=version-12
FRAPPE_VERSION=version-12
MARIADB_HOST=mariadb
MYSQL_ROOT_PASSWORD=admin
SITE_NAME=sub.domain.com
SITES=`sub.domain.com`
DB_ROOT_USER=root
ADMIN_PASSWORD=admin
INSTALL_APPS=erpnext
ENTRYPOINT_LABEL=traefik.http.routers.erpnext-nginx.entrypoints=web
CERT_RESOLVER_LABEL=erpnext.local.no-cert-resolver
HTTPS_REDIRECT_RULE_LABEL=erpnext.local.no-redirect-rule
HTTPS_REDIRECT_ENTRYPOINT_LABEL=erpnext.local.no-entrypoint
HTTPS_REDIRECT_MIDDLEWARE_LABEL=erpnext.local.no-middleware
HTTPS_USE_REDIRECT_MIDDLEWARE_LABEL=erpnext.local-no-redirect-middleware
SKIP_NGINX_TEMPLATE_GENERATION=0
WORKER_CLASS=gthread

docker-compose.yml

version: "3"

services:
#  traefik:
#    image: "traefik:v2.2"
#    command:
#      - "--providers.docker=true"
#      - "--providers.docker.exposedbydefault=false"
#      - "--entrypoints.web.address=:80"
#      - "--entrypoints.websecure.address=:443"
#      - "--certificatesresolvers.myresolver.acme.httpchallenge=true"
#      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
#      - "--certificatesresolvers.myresolver.acme.email=${LETSENCRYPT_EMAIL}"
#      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
#    labels:
#      # enable traefik
#      - "traefik.enable=true"
#      # global redirect to https for production only
#      - "${HTTPS_REDIRECT_RULE_LABEL}"
#      - "${HTTPS_REDIRECT_ENTRYPOINT_LABEL}"
#      - "${HTTPS_REDIRECT_MIDDLEWARE_LABEL}"
#      # middleware redirect for production only
#      - "${HTTPS_USE_REDIRECT_MIDDLEWARE_LABEL}"
#    ports:
#      - "80:80"
#      - "443:443"
#    volumes:
#      - cert-vol:/letsencrypt
#      - /var/run/docker.sock:/var/run/docker.sock:ro
#    userns_mode: "host"

  erpnext-nginx:
    image: frappe/erpnext-nginx:${ERPNEXT_VERSION}
    restart: on-failure
    environment:
      - FRAPPE_PY=erpnext-python
      - FRAPPE_PY_PORT=8000
      - FRAPPE_SOCKETIO=frappe-socketio
      - SOCKETIO_PORT=9000
      - SKIP_NGINX_TEMPLATE_GENERATION=${SKIP_NGINX_TEMPLATE_GENERATION}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.erpnext-nginx.rule=Host(${SITES})"
      - "${ENTRYPOINT_LABEL}"
      - "${CERT_RESOLVER_LABEL}"
      - "traefik.http.services.erpnext-nginx.loadbalancer.server.port=80"
    volumes:
      - sites-vol:/var/www/html/sites:rw
      - assets-vol:/assets:rw
    ports:
      - 8080:80

  erpnext-python:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: on-failure
    environment:
      - MARIADB_HOST=${MARIADB_HOST}
      - REDIS_CACHE=redis-cache:6379
      - REDIS_QUEUE=redis-queue:6379
      - REDIS_SOCKETIO=redis-socketio:6379
      - SOCKETIO_PORT=9000
      - AUTO_MIGRATE=1
      - WORKER_CLASS=${WORKER_CLASS}
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - assets-vol:/home/frappe/frappe-bench/sites/assets:rw

  frappe-socketio:
    image: frappe/frappe-socketio:${FRAPPE_VERSION}
    restart: on-failure
    depends_on:
      - redis-socketio
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - logs-vol:/home/frappe/frappe-bench/logs:rw

  erpnext-worker-default:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: on-failure
    command: worker
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - logs-vol:/home/frappe/frappe-bench/logs:rw

  erpnext-worker-short:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: on-failure
    command: worker
    environment:
      - WORKER_TYPE=short
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - logs-vol:/home/frappe/frappe-bench/logs:rw

  erpnext-worker-long:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: on-failure
    command: worker
    environment:
      - WORKER_TYPE=long
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

  erpnext-schedule:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: on-failure
    command: schedule
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - logs-vol:/home/frappe/frappe-bench/logs:rw

  redis-cache:
    image: redis:latest
    restart: on-failure
    volumes:
      - redis-cache-vol:/data

  redis-queue:
    image: redis:latest
    restart: on-failure
    volumes:
      - redis-queue-vol:/data

  redis-socketio:
    image: redis:latest
    restart: on-failure
    volumes:
      - redis-socketio-vol:/data

  mariadb:
    image: mariadb:10.6
    restart: on-failure
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci
      - --skip-character-set-client-handshake
      - --skip-innodb-read-only-compressed # Temporary fix for MariaDB 10.6
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      # Sometimes db initialization takes longer than 10 seconds and site-creator goes away.
      # Frappe doesn't use CONVERT_TZ() function that requires time zone info, so we can just skip it.
      - MYSQL_INITDB_SKIP_TZINFO=1
    volumes:
      - mariadb-vol:/var/lib/mysql

  site-creator:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: "no"
    command: new
    depends_on:
      - erpnext-python
    environment:
      - SITE_NAME=${SITE_NAME}
      - DB_ROOT_USER=${DB_ROOT_USER}
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - INSTALL_APPS=${INSTALL_APPS}
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - logs-vol:/home/frappe/frappe-bench/logs:rw

volumes:
  mariadb-vol:
  redis-cache-vol:
  redis-queue-vol:
  redis-socketio-vol:
  assets-vol:
  sites-vol:
  cert-vol:
  logs-vol:

I want to have 8080 as my port exposed outside docker, and nginx forwarding to there. I am using this simple nginx config (not in nginx.conf but as a site-enabled)

sub.domain.com.conf

server {
        listen 443;
        server_name sub.domain.com;

        ssl on;
        ssl_certificate      /etc/letsencrypt/live/domain.com/fullchain.pem;
        ssl_certificate_key  /etc/letsencrypt/live/domain.com/privkey.pem;
        ssl_session_timeout  5m;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
        ssl_prefer_server_ciphers   on;


        location / {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Frappe-Site-Name $host;
                proxy_set_header Host $http_host;
                proxy_pass http://127.0.0.1:8080;
        }

}

# http to https redirect
server {
        listen 80;
        server_name sub.domain.com;
        return 301 https://$host$request_uri;
}

If I do this from the terminal:

frappe@server:~/frappe_docker$ curl -L http://localhost:8080
curl: (56) Recv failure: Connection reset by peer

If I do it from outside:

me@LAPTOP:~$ curl -L sub.domain.com
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>

With nginx logging access:

192.168.2.254 - - [17/Dec/2021:15:19:08 +0100] "GET / HTTP/2.0" 502 182 "-" "curl/7.58.0"                                                                                                                        
192.168.2.254 - - [17/Dec/2021:15:19:09 +0100] "GET / HTTP/1.1" 301 194 "-" "curl/7.58.0"

logging error:

2021/12/17 15:19:09 [error] 25328#25328: *6521 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.254, server: sub.domain.com, request: "GET / HTTP/2.0

And my docker ps:

$ docker ps

CONTAINER ID   IMAGE                               COMMAND                  CREATED          STATUS                           PORTS                                             NAMES
5b7a8d718da6   frappe/erpnext-worker:version-12    "docker-entrypoint.s…"   28 minutes ago   Up 27 minutes                                                                      erp_erpnext-worker-default_1
3d049e40abfe   frappe/erpnext-worker:version-12    "docker-entrypoint.s…"   28 minutes ago   Up 27 minutes                                                                      erp_erpnext-schedule_1
b2752982537a   frappe/erpnext-worker:version-12    "docker-entrypoint.s…"   28 minutes ago   Up 27 minutes                                                                      erp_erpnext-worker-short_1
db884e4e03af   frappe/erpnext-worker:version-12    "docker-entrypoint.s…"   28 minutes ago   Up 27 minutes                                                                      erp_erpnext-worker-long_1
70b625a7bae3   frappe/frappe-socketio:version-12   "docker-entrypoint.s…"   28 minutes ago   Up 27 minutes                                                                      erp_frappe-socketio_1
4bb094a50436   redis:latest                        "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes                    6379/tcp                                          erp_redis-queue_1
6a3a62d87a81   frappe/erpnext-nginx:version-12     "/docker-entrypoint.…"   28 minutes ago   Up 27 minutes                    8080/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp   erp_erpnext-nginx_1
ae8aad8bffe7   frappe/erpnext-worker:version-12    "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes                                                                      erp_erpnext-python_1
3d9728679ff1   redis:latest                        "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes                    6379/tcp                                          erp_redis-cache_1
64952fe2f1a9   mariadb:10.6                        "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes                    3306/tcp                                          erp_mariadb_1
b702951609bb   redis:latest                        "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes                    6379/    tcp                                          erp_redis-socketio_1

... and other containers

My netstat:

netstat

frappe@server:~/frappe_docker$ sudo netstat -lnp | grep 8080
    tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      471/docker-proxy
    tcp6       0      0 :::8080                 :::*                    LISTEN      476/docker-proxy

What can I try?

[Mehmet_Ali_Anil]

Hey can I page you, @revant_one for this?
It seems that you have been closing in on these kind of issues lately.

[Mehmet_Ali_Anil]

Now I see it, the internal port that is used is 8080, not 80, therefore
ports:
-“8080:8080”

works for me.
The requests come to native nginx from 443 or 80, nginx relays them to 8080, which the container listens to. The app in the container though, does not listen to port 80 (as I thought a webserver would, albeit containerized) but uses 8080.

I realized this by actually installing netstat by opening an interactive terminal session in the container.