Unknown error with reverse proxy

I have ERPNext 13 installed on Ubuntu 20.04LTS in a virtual machine. I access the system via a nginx reverse proxy that handles the SSL certs for the domain.

While in general it works fine, I do get regular but random error messages on the clients. The proxy shows the error message then for a few minutes (or longer) and it then suddenly works again, without any intervention. But it is quite annoying, as it’s impossible to predict when and for how long the access is failing.

When checking the nginx error log on the proxy, it is full of these error messages:

2023/12/14 11:39:37 [error] 1874709#1874709: *300408 connect() failed (111: Unknown error) while connecting to upstream, client: 42.200.x.x, server: www.domain.com, request: "GET /socket.io/?EIO=3&transport=polling&t=OncQ8th HTTP/2.0", upstream: "http://192.168.3.8:80/socket.io/?EIO=3&transport=polling&t=OncQ8th", host: "www.xtc-domain.com", referrer: "https://www.domain.com/app/batch/10065"

When I access the system directly via the IP address is works just fine. Where on the ERPNext system can I find more details as to why it is creating that unkown error? What log files could I check for that?

My nginx reverse proxy config is as follows:

server {
    server_name www.domain.com;
    location / {
        proxy_pass http://192.168.3.8:80;
        proxy_redirect off;
        proxy_set_header Range $http_range;
        proxy_set_header If-Range $http_if_range;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        client_max_body_size 512M;
    }
    access_log /var/log/nginx/www.domain.com-access.log;
    error_log /var/log/nginx/www.domain.com-error.log;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/www.domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.domain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
    if ($host = www.domain.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name www.domain.com;

    listen 80;
    return 404; # managed by Certbot
}

Any hints highly appreciated.

So it seems that fail2ban was the culprit blocking access to the ERPNext system.

After I adjusted the jail.conf on the ERPNext system sudo nano /etc/fail2ban/jail.conf and added the local IP range to the ignoreip line, the errors disappeared.

I recently moved the ERPNext server and since it got the new local IP address, those errors kept popping up and started to really affect usage. Finally I seem to have found the issue.