Socket Failures

[fiveoaks]

I’m running ERPNext and Raven in a docker container and have noticed an issue with Socket.IO. When directly using Raven (https://example.com/raven) I see it calling the web socket via an https:// request and it’s successful.

However when ERPNext calls socket, it is using wss:// and these requests are failing:

Socket.IO is running and since the https:// request works, that seems to confirm it.

I’m questioning the Nginx configuration file:

upstream frappe-frappe {
        server 127.0.0.1:8000 fail_timeout=0;
}

upstream frappe-socketio-server {
        server 127.0.0.1:9000 fail_timeout=0;
}

# setup maps
# server blocks

server {
        listen 80;
        listen [::]:80;

        server_name
                default
                ;

        root /work/frappe/sites;

        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

        add_header X-Frame-Options "SAMEORIGIN";
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "same-origin, strict-origin-when-cross-origin";

        location /assets {
                try_files $uri =404;
                add_header Cache-Control "max-age=31536000";
        }

        location ~ ^/protected/(.*) {
                internal;
                try_files /default/$1 =404;
        }

        location /socket.io {
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header X-Frappe-Site-Name default;
                proxy_set_header Origin $scheme://$http_host;
                proxy_set_header Host $host;

                proxy_pass http://frappe-socketio-server;
        }

        location / {

                rewrite ^(.+)/$ $1 permanent;
                rewrite ^(.+)/index\.html$ $1 permanent;
                rewrite ^(.+)\.html$ $1 permanent;

                location ~* ^/files/.*.(htm|html|svg|xml) {
                        add_header Content-disposition "attachment";
                        try_files /default/public/$uri @webserver;
                }

                try_files /default/public/$uri @webserver;
        }

        location @webserver {
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Frappe-Site-Name default;
                proxy_set_header Host $host;
                proxy_set_header X-Use-X-Accel-Redirect True;
                proxy_read_timeout 120;
                proxy_redirect off;
                proxy_pass  http://frappe-frappe;
        }

        # error pages
        error_page 502 /502.html;
        location /502.html {
                root /usr/local/lib/python3.10/dist-packages/bench/config/templates;
                internal;
        }

        # optimizations
        sendfile on;
        keepalive_timeout 15;
        client_max_body_size 50m;
        client_body_buffer_size 16K;
        client_header_buffer_size 1k;

        # enable gzip compresion
        # based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
        gzip on;
        gzip_http_version 1.1;
        gzip_comp_level 5;
        gzip_min_length 256;
        gzip_proxied any;
        gzip_vary on;
        gzip_types
                application/atom+xml
                application/javascript
                application/json
                application/rss+xml
                application/vnd.ms-fontobject
                application/x-font-ttf
                application/font-woff
                application/x-web-app-manifest+json
                application/xhtml+xml
                application/xml
                font/opentype
                image/svg+xml
                image/x-icon
                text/css
                text/plain
                text/x-component
                ;
                # text/html is always compressed by HttpGzipModule
}

Anyone have any suggestions. I’m pulling my hair out a little bit on this one.

[fiveoaks]

After a lot of debugging, I was able to solve it. It came down to the Apache configuration. I have Apache in front of docker / nginx. I changed my Apache configuration (adding the rewrite rules) to the below and the socket via wss:// started working.

FYI, on my docker container I have port 9070 mapped to 8000 and 9071 mapped to 9000.

<VirtualHost *:443>
    ServerName frappe.example.com

    RewriteEngine On
    RewriteCond %{QUERY_STRING} transport=polling       [NC]
    RewriteRule /socket.io/(.*)           http://127.0.0.1:9070/socket.io/$1 [P]
    RewriteCond %{HTTP:Upgrade} websocket               [NC]
    RewriteRule /socket.io/(.*)           ws://127.0.0.1:9071/socket.io/$1  [P]

    ProxyPass / http://127.0.0.1:9070/
    ProxyPassReverse / http://127.0.0.1:9070/
    ProxyPassReverse / https://frappe.example.com/
    ProxyRequests Off
    ProxyPreserveHost On

    SSLEngine On
    SSLCertificateFile /path/to/certificate.cer
    SSLCertificateKeyFile /path/to/key.key
    SSLCertificateChainFile /path/to/fullchain.cer
</VirtualHost>

[murtaza.ghadiali]

Hello @fiveoaks I am also doing the same setup for Apache, but it didn’t work for me can you please help me?

  ServerName frappe.example.com
    ProxyPreserveHost On



    ProxyPass / http://localhost:9080/
    ProxyPassReverse / http://localhost:9080/

 # Proxy WebSocket connections
RewriteEngine On
    RewriteCond %{QUERY_STRING} transport=polling       [NC]
    RewriteRule /socket.io/(.*)           http://localhost:10090/socket.io/$1 [P]
    RewriteCond %{HTTP:Upgrade} websocket               [NC]
    RewriteRule /socket.io/(.*)           ws://localhost:10090/socket.io/$1  [P]




SSLEngine On
    SSLCertificateFile /path/to/certificate.cer
    SSLCertificateKeyFile /path/to/key.key
    SSLCertificateChainFile /path/to/fullchain.cer

[fiveoaks]

What didn’t work exactly? Is Apache failing to start with the config file? Or are you seeing errors in the browser? Can you provide screenshots.

[murtaza.ghadiali]

I see error in browser

[fiveoaks]

Can you confirm Redis is running and listening on port 10090? A simple test from the host machine running the docker container is running telnet:

telnet 127.0.0.1 10090

Does it connect or fail to?

[murtaza.ghadiali]

@fiveoaks yes it’s running

[fiveoaks]

Is the SSL certificate real or self-signed?

[fiveoaks]

I just thought of what else it could be. Do you have the proxy_wstunnel module enabled in Apache2? That is required for this to work.