Socket IO not working in Production Ubuntu 18.04

I have setup ERPNext with easy install script. Activated dns multi tenant setup with letsencrypt.
erpnext 12.8.0
frappe 12.5.1

Every thing is working fine except the socketIO. when I try to use frappe.publish_realtime. No action takes place.

e.g frappe.publish_realtime(“msgprint”,message,user)

Other functionality like who is viewing the document, auto refresh of list view. Nothing is working that depends on socketio

Browser console is also not showing any connectivity issue.

I have checked the auto generated nginx config and they look fine. Please guide me in this matter. Below is node-socketio.log

Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL     routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

No response for doc_subscribe
No response for doc_subscribe
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

No response for doc_subscribe
No response for doc_subscribe
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

No response for doc_subscribe
No response for doc_subscribe
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

No response for doc_subscribe
No response for doc_subscribe
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
Unable to join chat room. Error: write EPROTO 140022498355008:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
listening on *: 9000
Unable to join chat room. Error: write EPROTO 140148206663488:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

Unable to join chat room. Error: write EPROTO 140148206663488:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
Unable to join chat room. Error: write EPROTO 140148206663488:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

frappe.chat: Subscribing ahmad@havenir.com to room CR00001
frappe.chat: Subscribing ahmad@havenir.com to event srca.havenir.com:room:CR00001
listening on *: 9000
Unable to join chat room. Error: write EPROTO 139758533908288:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

I have the same problem with one of my two instances of ERPNext. They both have the exact same setup (Ubuntu 18.04, latest ERPNext v12 and Frappe v12, …), the only difference is that the instance that is having the problem uses a self-signed ssl certificate and is accessed via https, while the other isn’t.

So I think the culprit here is SSL.

A solution to this problem was suggested her, however it didn’t work for me:

Self-signed certificates must be trusted on client devices by installing into client’s certificate store.

I already did this and my browser also shows erpnext as secure connection.
However, I didn’t create a certificate authority to sign the certificates and bench shows a warning that ssl stapling will be ignored since there is no authority mentioned in the certificate, so maybe that’s the problem.

This is what OpenSSL shows when connecting to the site:

CONNECTED(00000005)
depth=0 C = DE, L = example-city, O = example-company, CN = site1.local
verify error:num=18:self signed certificate
verify return:1
depth=0 C = DE, L = example-city, O = example-company, CN = site1.local
verify return:1
---
Certificate chain
0 s:C = DE, L = example-city, O = example-company, CN = site1.local
i:C = DE, L = example-city, O = example-company, CN = site1.local
---
Server certificate
-----BEGIN CERTIFICATE-----
xyz
-----END CERTIFICATE-----
subject=C = DE, L = example-city, O = example-company, CN = site1.local

issuer=C = DE, L = example-city, O = example-company, CN = site1.local

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 1631 bytes and written 801 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: xyz.....
    Session-ID-ctx:
    Resumption PSK: xyz.....
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000...
    0010...

    Start Time: 1590481036
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: xyz....
    Session-ID-ctx:
    Resumption PSK: xyz.....
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000...
    0010...

    Start Time: 1590481036
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

This issue is with the this command:
sudo -H bench setup lets-encrypt [site-name]
If alters the nginx config in the wrong manner and breaks the socket io connectivity.

I have reported this issue.
To fix this issue there is a work around.

  1. Goto site_config.json remove the ssl certifactes
  2. Convert the site back to DNS based multitenancy
  3. Re generate nginx config bench setup nginx
  4. Install certbot
  5. setup ssl with certbot --nginx
  6. Allow certbot to make changes to redirect http to https

This will fix the socketio connectivity

2 Likes

Here’s what worked for me. Node JS v8 is what gets installed with easy install script. I just upgraded to v12 (latest stable) and the SSL error went away.

It still doesn’t work for me.
NodeJS: 12.18.3
ERPNext: v12.6.0
Frappe Framework: v12.4.1

Even commenting or setting ssl_ecdh_curve to auto doesn’t help. Even tried with previously supported set of cipher suites.

It works perfectly without SSL though.

Its worth noting that I do see below events in websocket request in chrome developer tools:

Below is the current nginx config:

upstream frappe-bench-frappe {
	server 127.0.0.1:8001 fail_timeout=0;
}

upstream frappe-bench-socketio-server {
	server 127.0.0.1:9001 fail_timeout=0;
}

# setup maps

map $host $site_name_jlsumov {
	mysite.mydomain.com mysite;
	# default $host;
	
}

# server blocks

server {
	
	listen 80;	

	server_name
		mysite
		;

	root /home/frappe/frappe-bench/sites;

	add_header X-Frame-Options "SAMEORIGIN";
	add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
	add_header X-Content-Type-Options nosniff;
	add_header X-XSS-Protection "1; mode=block";

	location /assets {
		try_files $uri =404;
	}

	location ~ ^/protected/(.*) {
		internal;
		try_files /$site_name_jlsumov/$1 =404;
	}

	location /socket.io {
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header X-Frappe-Site-Name $site_name_jlsumov;
		proxy_set_header Origin $scheme://$http_host;
		proxy_set_header Host $host;

		proxy_pass http://frappe-bench-socketio-server;
	}

	location / {

 		rewrite ^(.+)/$ $1 permanent;
  		rewrite ^(.+)/index\.html$ $1 permanent;
  		rewrite ^(.+)\.html$ $1 permanent;

		location ~ ^/files/.*.(htm|html|svg|xml) {
			add_header Content-disposition "attachment";
			try_files /$site_name_jlsumov/public/$uri @webserver;
		}

		try_files /$site_name_jlsumov/public/$uri @webserver;
	}

	location @webserver {
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Frappe-Site-Name $site_name_jlsumov;
		proxy_set_header Host $host;
		proxy_set_header X-Use-X-Accel-Redirect True;
		proxy_read_timeout 300;
		proxy_redirect off;

		proxy_pass  http://frappe-bench-frappe;
	}

	# error pages
	error_page 502 /502.html;
	location /502.html {
		root /home/frappe/.bench/bench/config/templates;
		internal;
	}

	# optimizations
	sendfile on;
	keepalive_timeout 15;
	client_max_body_size 50m;
	client_body_buffer_size 16K;
	client_header_buffer_size 1k;

	# enable gzip compresion
	# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
	gzip on;
	gzip_http_version 1.1;
	gzip_comp_level 5;
	gzip_min_length 256;
	gzip_proxied any;
	gzip_vary on;
	gzip_types
		application/atom+xml
		application/javascript
		application/json
		application/rss+xml
		application/vnd.ms-fontobject
		application/x-font-ttf
		application/font-woff
		application/x-web-app-manifest+json
		application/xhtml+xml
		application/xml
		font/opentype
		image/svg+xml
		image/x-icon
		text/css
		text/plain
		text/x-component
		;
		# text/html is always compressed by HttpGzipModule
}


	
server {
	
	listen 443 ssl;
	

	server_name
		mysite.mydomain.com
		;

	root /home/frappe/frappe-bench/sites;

	

	
	ssl on;
	ssl_certificate      /etc/letsencrypt/live/mysite.mydomain.com/fullchain.pem;
	ssl_certificate_key  /etc/letsencrypt/live/mysite.mydomain.com/privkey.pem;
	ssl_session_timeout  5m;
	# ssl_session_cache shared:SSL:10m;
	ssl_session_tickets off;
	ssl_stapling on;
	ssl_stapling_verify on;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
	ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
	# ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
	# ssl_ecdh_curve auto;
	ssl_prefer_server_ciphers on;
	

	add_header X-Frame-Options "SAMEORIGIN";
	add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
	add_header X-Content-Type-Options nosniff;
	add_header X-XSS-Protection "1; mode=block";

	location /assets {
		try_files $uri =404;
	}

	location ~ ^/protected/(.*) {
		internal;
		try_files /$site_name_jlsumov/$1 =404;
	}

	location /socket.io {
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header X-Frappe-Site-Name $site_name_jlsumov;
		proxy_set_header Origin $scheme://$http_host;
		proxy_set_header Host $host;

		proxy_pass http://frappe-bench-socketio-server;
	}

	location / {

 		rewrite ^(.+)/$ $1 permanent;
  		rewrite ^(.+)/index\.html$ $1 permanent;
  		rewrite ^(.+)\.html$ $1 permanent;

		location ~ ^/files/.*.(htm|html|svg|xml) {
			add_header Content-disposition "attachment";
			try_files /$site_name_jlsumov/public/$uri @webserver;
		}

		try_files /$site_name_jlsumov/public/$uri @webserver;
	}

	location @webserver {
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Frappe-Site-Name $site_name_jlsumov;
		proxy_set_header Host $host;
		proxy_set_header X-Use-X-Accel-Redirect True;
		proxy_read_timeout 300;
		proxy_redirect off;

		proxy_pass  http://frappe-bench-frappe;
	}

	# error pages
	error_page 502 /502.html;
	location /502.html {
		root /home/frappe/.bench/bench/config/templates;
		internal;
	}

	# optimizations
	sendfile on;
	keepalive_timeout 15;
	client_max_body_size 50m;
	client_body_buffer_size 16K;
	client_header_buffer_size 1k;

	# enable gzip compresion
	# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
	gzip on;
	gzip_http_version 1.1;
	gzip_comp_level 5;
	gzip_min_length 256;
	gzip_proxied any;
	gzip_vary on;
	gzip_types
		application/atom+xml
		application/javascript
		application/json
		application/rss+xml
		application/vnd.ms-fontobject
		application/x-font-ttf
		application/font-woff
		application/x-web-app-manifest+json
		application/xhtml+xml
		application/xml
		font/opentype
		image/svg+xml
		image/x-icon
		text/css
		text/plain
		text/x-component
		;
		# text/html is always compressed by HttpGzipModule
}

# http to https redirect
	server {
		listen 80;
		server_name
			mysite.mydomain.com
			;

		return 301 https://$host$request_uri;
	}

Try adding the following line at the top of the frappe/socketio.js file:
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

Be careful though, because it is not a good practice.

Reference:

Here is my experience with this SocketIO issue:

First I found an error looking similar to

error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:802:

in my “/frappe-bench/logs/node-socketio.log” file.

Turns out this exact error is thrown due to a bug in node v8.

So, I started looking around for what my node version is with “node -v”. It showed the version as v14. But, I was still getting this error.

Here is my solution and what I found:

It turns out that my “frappe-bench/config/supervisor.conf” file was using “/usr/bin/node” to start the “-node-socketio” process. But, Since I had nvm installed to manage my node versions, the node version used in the terminal was installed locally and “/usr/bin/node” was left at v8.

This issue was resolved for me after updating the node in “/usr/bin/node” and restarting the supervisor.

2 Likes