Socket.io connection refused!

Install / Update Deployment
1

Hello.

I have a problem with socket.io not accepting connections.
I have setup my ERPNext manually in a VPS, Almost everything works well except for socket.io which gives 502 error in the browser network tab like this:

GET /socket.io/?EIO=3&transport=polling&t=NrPeAQm HTTP/3
Host: example.com

and the response is like:

HTTP/3 502 Bad Gateway
content-type: text/html; charset=UTF-8
server: cloudflare

Here’s an example of repetitive error message in /var/logs/nginx/error.log:

2021/11/26 00:43:26 [error] 718#718: *243 connect() failed (111: Connection refused) while connecting to upstream, client: 172.70.162.26, server: example.com, request: "GET /socket.io/?EIO=3&transport=polling&t=NrPid6p HTTP/1.1", upstream: "http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=NrPid6p", host: "example.com",referrer: "https://example.com/app"

And here’s sudo supervisorctl status all output

frappe-bench-redis:frappe-bench-redis-cache                 RUNNING   pid 820, uptime 0:41:41
frappe-bench-redis:frappe-bench-redis-queue                 RUNNING   pid 821, uptime 0:41:41
frappe-bench-redis:frappe-bench-redis-socketio              RUNNING   pid 822, uptime 0:41:41
frappe-bench-web:frappe-bench-frappe-web                    RUNNING   pid 823, uptime 0:41:41
frappe-bench-workers:frappe-bench-frappe-default-worker-0   RUNNING   pid 825, uptime 0:41:41
frappe-bench-workers:frappe-bench-frappe-long-worker-0      RUNNING   pid 827, uptime 0:41:41
frappe-bench-workers:frappe-bench-frappe-schedule           RUNNING   pid 824, uptime 0:41:41
frappe-bench-workers:frappe-bench-frappe-short-worker-0     RUNNING   pid 826, uptime 0:41:41
1 Like
2

It looks like you’re missing node-socketio process.
frappe-bench-web:frappe-bench-node-socketio RUNNING pid 242199, uptime 2 days, 3:57:45

Did you run bench setup socketio ?

3

Just ran it again , but no luck !

Supervisor:

frappe-bench-redis:frappe-bench-redis-cache                 RUNNING   pid 258090, uptime 0:00:07
frappe-bench-redis:frappe-bench-redis-queue                 RUNNING   pid 258091, uptime 0:00:07
frappe-bench-redis:frappe-bench-redis-socketio              RUNNING   pid 258092, uptime 0:00:07
frappe-bench-web:frappe-bench-frappe-web                    RUNNING   pid 258093, uptime 0:00:07
frappe-bench-workers:frappe-bench-frappe-default-worker-0   RUNNING   pid 258095, uptime 0:00:07
frappe-bench-workers:frappe-bench-frappe-long-worker-0      RUNNING   pid 258097, uptime 0:00:07
frappe-bench-workers:frappe-bench-frappe-schedule           RUNNING   pid 258094, uptime 0:00:07
frappe-bench-workers:frappe-bench-frappe-short-worker-0     RUNNING   pid 258096, uptime 0:00:07

And nginx logs:

2021/11/27 00:28:40 [error] 718#718: *15198 connect() failed (111: Connection refused) while connecting to upstream, client: x.x.x.x server: example.com, request: "GET /socket.io/?EIO=3&transport=polling&t=NrUoqST HTTP/1.1", upstream: "http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=NrUoqST", host: "example.com", referrer: "https://example.com/app/"
4

Trying restarting the VPS. I know every time I install ERPNext on a new server socketio is always giving me issues. After trying a new config then running sudo reboot It magically worked for me.

5

I restarted but still nothing changed!
I already did the following (not in order) :

  • bench setup socketio
  • npm install -g socket.io
  • bench update --reset followed by bench migrate
  • bench setup requirements --node
  • bench setup production <username>
  • bench restart
  • Restarted the server several times either with sudo reboot or systemctl reboot.

But that socket.io error still exist (The rest of website works fine though).

Here’s my current sudo supervisorctl status all output:

frappe-bench-redis:frappe-bench-redis-cache                 RUNNING   pid 880, uptime 0:21:18
frappe-bench-redis:frappe-bench-redis-queue                 RUNNING   pid 881, uptime 0:21:18
frappe-bench-redis:frappe-bench-redis-socketio              RUNNING   pid 882, uptime 0:21:18
frappe-bench-web:frappe-bench-frappe-web                    RUNNING   pid 883, uptime 0:21:18
frappe-bench-workers:frappe-bench-frappe-default-worker-0   RUNNING   pid 885, uptime 0:21:18
frappe-bench-workers:frappe-bench-frappe-long-worker-0      RUNNING   pid 887, uptime 0:21:18
frappe-bench-workers:frappe-bench-frappe-schedule           RUNNING   pid 884, uptime 0:21:18
frappe-bench-workers:frappe-bench-frappe-short-worker-0     RUNNING   pid 886, uptime 0:21:18
1 Like
6

Did you try sudo bench setup production?

7

Yes, once with sudo and the other without

8

It may help to describe the manual setup that was used to install. Operating environment, versions of apps etc and steps taken to install. That may provide clues.

9

make sure your supervisor.conf has the entry for the socketio process to run, something like:

[program:frappe-bench-node-socketio]
command=/usr/bin/node /home/revant/frappe-bench/apps/frappe/socketio.js
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/revant/frappe-bench/logs/node-socketio.log
stderr_logfile=/home/revant/frappe-bench/logs/node-socketio.error.log
user=revant
directory=/home/revant/frappe-bench

supervisorctl status it is missing the entry for frappe-bench-node-socketio, frappe-bench-web: should have one more process for socketio.

Run the command bench setup supervisor to setup the supervisor.conf.

6 Likes
10

Thanks so much @revant_one , POST requests are working now but not GET unfortunately, It doesn’t even get initialized by the browser idk why!! Anyway, I’m happy about the current setup and I’ve already documented all of my manual setup instructions from the beginning, and I’m going to post them here in the forum.

11

we facing the following error even socketio entry in supervisor.conf file

*395972 connect() failed (111: Connection refused) while connecting to upstream, client: XX.XX.XX.XX, server: XXX.XXX.com, request: “GET /socket.io/?EIO=3&transport=polling&t=O4gtUEi HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=O4gtUEi”, host: “XXX.XXXX.com”, referrer: “https://XX.XXXX.com/desk”

please help us

12

Answer by @revant_one is the solution. However, additionally make sure that:

  • supervisor is reloaded with new config file –
    service supervisor restart
    bench restart

in my case even after the config file was showing the [program:frappe-bench-node-socketio]
but bench restart was never listing the frappe-bench-web:frappe-bench-node-socketio: (for any status, as revant_one was also wondering!), this pointed me to realize supervisorctl is not reading the updated config file.

@revant_one ideally isn’t bench restart be doing service supervisor restart internally? I know this could spoil the redis cache. But no point restarting bench with old config. Or alternatively there could be a switch say bench restart --refresh-supervisor-config.

1 Like