Bench Start not working Socket.io starts and gets stopped

Install / Update
1

Hello everyone,

I have recently setup erpnext on my amazon machine (development setup). I am actually unable to do a bench start, Seems like redis socket.io issue but unable to do the fix. Below are the logs

I have already done bench update and bench setup procfile. bench serve is working fine.
I am sure this is redis issue, since logs shows that the socket.io starts and eventually stops.

12:42:02 system | redis_queue.1 started (pid=14993)
12:42:02 redis_queue.1 | [15003] 26 Jul 12:42:02.763 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984.
12:42:02 redis_queue.1 | .
12:42:02 redis_queue.1 | .-__ ''-._ 12:42:02 redis_queue.1 | _.- . . ‘’-._ Redis 2.8.4 (00000000/0) 64 bit
12:42:02 redis_queue.1 | .- .-```. ```\/ _.,_ ''-._ 12:42:02 redis_queue.1 | ( ' , .-` | `, ) Running in stand alone mode 12:42:02 redis_queue.1 | |`-._`-...-` __...-.-.|'_.-'| Port: 11000 12:42:02 redis_queue.1 | | -. ._ / _.-' | PID: 15003 12:42:02 redis_queue.1 | -._ -._ -./ .-’ .-’
12:42:02 redis_queue.1 | |-._-. -.__.-' _.-'_.-'| 12:42:02 redis_queue.1 | | -.-._ _.-'_.-' | http://redis.io 12:42:02 redis_queue.1 | -._ -._-..-'.-’ .-’
12:42:02 redis_queue.1 | |-._-. -.__.-' _.-'_.-'| 12:42:02 redis_queue.1 | | -.-._ _.-'_.-' | 12:42:02 redis_queue.1 | -._ -._-..-‘.-’ .-’
12:42:02 redis_queue.1 | -._ -..-’ _.-’
12:42:02 redis_queue.1 | -._ _.-' 12:42:02 redis_queue.1 | -..-’
12:42:02 redis_queue.1 |
12:42:02 redis_queue.1 | [15003] 26 Jul 12:42:02.771 # Server started, Redis version 2.8.4
12:42:02 redis_queue.1 | [15003] 26 Jul 12:42:02.771 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:42:02 redis_queue.1 | [15003] 26 Jul 12:42:02.771 * The server is now ready to accept connections on port 11000
12:42:02 system | watch.1 started (pid=14992)
12:42:02 system | worker_long.1 started (pid=14994)
12:42:02 system | web.1 started (pid=14995)
12:42:02 system | schedule.1 started (pid=14999)
12:42:02 system | redis_socketio.1 started (pid=15000)
12:42:02 redis_socketio.1 | [15016] 26 Jul 12:42:02.808 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984.
12:42:02 redis_socketio.1 | .
12:42:02 redis_socketio.1 | .-__ ''-._ 12:42:02 redis_socketio.1 | _.- . . ‘’-. Redis 2.8.4 (00000000/0) 64 bit
12:42:02 redis_socketio.1 | .- .-```. ```\/ _.,_ ''-._ 12:42:02 redis_socketio.1 | ( ' , .-` | `, ) Running in stand alone mode 12:42:02 redis_socketio.1 | |`-._`-...-` __...-.-.|’_.-'| Port: 12000 12:42:02 redis_socketio.1 | | -._ ._ / _.-' | PID: 15016 12:42:02 redis_socketio.1 | -._ -._ -./ .-’ .-’
12:42:02 redis_socketio.1 | |-._-. -.__.-' _.-'_.-'| 12:42:02 redis_socketio.1 | | -.-._ _.-'_.-' | http://redis.io 12:42:02 redis_socketio.1 | -._ -._-..-'.-’ .-’
12:42:02 redis_socketio.1 | |-._-. -.__.-' _.-'_.-'| 12:42:02 redis_socketio.1 | | -.-._ _.-'_.-' | 12:42:02 redis_socketio.1 | -._ -._-..-‘.-’ .-’
12:42:02 redis_socketio.1 | -._ -..-’ _.-’
12:42:02 redis_socketio.1 | -._ _.-' 12:42:02 redis_socketio.1 | -..-’
12:42:02 redis_socketio.1 |
12:42:02 redis_socketio.1 | [15016] 26 Jul 12:42:02.816 # Server started, Redis version 2.8.4
12:42:02 redis_socketio.1 | [15016] 26 Jul 12:42:02.816 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:42:02 redis_socketio.1 | [15016] 26 Jul 12:42:02.816 * The server is now ready to accept connections on port 12000
12:42:02 system | worker_default.1 started (pid=15001)
12:42:02 system | redis_cache.1 started (pid=15006)
12:42:02 redis_cache.1 | [15023] 26 Jul 12:42:02.848 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984.
12:42:02 redis_cache.1 | .
12:42:02 redis_cache.1 | .-__ ''-._ 12:42:02 redis_cache.1 | _.- . . ‘’-. Redis 2.8.4 (00000000/0) 64 bit
12:42:02 redis_cache.1 | .- .-```. ```\/ _.,_ ''-._ 12:42:02 redis_cache.1 | ( ' , .-` | `, ) Running in stand alone mode 12:42:02 redis_cache.1 | |`-._`-...-` __...-.-.|’_.-'| Port: 13000 12:42:02 redis_cache.1 | | -._ ._ / _.-' | PID: 15023 12:42:02 redis_cache.1 | -._ -._ -./ .-’ .-’
12:42:02 redis_cache.1 | |-._-. -.__.-' _.-'_.-'| 12:42:02 redis_cache.1 | | -.-._ _.-'_.-' | http://redis.io 12:42:02 redis_cache.1 | -._ -._-..-'.-’ .-’
12:42:02 redis_cache.1 | |-._-. -.__.-' _.-'_.-'| 12:42:02 redis_cache.1 | | -.-._ _.-'_.-' | 12:42:02 redis_cache.1 | -._ -._-..-‘_.-’ _.-’
12:42:02 redis_cache.1 | -._ -..-’ _.-’
12:42:02 redis_cache.1 | -._ _.-' 12:42:02 redis_cache.1 | -..-’
12:42:02 redis_cache.1 |
12:42:02 redis_cache.1 | [15023] 26 Jul 12:42:02.863 # Server started, Redis version 2.8.4
12:42:02 redis_cache.1 | [15023] 26 Jul 12:42:02.863 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:42:02 redis_cache.1 | [15023] 26 Jul 12:42:02.863 * The server is now ready to accept connections on port 13000
12:42:03 system | socketio.1 started (pid=15005)
12:42:03 system | worker_short.1 started (pid=15013)
12:42:03 system | socketio.1 stopped (rc=1)
12:42:03 system | sending SIGTERM to redis_socketio.1 (pid 15000)
12:42:03 system | sending SIGTERM to worker_long.1 (pid 14994)
12:42:03 system | sending SIGTERM to watch.1 (pid 14992)
12:42:03 system | sending SIGTERM to redis_queue.1 (pid 14993)
12:42:03 system | sending SIGTERM to web.1 (pid 14995)
12:42:03 system | sending SIGTERM to schedule.1 (pid 14999)
12:42:03 system | sending SIGTERM to worker_default.1 (pid 15001)
12:42:03 system | sending SIGTERM to redis_cache.1 (pid 15006)
12:42:03 system | sending SIGTERM to worker_short.1 (pid 15013)
12:42:03 redis_socketio.1 | [15016 | signal handler] (1469536923) Received SIGTERM, scheduling shutdown…
12:42:03 redis_queue.1 | [15003 | signal handler] (1469536923) Received SIGTERM, scheduling shutdown…
12:42:03 redis_queue.1 | [15003] 26 Jul 12:42:03.005 # User requested shutdown…
12:42:03 redis_queue.1 | [15003] 26 Jul 12:42:03.005 # Redis is now ready to exit, bye bye…
12:42:03 system | redis_queue.1 stopped (rc=-15)
12:42:03 redis_cache.1 | [15023 | signal handler] (1469536923) Received SIGTERM, scheduling shutdown…
12:42:03 redis_cache.1 | [15023] 26 Jul 12:42:03.009 # User requested shutdown…
12:42:03 redis_cache.1 | [15023] 26 Jul 12:42:03.009 # Redis is now ready to exit, bye bye…
12:42:03 system | redis_cache.1 stopped (rc=-15)
12:42:03 system | watch.1 stopped (rc=-15)
12:42:03 system | worker_long.1 stopped (rc=-15)
12:42:03 redis_socketio.1 | [15016] 26 Jul 12:42:03.017 # User requested shutdown…
12:42:03 redis_socketio.1 | [15016] 26 Jul 12:42:03.017 # Redis is now ready to exit, bye bye…
12:42:03 system | redis_socketio.1 stopped (rc=-15)
12:42:03 system | schedule.1 stopped (rc=-15)
12:42:03 system | worker_default.1 stopped (rc=-15)
12:42:03 system | web.1 stopped (rc=-15)
12:42:03 system | worker_short.1 stopped (rc=-15)

2

Are you set up in production? If so, bench is handled with background workers rather than bench start.

If not, sometimes the redis workers don’t terminate. What’s the output of the following?
sudo supervisorctl status
ps aux | grep redis
ps aux | grep frappe

3

@alec_ruizramon1 I am using development setup, No other processes are running for frappe and redis.

sudo supervisorctl status not working, Using development setup.

Thanks

4

Is redis-server running?

5

Yes redis-server works perfect, But while doing bench start I stop all already running redis processes.

Thanks

6

Try this:
bench setup redis

then start again.

7

@alec_ruizramon1 let me give you more details, Thanks for responding.

Actually initially I was getting bin/sh not found error, I figured out and got to know that nodejs was missing on my machine than I install nodejs and executed bench setup socketio.

No luck with bench setup redis :frowning:

One thing more, When I use bench serve I am not getting all erpnext modules displayed on desktop. Is the above issue seems to be preventing that?

Thanks

8

@vjFaLk, @pdvyas can you help on this?

9

The issue with socketio shouldn’t affect which modules show on desktop - check the Setup as well as make sure that those apps are installed on the site you’re using.

I wonder if we could gain insight by seeing if redis/socketio works in a production setup.

Sorry that I didn’t have a quick fix :frowning:

10

@alec_ruizramon1 Thanks

Well I was expecting the get-app to install the erpnext but it didn’t.

So I used bench --site site1.local install-app erpnext , Apps appearing now.

but bench start still not working. using bench serve

Thanks again.

11

Not sure if it’s related but it’s needed to update to nodejs 5 or 6 …in order to have socket working

12

ubuntu@ip-172-31-14-55:~$ sudo supervisorctl status
frappe-bench-redis:frappe-bench-redis-cache RUNNING pid 2182, uptime 0:06:26
frappe-bench-redis:frappe-bench-redis-queue FATAL Exited too quickly (process log may have details)
frappe-bench-redis:frappe-bench-redis-socketio RUNNING pid 2183, uptime 0:06:26
frappe-bench-web:frappe-bench-frappe-web RUNNING pid 2174, uptime 0:06:26
frappe-bench-web:frappe-bench-node-socketio FATAL Exited too quickly (process log may have details)
frappe-bench-workers:frappe-bench-frappe-default-worker-0 RUNNING pid 2155, uptime 0:06:26
frappe-bench-workers:frappe-bench-frappe-long-worker-0 RUNNING pid 2157, uptime 0:06:26
frappe-bench-workers:frappe-bench-frappe-schedule RUNNING pid 2156, uptime 0:06:26
frappe-bench-workers:frappe-bench-frappe-short-worker-0 RUNNING pid 2158, uptime 0:06:26