[solution] 'supervisorctl start all' has no response or affect

Install / Update Deployment
1

Here’s my problem report and solution -

This command below had no response or effect:

sudo supervisorctl start all

So I inspected /var/log/supervisor/supervisord.log for a ‘working’ startup session, for eg this output here:

2017-07-10 08:20:18,507 CRIT Supervisor running as root (no user in config file)
2017-07-10 08:20:18,531 WARN Included extra file “/etc/supervisor/conf.d/frappe-bench.conf” during parsing
2017-07-10 08:20:18,580 INFO RPC interface ‘supervisor’ initialized
2017-07-10 08:20:18,580 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2017-07-10 08:20:18,592 INFO daemonizing the supervisord process
2017-07-10 08:20:18,595 INFO supervisord started with pid 1557
2017-07-10 08:20:19,603 INFO spawned: ‘frappe-bench-frappe-default-worker-0’ with pid 1588
2017-07-10 08:20:19,631 INFO spawned: ‘frappe-bench-frappe-schedule’ with pid 1590
2017-07-10 08:20:19,663 INFO spawned: ‘frappe-bench-frappe-long-worker-0’ with pid 1596
2017-07-10 08:20:19,698 INFO spawned: ‘frappe-bench-frappe-short-worker-0’ with pid 1602
2017-07-10 08:20:19,759 INFO spawned: ‘frappe-bench-frappe-web’ with pid 1612
2017-07-10 08:20:19,789 INFO spawned: ‘frappe-bench-node-socketio’ with pid 1622
2017-07-10 08:20:19,880 INFO spawned: ‘frappe-bench-redis-queue’ with pid 1628
2017-07-10 08:20:19,934 INFO spawned: ‘frappe-bench-redis-cache’ with pid 1629
2017-07-10 08:20:20,029 INFO spawned: ‘frappe-bench-redis-socketio’ with pid 1630
2017-07-10 08:20:21,443 INFO success: frappe-bench-frappe-default-worker-0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-frappe-schedule entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-frappe-long-worker-0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-frappe-short-worker-0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-frappe-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-node-socketio entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-redis-queue entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-redis-cache entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-10 08:20:21,443 INFO success: frappe-bench-redis-socketio entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

The log confirmed this link /etc/supervisor/conf.d/frappe-bench.conf noted in the above log was now gone (somehow deleted!?)

The fix was simply to recreate that symbolic link like so:

sudo ln -s /home/frappe/frappe-bench/config/supervisor.conf /etc/supervisor/conf.d/frappe-bench.conf

To restore the supervisor service, I then ran these commands:

sudo bench setup supervisor
sudo supervisorctl reread
sudo supervisorctl update

sudo supervisorctl stop all
sudo supervisorctl start all

Here is an example of a correctly working supervisor

frappe@erpnext:~/frappe-bench$ sudo supervisorctl status
frappe-bench-redis:frappe-bench-redis-cache RUNNING pid 5367, uptime 1:28:41
frappe-bench-redis:frappe-bench-redis-queue RUNNING pid 5366, uptime 1:28:41
frappe-bench-redis:frappe-bench-redis-socketio RUNNING pid 5373, uptime 1:28:41
frappe-bench-web:frappe-bench-frappe-web RUNNING pid 5364, uptime 1:28:41
frappe-bench-web:frappe-bench-node-socketio RUNNING pid 5365, uptime 1:28:41
frappe-bench-workers:frappe-bench-frappe-default-worker-0 RUNNING pid 5337, uptime 1:28:45
frappe-bench-workers:frappe-bench-frappe-long-worker-0 RUNNING pid 5346, uptime 1:28:44
frappe-bench-workers:frappe-bench-frappe-schedule RUNNING pid 5336, uptime 1:28:45
frappe-bench-workers:frappe-bench-frappe-short-worker-0 RUNNING pid 5357, uptime 1:28:42
frappe@erpnext:~/frappe-bench$ sudo rm /etc/supervisor/conf.d/frappe-bench.conf
frappe@erpnext:~/frappe-bench$ sudo supervisorctl status
frappe-bench-redis:frappe-bench-redis-cache RUNNING pid 5367, uptime 1:29:07
frappe-bench-redis:frappe-bench-redis-queue RUNNING pid 5366, uptime 1:29:07
frappe-bench-redis:frappe-bench-redis-socketio RUNNING pid 5373, uptime 1:29:07
frappe-bench-web:frappe-bench-frappe-web RUNNING pid 5364, uptime 1:29:07
frappe-bench-web:frappe-bench-node-socketio RUNNING pid 5365, uptime 1:29:07
frappe-bench-workers:frappe-bench-frappe-default-worker-0 RUNNING pid 5337, uptime 1:29:11
frappe-bench-workers:frappe-bench-frappe-long-worker-0 RUNNING pid 5346, uptime 1:29:10
frappe-bench-workers:frappe-bench-frappe-schedule RUNNING pid 5336, uptime 1:29:11
frappe-bench-workers:frappe-bench-frappe-short-worker-0 RUNNING pid 5357, uptime 1:29:08

5 Likes
2

Also a problem here was that nginx failed to bind to port 80 as expected. For example no sign of these ‘healthy’ diagnostics in the logs:

frappe@erpnext:~/frappe-bench$ sudo netstat -tlnp | grep 80
tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 1639/python
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1946/nginx

frappe@erpnext:~/frappe-bench$ less /var/log/nginx/error.log | grep bind
2017/09/30 05:33:14 [debug] 2392#0: bind() 0.0.0.0:80 #6

The problem was this link was missing:

frappe@erpnext:~/frappe-bench$ ls -al /etc/nginx/conf.d/
total 8
drwxr-xr-x 2 root root 4096 Sep 30 05:32 .
drwxr-xr-x 5 root root 4096 Sep 30 05:39 …
lrwxrwxrwx 1 root root 43 Sep 30 05:32 frappe-bench.conf → /home/frappe/frappe-bench/config/nginx.conf

To restore this symbolic link, do this:

frappe@erpnext:~/frappe-bench$ sudo ln -s /home/frappe/frappe-bench/config/nginx.conf /etc/nginx/conf.d/frappe-bench.conf

restart nginx to load the change:
frappe@erpnext:~/frappe-bench$ sudo service nginx restart

  • Restarting nginx nginx [ OK ]

btw to enable and view debug output in /var/log/nginx/error.log
to this /etc/nginx/nginx.conf add this line:
error_log /var/log/nginx/error.log debug;

1 Like
3

Worked for me…Thanks

4

This worked for me for ERPNext in Ubuntu 22.04 LTS.