Our Erpnext instance was up and running all fine until late evening yesterday. However, Today when our employees tried to login, the web application did not launch. The browser returned this error:
Site Cannot be reached.
I rebooted the OS and then bench start gave error saying some port was already in use. I tried to kill all those processes and run bench start again which failed multiple times. Then again repeating the same task gave this message:
bench start
12:12:16 system | redis_cache.1 started (pid=4675)
12:12:16 system | redis_socketio.1 started (pid=4678)
12:12:16 system | web.1 started (pid=4681)
12:12:16 system | redis_queue.1 started (pid=4688)
12:12:16 redis_cache.1 | 4679:C 20 Jan 2023 12:12:16.707 # oO0OoO0OoO0Oo Redi s is starting oO0OoO0OoO0Oo
12:12:16 redis_cache.1 | 4679:C 20 Jan 2023 12:12:16.707 # Redis version=6.0. 16, bits=64, commit=00000000, modified=0, pid=4679, just started
12:12:16 redis_cache.1 | 4679:C 20 Jan 2023 12:12:16.707 # Configuration load ed
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.708 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:12:16 system | watch.1 started (pid=4689)
12:12:16 redis_socketio.1 | 4696:C 20 Jan 2023 12:12:16.728 # oO0OoO0OoO0Oo Redi s is starting oO0OoO0OoO0Oo
12:12:16 redis_socketio.1 | 4696:C 20 Jan 2023 12:12:16.741 # Redis version=6.0. 16, bits=64, commit=00000000, modified=0, pid=4696, just started
12:12:16 redis_socketio.1 | 4696:C 20 Jan 2023 12:12:16.742 # Configuration load ed
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.743 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.744 * Running mode=stand alone, port=12000.
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.745 # Server initialized
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.745 # WARNING overcommit _memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot o r run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.746 * Ready to accept co nnections
12:12:16 system | worker_default.1 started (pid=4700)
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 * Running mode=stand alone, port=13000.
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 # Server initialized
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 # WARNING overcommit _memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot o r run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 * Ready to accept co nnections
12:12:16 system | worker_short.1 started (pid=4697)
12:12:16 system | schedule.1 started (pid=4698)
12:12:16 redis_queue.1 | 4692:C 20 Jan 2023 12:12:16.799 # oO0OoO0OoO0Oo Redi s is starting oO0OoO0OoO0Oo
12:12:16 redis_queue.1 | 4692:C 20 Jan 2023 12:12:16.799 # Redis version=6.0. 16, bits=64, commit=00000000, modified=0, pid=4692, just started
12:12:16 redis_queue.1 | 4692:C 20 Jan 2023 12:12:16.799 # Configuration load ed
12:12:16 system | worker_long.1 started (pid=4709)
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.817 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.819 * Running mode=stand alone, port=11000.
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.819 # Server initialized
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.819 # WARNING overcommit _memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot o r run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.820 * Ready to accept co nnections
12:12:16 system | socketio.1 started (pid=4695)
12:12:23 watch.1 |
12:12:25 web.1 | Address already in use
12:12:25 web.1 | Port 8000 is in use by another program. Either ident ify and stop that program, or start the server with a different port.
12:12:25 system | web.1 stopped (rc=1)
12:12:25 system | sending SIGTERM to redis_cache.1 (pid 4675)
12:12:25 system | sending SIGTERM to redis_socketio.1 (pid 4678)
12:12:25 system | sending SIGTERM to redis_queue.1 (pid 4688)
12:12:25 system | sending SIGTERM to socketio.1 (pid 4695)
12:12:25 system | sending SIGTERM to watch.1 (pid 4689)
12:12:25 system | sending SIGTERM to schedule.1 (pid 4698)
12:12:25 system | sending SIGTERM to worker_short.1 (pid 4697)
12:12:25 system | sending SIGTERM to worker_long.1 (pid 4709)
12:12:25 system | sending SIGTERM to worker_default.1 (pid 4700)
12:12:25 system | worker_default.1 stopped (rc=-15)
12:12:25 redis_socketio.1 | 4696:signal-handler (1674196045) Received SIGTERM sc heduling shutdown…
12:12:25 redis_cache.1 | 4679:signal-handler (1674196045) Received SIGTERM sc heduling shutdown…
12:12:25 redis_queue.1 | 4692:signal-handler (1674196045) Received SIGTERM sc heduling shutdown…
12:12:25 system | worker_short.1 stopped (rc=-15)
12:12:25 redis_cache.1 | 4679:M 20 Jan 2023 12:12:25.627 # User requested shu tdown…
12:12:25 redis_cache.1 | 4679:M 20 Jan 2023 12:12:25.627 * Removing the pid f ile.
12:12:25 redis_cache.1 | 4679:M 20 Jan 2023 12:12:25.627 # Redis is now ready to exit, bye bye…
12:12:25 redis_queue.1 | 4692:M 20 Jan 2023 12:12:25.629 # User requested shu tdown…
12:12:25 redis_queue.1 | 4692:M 20 Jan 2023 12:12:25.629 * Removing the pid f ile.
12:12:25 redis_queue.1 | 4692:M 20 Jan 2023 12:12:25.629 # Redis is now ready to exit, bye bye…
12:12:25 system | redis_queue.1 stopped (rc=-15)
12:12:25 system | redis_cache.1 stopped (rc=-15)
12:12:25 system | worker_long.1 stopped (rc=-15)
12:12:25 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:25.635 # User requested shu tdown…
12:12:25 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:25.635 * Removing the pid f ile.
12:12:25 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:25.635 # Redis is now ready to exit, bye bye…
12:12:25 system | redis_socketio.1 stopped (rc=-15)
12:12:25 system | schedule.1 stopped (rc=-15)
12:12:25 system | socketio.1 stopped (rc=-15)
12:12:25 system | watch.1 stopped (rc=-15)
I can see two problems:
- port 8000 in use
- Received SIGTERM scheduling shutdown…
Question 1: What is causing this issue, especially when no one has accessed the server or made any changes since yesterday ?
Question 2: How can I solve this issue ?