Cannot launch ERPNext

Our Erpnext instance was up and running all fine until late evening yesterday. However, Today when our employees tried to login, the web application did not launch. The browser returned this error:
Site Cannot be reached.

I rebooted the OS and then bench start gave error saying some port was already in use. I tried to kill all those processes and run bench start again which failed multiple times. Then again repeating the same task gave this message:

bench start
12:12:16 system | redis_cache.1 started (pid=4675)
12:12:16 system | redis_socketio.1 started (pid=4678)
12:12:16 system | web.1 started (pid=4681)
12:12:16 system | redis_queue.1 started (pid=4688)
12:12:16 redis_cache.1 | 4679:C 20 Jan 2023 12:12:16.707 # oO0OoO0OoO0Oo Redi s is starting oO0OoO0OoO0Oo
12:12:16 redis_cache.1 | 4679:C 20 Jan 2023 12:12:16.707 # Redis version=6.0. 16, bits=64, commit=00000000, modified=0, pid=4679, just started
12:12:16 redis_cache.1 | 4679:C 20 Jan 2023 12:12:16.707 # Configuration load ed
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.708 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:12:16 system | watch.1 started (pid=4689)
12:12:16 redis_socketio.1 | 4696:C 20 Jan 2023 12:12:16.728 # oO0OoO0OoO0Oo Redi s is starting oO0OoO0OoO0Oo
12:12:16 redis_socketio.1 | 4696:C 20 Jan 2023 12:12:16.741 # Redis version=6.0. 16, bits=64, commit=00000000, modified=0, pid=4696, just started
12:12:16 redis_socketio.1 | 4696:C 20 Jan 2023 12:12:16.742 # Configuration load ed
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.743 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.744 * Running mode=stand alone, port=12000.
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.745 # Server initialized
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.745 # WARNING overcommit _memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot o r run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:12:16 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:16.746 * Ready to accept co nnections
12:12:16 system | worker_default.1 started (pid=4700)
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 * Running mode=stand alone, port=13000.
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 # Server initialized
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 # WARNING overcommit _memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot o r run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:12:16 redis_cache.1 | 4679:M 20 Jan 2023 12:12:16.762 * Ready to accept co nnections
12:12:16 system | worker_short.1 started (pid=4697)
12:12:16 system | schedule.1 started (pid=4698)
12:12:16 redis_queue.1 | 4692:C 20 Jan 2023 12:12:16.799 # oO0OoO0OoO0Oo Redi s is starting oO0OoO0OoO0Oo
12:12:16 redis_queue.1 | 4692:C 20 Jan 2023 12:12:16.799 # Redis version=6.0. 16, bits=64, commit=00000000, modified=0, pid=4692, just started
12:12:16 redis_queue.1 | 4692:C 20 Jan 2023 12:12:16.799 # Configuration load ed
12:12:16 system | worker_long.1 started (pid=4709)
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.817 * Increased maximum number of open files to 10032 (it was originally set to 1024).
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.819 * Running mode=stand alone, port=11000.
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.819 # Server initialized
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.819 # WARNING overcommit _memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot o r run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
12:12:16 redis_queue.1 | 4692:M 20 Jan 2023 12:12:16.820 * Ready to accept co nnections
12:12:16 system | socketio.1 started (pid=4695)
12:12:23 watch.1 |
12:12:25 web.1 | Address already in use
12:12:25 web.1 | Port 8000 is in use by another program. Either ident ify and stop that program, or start the server with a different port.
12:12:25 system | web.1 stopped (rc=1)
12:12:25 system | sending SIGTERM to redis_cache.1 (pid 4675)
12:12:25 system | sending SIGTERM to redis_socketio.1 (pid 4678)
12:12:25 system | sending SIGTERM to redis_queue.1 (pid 4688)
12:12:25 system | sending SIGTERM to socketio.1 (pid 4695)
12:12:25 system | sending SIGTERM to watch.1 (pid 4689)
12:12:25 system | sending SIGTERM to schedule.1 (pid 4698)
12:12:25 system | sending SIGTERM to worker_short.1 (pid 4697)
12:12:25 system | sending SIGTERM to worker_long.1 (pid 4709)
12:12:25 system | sending SIGTERM to worker_default.1 (pid 4700)
12:12:25 system | worker_default.1 stopped (rc=-15)
12:12:25 redis_socketio.1 | 4696:signal-handler (1674196045) Received SIGTERM sc heduling shutdown…
12:12:25 redis_cache.1 | 4679:signal-handler (1674196045) Received SIGTERM sc heduling shutdown…
12:12:25 redis_queue.1 | 4692:signal-handler (1674196045) Received SIGTERM sc heduling shutdown…
12:12:25 system | worker_short.1 stopped (rc=-15)
12:12:25 redis_cache.1 | 4679:M 20 Jan 2023 12:12:25.627 # User requested shu tdown…
12:12:25 redis_cache.1 | 4679:M 20 Jan 2023 12:12:25.627 * Removing the pid f ile.
12:12:25 redis_cache.1 | 4679:M 20 Jan 2023 12:12:25.627 # Redis is now ready to exit, bye bye…
12:12:25 redis_queue.1 | 4692:M 20 Jan 2023 12:12:25.629 # User requested shu tdown…
12:12:25 redis_queue.1 | 4692:M 20 Jan 2023 12:12:25.629 * Removing the pid f ile.
12:12:25 redis_queue.1 | 4692:M 20 Jan 2023 12:12:25.629 # Redis is now ready to exit, bye bye…
12:12:25 system | redis_queue.1 stopped (rc=-15)
12:12:25 system | redis_cache.1 stopped (rc=-15)
12:12:25 system | worker_long.1 stopped (rc=-15)
12:12:25 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:25.635 # User requested shu tdown…
12:12:25 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:25.635 * Removing the pid f ile.
12:12:25 redis_socketio.1 | 4696:M 20 Jan 2023 12:12:25.635 # Redis is now ready to exit, bye bye…
12:12:25 system | redis_socketio.1 stopped (rc=-15)
12:12:25 system | schedule.1 stopped (rc=-15)
12:12:25 system | socketio.1 stopped (rc=-15)
12:12:25 system | watch.1 stopped (rc=-15)

I can see two problems:

  1. port 8000 in use
  2. Received SIGTERM scheduling shutdown…

Question 1: What is causing this issue, especially when no one has accessed the server or made any changes since yesterday ?
Question 2: How can I solve this issue ?

Stop the bench using this script and then start the bench again (“bench start”)

Bench stop was successful. Bench start was also okay. But I still cannot access web application.

Need more info, like bench logs, your site configurations, URL that you’re accessing and any other info that you can provide.

I assume you are running this in development ?

Try accessing it on the IP:PORT instead of erp.ajiragroup.com you might have domain issues.

I am running in Production.
Running IP:port also doesnot work. I had already checked it before posting here.

Where can I find bench logs ? And what site configurations do you need to know ? Please let me know so that I can share them.

I can access the site (http://erp.ajiragroup.com/) from Chrome mobile browser. If you are using production then you don’t need to do bench start. This command is only for development. Just reboot the server and let system auto start all the required process.

Thank you for the information.

I discovered it had something to do with lets encrypt ssl certificate.