Job scheduler stopped working

mrPauwHaan · August 23, 2024, 6:28am

My job scheduler stopped working randomly last night at 12am. After that time, all jobs are not running anymore.

I tried re-enabling the job-scheduler but that is not working. Also bench migrate does not help and re-abling the server script job does also not work. The jobs are scheduled in ‘scheduled job type’ but just not running.

How do I fix this?

avc · August 23, 2024, 6:43am

Hi @mrPauwHaan

Tried bench scheduler resume ?

Anyway, I’ve faced something like that weeks ago and finally reinstalled the site restoring backup because I was unable to know what was happening … version-15?

mrPauwHaan · August 23, 2024, 7:38am

Thanks for your quick response! I tried that but it also did not work. It is indeed in the latest version.

mrPauwHaan · August 24, 2024, 7:04am

For an unknown reason it started working again and the scheduler runs fine now. I have still no clue what causes this issue but it happens regularly that the scheduler has its own plans

avc · August 24, 2024, 7:14am

Scheduler is the rebel teenager!

gtapias · August 25, 2024, 4:28am

Login out and login back in has helped me before

avc · August 25, 2024, 8:47am

Hi @gtapias:

Thanks, I will try if this happen again!
I suspect that REDIS is involved somehow here, but …

avc · August 26, 2024, 3:38pm

@gtapias:

Didn’t wait too long …
Today, suddenly, scheduler stops on one site …

After logout/login, it starts again …
Ay, qué cosas!

gtapias · August 26, 2024, 3:51pm

It’s the little imperfections that make you perfect in my eyes.

avc · August 26, 2024, 4:19pm

Hi again @gtapias:

I think I found the reason for that …

If the system detects no interactive activity (mean manual login) in the site for the last x days (defined on system settings) and you are not in developer_mode (since v15.38.0) … jobs will get “dormant” status … and, theorically, it will run just daily. I think something like this is happening. This is the reason why it restarts after logout/login

github.com

frappe/frappe/blob/a162e61e26a5b9f4b8a6e0c3ac02d1d900817a99/frappe/utils/scheduler.py#L184


      
          
          def enable_scheduler():
          	toggle_scheduler(True)
          
          
          def disable_scheduler():
          	toggle_scheduler(False)
          
          
          @redis_cache(ttl=60 * 60)
          def schedule_jobs_based_on_activity(check_time=None):
          	"""Return True for active sites as defined by `Activity Log`.
          	Also return True for inactive sites once every 24 hours based on `Scheduled Job Log`."""
          	if is_dormant(check_time=check_time):
          		# ensure last job is one day old
          		last_job_timestamp = _get_last_creation_timestamp("Scheduled Job Log")
          		if not last_job_timestamp:
          			return True
          		else:
          			if ((check_time or now_datetime()) - last_job_timestamp).total_seconds() >= 86400:
          				# one day is passed since jobs are run, so lets do this

And finally … check this:

github.com/frappe/frappe

bug: Scheduler sometimes does not work

opened 06:13AM - 31 Jul 24 UTC

closed 10:52AM - 12 Aug 24 UTC

arunmathaisk

bug released

**Story**: My site was working just fine until one day it did not. All the s…cheduled jobs were not being processed. After much debugging with @adityahase, the issue was figured out. **Premise:** I had not logged out from the site in a while and somehow the site went into dormant mode. (Activity Log). And even after the logout and login, nothing changed cause "cache" :) Clearing the cache solved the issue. **Solution** Give no regard to the dormant mode when a site is in developer mode.

Maybe bench clear-cache could help too in some cases.

gtapias · August 26, 2024, 5:14pm

Oh, wow! Thank you, @avc

We have about 40 users, still goes dormant once in a while, meaning logins and logouts combinations for 40 users still probable enough to not have activity for that number of days. Increasing to 30 in my production system (unless there is a good reason not to do so).

Renaming that section to make it even more clear that we are talking about days and not number of workers would help.

Background Workers Inactivity Days

Reduce job runs frequency to once per day after the above number of inactive days.

Again, tooltips would be of great use here to add detail (separate rant ).

mrPauwHaan · September 14, 2024, 8:45am

Indeed, in activity log the last user logged in 4 days ago. Since logins are handled by a social login this is not always seen as a login/logout and causes issues with this 4-day scheduler rule. I just changed the amount of days from 4 to 365 to fix it.