Event Scheduler (aka Background Jobs Scheduler)

Frappe Framework
1

Hi all,

i’ve added an event scheduler in frappe. It’s based on rq-scheduler and allow to set backgound jobs using cron style.

It has been added in setup → settings:

40
402560×1600 288 KB

and will be possible to create jobs for any app installed in the site:

24
242380×1322 285 KB

Once saved it will enqueued based on the given schedule. When disabled or deleted job will be removed from scheduler.

You can test by installing frappe (develop branch) from:

https://github.com/joezsweet/frappe

and adding rqscheduler: bench rq-scheduler to Procfile

Any hint appreciated! @rmehta @kickapoo

To be done:

  • store jobs at app install time
  • remove jobs on app uninstall
6 Likes
2

@JoEz very cool figure. Today I will test it. :slight_smile:

3

@kickapoo :+1: …let me know

4

Has this been pushed to the frappe repository? This is a really cool feature that I’d love to use in my production environment

5

@Ben_Cornwell_Mott It has been committed but in a slightly different way:

https://github.com/frappe/frappe/pull/4339

3 Likes
6

Hi, @JoEz

Sorry, I know this is an old post but I’m trying to do something almost like this. (Adding UI for user to add scheduler)

May I ask why did you change from having UI into current design which is add scheduler to hook.py file?

7

Hi, because i didn’t really needed cron from UI. Feel free to code it back.

8

Would you mind sharing the previous code for reference so that we can take reference of it?

9

I “believe” I have found this repository, under a different name.

1 Like
10

Was this integrated with frappe as background_jobs?
I notice that uses rq and also noticed there was a move away from celery:

Related:

11

@casesolved-co-uk
Short answer: The repo @JoEz created (that I linked in my last reply) was “partially” integrated into Frappe Framework.

Explanation:

Frappe took certain pieces/ideas from Giovanni’s repo (but not all). Then created the 'Scheduled Job Type' DocType and scheduling solution that we’re using today in v13.

  • Scheduled Job Types are using Python RQ for queues and workers.
  • Unlike Giovanni’s solution, the schedules are hard-coded in hooks.py, using either cron syntax, or schedule syntax.
  • Frappe does not use rq-scheduler. Instead, it has its own scheduler app.

Aside / Fun Fact:
After my post last August, I spent a lot of time learning about Giovanni’s (very clever) 2017 solution, RQ, task schedulers, pickle, and more. I desperately needed a robust scheduler for ERPNext: Scheduled Job Types were not nearly sufficient.

I eventually decided to create a successor to his original work. Along the way, I ran into other challenges with stability. So wrote my own version of ‘rq-scheduler’ too, starting almost from scratch.

My efforts concluded last month (January 2022). My job scheduling solution is 98% finished, and currently being Beta tested. Once I’m confident it’s ready, and I’ve published all the online documentation (Installation, Configuration, Troubleshooting)? Then my intent has been to announce and share this bolt-on scheduling solution with the Frappe/ERPNext Community.

Perhaps in another 2-4 weeks? I really want solid documentation available, so people aren’t struggling to understand and leverage what I’ve built. But this is a busy month/quarter for me.

10 Likes
12

Ah @brian_pond, this would be such a great addition. I cannot wait to hear from you again regarding it’s release. In anticipation, thanks so much for the contribution.

1 Like
13

I am waiting for your released version too,:grinning:

14

Sounds interesting. Waiting for it.

15

Hey @brian_pond,

This would be a solid addition! Eagerly awaiting your version :smiley:

16

I’ve got 1 client using it in Production. Before I’m ready to “officially” announce/release, I need to do a few more things:

  • Finish the online documentation (Installation, Configuration)
  • Create and host a web forum, so people can chat about it, ask questions, etc.
  • Get 2-3 volunteers willing to try installing and using themselves, testing, and provide feedback.
7 Likes
17

@EugeneP @szufisher @TurkerTunali @nasir .

I’ve never forgotten about this thread.

I’m now ready to help people with Background Tasks Unleashed (BTU), a task scheduling App for Frappe Framework.

  1. First, you’re welcome to join me on my forums, to discuss the tool and your experiences. I’ll support you as best I can: https://forums.datahenge.com. This thread here provides some nice screenshots of the tool.

  2. Next, the actual tool consists of 2 separate, open source GitHub repositories:

Both are required. However, after installation and configuration, the scheduling daemon is meant to be “hands-off.” You always interact with the App in your web browser. There are 4 main DocTypes.

  1. Official Documentation is a GitHub Pages site. The source is just markdown documents, located in the ‘docs’ subfolder of the main GitHub repo here.

  2. If you’re unable to chat on the forums, or prefer a private chat, just email me: brian@datahenge.com

I still need to write a lot more documentation/tutorials. Tie up some loose ends. I’m sure someone will want a binary daemon compatible with other Linux distributions, such as CentOS/RedHat.

But, it’s now “good enough” to start sharing with others in the community. One of my clients has been using in Production for over 2 months, running a dozen Tasks, and thousands of Sub Tasks every day.

7 Likes
18

Excellent @brian_pond. It seems very useful and makes things simple. We would like to use and update our feedback.

2 Likes