Will frappe 7 support Python RQ Workers?

Frappe Framework
1

Due frappé 7 supports Python RQ, I have some resources to develop that requires long pooling workers, like my ESB.

I’m thinking if frappe 7 will support Python RQ Workers?

2

Yes, we start three workers for each queue: Short , default and long. Jobs are given to them according to the kind of task it is. Long jobs like “Dropbox Backup” are put into the long queue, so the long worker will handle it. Have a look at background_jobs.py. To enqueue a job, just call enqueue(<method>)

To add more workers you can just use the syntax we use in “Procfile”

3

@vjFaLk, thaks for you answer!

The last point is:

Can we create a pattern using hooks for registry new Workers
The most important to me, is that there workers should be managed by a visual interface, but I dont want do the service orquestration in the backed everytime that I need start a Worker!

I’m just thinking about a option, to the worker be started too by supervisor, with the others frappe workers.

I have a giant list of things that can be solved by this scheme, the first is API integrations.

4

So currently, if you have a production setup Supervisor will handle the workers. Problem is that there isn’t a way to start multiple workers using code.

You can also use the bench worker [queue] command to start a new worker

What do you want to create more workers for anyway?

5

@vjFaLk, I want delegate specific functions for Workers, for example:

I need call a Rest API to get some data, I have a Worker running that listen a inbound queue, when this worker receive some data in the inbound queue, they will process and put the content in a outgoing queue, for another Worker

A simple case is:

You have a register in a WebHook API, to it call a end point at your side:

The idea is:

  • The webhook call you
  • You get the content from webhook and put it in a queue
  • In this queue you have a worker listening this
  • The worker get the content of the queue, process, and put it in another queue.

The advantage that I can see in this scheme is, your never will lose your data, due queue are durables, and if you got something wrong, you apply a patch and the flow continue!

Is the same concept of Email queue

1 Like
6

@max_morais_dmm

Sorry for the late reply!

I think creating a separate queue and dedicating a worker for it, and running it via Procfile or Supervisor will work out for you. Then you can just use the enqueue function and push your jobs into that queue.

1 Like
7

Sorry for reviving this topic but I search a lot and can’t find a solution yet…
How is possible to do this…