Background jobs and Kafka

I saw this video Himanshu and Abhishek - Kafka for Background Jobs - YouTube and I was wondering if there is already a PR for it.
And I want to know if someone can help me with this question: We are having a lot of problems with our background jobs and multitenant, and we are planning to implement more redis queues to prevent the long jobs block the shorter ones (long jobs of about 1 hour). Do you think that this implementation of Kafka + Greenlets can help us or should we stick to our plan of create more redis queues?

I’d like to have some insights of the people in the video, but I couldn’t find them in discuss to tag them. Can someone tag them for me?

choose the simple solution before looking at the complex solution.

Just add more queues/workers. If that works, you’re done. If not, then start looking at more complex solutions.

1 Like

The simplest solution (add more workers) is not enough, because in one of our queues we need to do the jobs in order and if we add more workers then they do simultaneous jobs.
So, the “simplest” solution is to reuse the code from a very old PR and add our custom queues. But we are interested in this bacause maybe this can help us.

1 Like

Hi, I know it’s an old thread… but I will be glad to know about this problem,
could you create a new custom worker queue like explained in that PR?