Any way to run a background task or shared object idling?

I got a task which takes too much time to start up all the dependencies and will therefore block my custom app. The main task itself very quickly finished, once everything is loaded up. I’m searching for a solution in order to prevent the startup-delay:

Is there Any option to run my python task idling in the background and ready for action? It’s one Python object, which should be ready & accessible throughout my frappe app in order to access the background-worker.

Are there any other ideas or options?

What i’ve tried so far:

  • Using singletons and initialize the object on initialization. But failed due to cannot switch to a different thread
  • Checking out background jobs… didn’t found anything useful. I need a constantly-running background process (the process won’t take a lot of resources)

Not sure I’m entirely following…

…but if I was doing this, I might just write a standalone Python program. Teach it to do whatever it’s supposed to do. Then have run it “always-on” in the background as a daemon.

Then to interact with my background program, either:

  • add a tiny web server or Unix socket.
  • or if I only need data, teach my program to write its output to Redis/SQL. Then I could query the results whenever I needed them from [somewhere else]

I wrote my own background task scheduler for Frappe a few years ago, doing basically that ^.

2 Likes

Wow, thank you @brian_pond ! You totally got me, that’s the solution I’m heading for. I already discovered your BTU Scheduler Daemon with joy :slight_smile:

Just one last question: I don’t think there’s a good option to integrate a solution like this fully into my custom app, am I right? It would be perfect if the user just needs to install the app, migrate and everything (including the deamon) is ready to go.

That’s a fantastic question. There’s definitely no straightforward way to make this happen automatically (unfortunately).

But I can think of a few viable workarounds …

Option 1: During ‘bench get-app’ and ‘bench install-app’

When these 2 commands are run, whatever Bench and Frappe Framework are doing? That’s largely out of our control.

But there -is- something thing we can control: our App’s 'pyproject.toml' configuration file.

  • We can add additional Python dependencies (perhaps your custom daemon)
  • We can teach our App’s package to run custom code/scripts during installation. For example:
[project.scripts]
my-install-script = "my_package.my_module:post_install_function"

Then write a 'post_install_function()' to accomplish whatever you need. Downloading files from the internet, create directories, running Bash scripts, building system unit files, etc.

Here’s a good guide on ‘pyproject.toml’
Configuring setuptools using pyproject.toml files - setuptools 74.1.2.post20240906 documentation

Option 2: During module import.

Whenever the Python runtime loads our App’s Python module, the code in __init__.py is executed. Normally in '__init__.py' we’d add things like global variables, functions, or classes. But we could also just run code whenever that file is loaded. Code that does “installation stuff”.

""" __init__.py """

import frappe
__version__ = '0.1.0'

def do_post_installation_things():
    print("I'm going to examine directories, and make sure things are installed and configured."
    print("If they are not, I'm going to automatically fix them.")

do_post_installation_things()  # this calls the function whenever __init__.py is loaded

Granted, this solution is a bit clunky and will be surprising to other developers. Also, your function do_post_installation_things() is going to be called many times, often somewhat out of your control. So you’ll need to be careful. Make sure it’s safe for the function to be repeatedly called, but only actually install stuff when it’s truly missing (so it doesn’t slow down the system). But this would work.

Option 3: By altering your App’s ‘hooks.py’

The ‘hooks.py’ code is being imported and run all the time (possibly too-often, but that’s a topic for another day). Because you can count on it being run, you could add your post-installation steps in there. Either by just calling a function like I did in Option 2. Or by configuring a scheduler, to check for things automatically:

# Every 5 minutes, make sure everything is installed the way it's supposed to be.
scheduler_events = {

	"cron": {
	 	"0/5 * * * *": [
          "my_app.my_module.check_for_installed_things",
	 	]
	}
}
2 Likes

Thank you very much, that’s very valuable for me and solved my (two) questions to it’s best - even though your solution led me onto a different path than I thought of initially :slight_smile: . I was stuck for days finding a good alternative, and I’m happy to keep going on. Your answer is way more than I expected, btw.

Thanks for solving my blocking task in order to proceed with the implementation of ERPNext on my end and have a nice week.