Gracefully reload Gunicorn for changes to Python Code

Hello,

We have a custom Frappe based app. Often times we’re required to deploy quick fixes (e.g. fix to reports, tiny bug fixes in Py code) on Live Server under load. Currently we have to run sudo supervisorctl reload which sends out HTTP status 500 for 5-10 seconds to all users. Additionally there is risk due to abrupt restart of a worker servicing an active request.

Is there a way of gracefully deploying such tiny changes to Py files without having to do a full reload?

Some googling lead me to below article, but I don’t understand all the moving parts of frappe yet so I’m not sure if it can break things on a full steam production server.
https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html

Can anyone share possible solutions or experience in this regard?

Thanks,
Vamyip

2 Likes

Does this work for your case?

kill -HUP masterpid

https://docs.gunicorn.org/en/stable/faq.html#how-do-i-reload-my-application-in-gunicorn

2 Likes

Looks like this will work.

Added below snippets to make a bench command for it.

In bench/commands/__init__.py

from bench.commands.utils import pyreload
bench_command.add_command(pyreload)

In bench/commands/utils.py

@click.command('pyreload')
def pyreload():
	"Reload Gunicorn workers for small Py changes"
	from bench.utils import exec_cmd, get_bench_name
	folder = get_bench_name('.')
	print(os.path.abspath(os.path.join('.',os.pardir)))
	print(folder)
	exec_cmd(r'sudo supervisorctl status {0}-web:{0}-frappe-web  | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP'.format(folder))
$ cd bench-folder
$ bench pyreload
INFO:bench.utils:sudo supervisorctl status bench-folder-web:bench-folder-frappe-web  | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP
3 Likes

Found a better option which doesn’t requiring a bench command:
sudo supervisorctl signal HUP frappe-bench-web:frappe-bench-frappe-web

In above command replace frappe-bench with the name of your bench.

2 Likes

I usually wait until night to run a bench update --no-backup that deploy in one time all patch of the day

Yes. But when you have to deploy an emergency fix in production without downtime, in those cases graceful reload comes handy. E.g. of such scenarios are a fix in query of report, some misfiring validations, etc