Scheduler & Email not working [A band-aid solution]

ERPNext
,
1

I’m not sure if there are others out there experiencing a similar problem, but recently we’ve experienced problems with the scheduled tasks and email not working and I’d just like to share how we got to fixing it on V7. The fix works with other versions as well. (tl;dr to the libraries and versions at the bottom)

How we troubleshooted/fixed the error/s:

  1. Used the bench commands (like bench doctor) to check if bench is working as it should. Workers were alive but they just weren’t doing anything. bench clear-cache, bench migrate did nothing. So, we checked the logs.

  2. Read the log files (~/frappe-bench/logs) and read the tracebacks. We checked frappe.log and all the redis logs to narrow down the problem (to a number of libraries that required a downgrade for v7). Logs showed the following errors:

  • ImportError: cannot import name 'OpenIDConnectAuthCode’
  • RedisError: ZADD requires an equal number of values and scores (caused by site-packages/rq/worker.py)
  • FeatureNotFound: Couldn’t find a tree builder with the features you requested: html5lib
  1. Checked the library versions and dependencies. Some libraries/features need you to upgrade/downgrade a different library/feature for it to work.
  • doing a quick discuss search would tell you to downgrade oauthlib to oauthlib 2.1.0 and redis to redis 2.10.6, and install html5lib 1.0.1
  • What it won’t tell you: rq 0.13 and higher require redis 3. Downgrading redis to 2.10.6 would make rq stop working. You’ll need to downgrade rq to 0.11.00.

  1. Deleted the logs and did a reboot. We started with the OpenIDConnectAuthCode error before finding the ZADD/rq/redis error in one of the redis log files. The logs should be auto generated once bench attempts to run the scheduler tasks (which includes email queue’s flush). If there’s something wrong, it’ll appear in either the frappe.log file or one of the error files. Troubleshoot from there.

  2. Have patience. Lots of patience. And pray that nothing else goes wrong.

TL;DR:

cd ~/frappe-bench/env/bin && source activate
pip install --force-reinstall rq==0.11.00
pip install --force-reinstall oauthlib==2.1.0
pip install --force-reinstall redis==2.10.6
pip install --force-reinstall html5lib==1.0.1
deactivate
sudo reboot

If there are additional problems, try checking the log files again and working from there.

Hopes this helps someone in this community.

Cheers.

Littlehera

7 Likes
2

Thank you for sharing, its greatly appreciated

1 Like
3

Thanks @littlehera was in the same situation this helped me to resolve it

4

What could the specs be for v13?