I’m not sure if there are others out there experiencing a similar problem, but recently we’ve experienced problems with the scheduled tasks and email not working and I’d just like to share how we got to fixing it on V7. The fix works with other versions as well. (tl;dr to the libraries and versions at the bottom)
How we troubleshooted/fixed the error/s:
Used the bench commands (like
bench doctor) to check if bench is working as it should. Workers were alive but they just weren’t doing anything.
bench migratedid nothing. So, we checked the logs.
Read the log files (~/frappe-bench/logs) and read the tracebacks. We checked frappe.log and all the redis logs to narrow down the problem (to a number of libraries that required a downgrade for v7). Logs showed the following errors:
- ImportError: cannot import name 'OpenIDConnectAuthCode’
- RedisError: ZADD requires an equal number of values and scores (caused by site-packages/rq/worker.py)
- FeatureNotFound: Couldn’t find a tree builder with the features you requested: html5lib
- Checked the library versions and dependencies. Some libraries/features need you to upgrade/downgrade a different library/feature for it to work.
- doing a quick discuss search would tell you to downgrade
redis 2.10.6, and install
- What it won’t tell you:
rq0.13 and higher require
redis 3. Downgrading redis to 2.10.6 would make rq stop working. You’ll need to downgrade rq to 0.11.00.
Deleted the logs and did a reboot. We started with the OpenIDConnectAuthCode error before finding the ZADD/rq/redis error in one of the redis log files. The logs should be auto generated once bench attempts to run the scheduler tasks (which includes email queue’s flush). If there’s something wrong, it’ll appear in either the frappe.log file or one of the error files. Troubleshoot from there.
Have patience. Lots of patience. And pray that nothing else goes wrong.
cd ~/frappe-bench/env/bin && source activate pip install --force-reinstall rq==0.11.00 pip install --force-reinstall oauthlib==2.1.0 pip install --force-reinstall redis==2.10.6 pip install --force-reinstall html5lib==1.0.1 deactivate sudo reboot
If there are additional problems, try checking the log files again and working from there.
Hopes this helps someone in this community.