Background job queue keep growing?

Hi @tmatteson ,
Looking for help to start/stop background job since it keeps growing with orange dot and short status.

Due to this issue email is not sent also email queue keep growing?

Thx

Can you please check error log and post here?

You should increase your background workers and gunicorn workers. I have beefy multiple VPS, and I started messing with performance tuning for my ERPNext instances, and editing those after increasing innodb cache size really helps. My emails are sent in an instant.

Just my experience though, your mileage may vary :slight_smile:

Here’s an old thread discussing about performance: What is the difference between background_workers and gunicorn_workers

@nmami Nofal, this was not the question I was expecting from the Telegram chat. I think @iMoshi’s suggestions are a good starting place. For further debug, run bench doctor and/or bench enable-scheduler or bench disable-scheduler.

Exited, it’s resolved.

Thanks for your support.

Action plan:
1- edit file frappe-bench/sites/common_site_config.json
2- update value of background_workers from 1 to 3
3- save
4- restart server

expected result :
Background jobs get cleared including sending email job which is added under hooks.py file as cron.

But is it save to update hooks.py under erpnext app? next update will get overwritten right?

what is the best approach to add cron part in the hooks?

my /home/frappe/frappe-bench/apps/erpnext/erpnext/hooks.py file after adding cron part:

scheduler_events = {
"all": [
	"erpnext.projects.doctype.project.project.project_status_update_reminder"
],
"hourly": [
	'erpnext.hr.doctype.daily_work_summary_group.daily_work_summary_group.trigger_emails',
	"erpnext.accounts.doctype.subscription.subscription.process_all",
	"erpnext.erpnext_integrations.doctype.amazon_mws_settings.amazon_mws_settings.schedule_get_order_details",
	"erpnext.accounts.doctype.gl_entry.gl_entry.rename_gle_sle_docs",
	"erpnext.erpnext_integrations.doctype.plaid_settings.plaid_settings.automatic_synchronization",
	"erpnext.projects.doctype.project.project.hourly_reminder",
	"erpnext.projects.doctype.project.project.collect_project_status",
	"erpnext.hr.doctype.shift_type.shift_type.process_auto_attendance_for_all_shifts",
	"erpnext.support.doctype.issue.issue.set_service_level_agreement_variance",
],
"daily": [
	"erpnext.stock.reorder_item.reorder_item",
	"erpnext.support.doctype.issue.issue.auto_close_tickets",
	"erpnext.crm.doctype.opportunity.opportunity.auto_close_opportunity",
	"erpnext.controllers.accounts_controller.update_invoice_status",
	"erpnext.accounts.doctype.fiscal_year.fiscal_year.auto_create_fiscal_year",
	"erpnext.hr.doctype.employee.employee.send_birthday_reminders",
	"erpnext.projects.doctype.task.task.set_tasks_as_overdue",
	"erpnext.assets.doctype.asset.depreciation.post_depreciation_entries",
	"erpnext.hr.doctype.daily_work_summary_group.daily_work_summary_group.send_summary",
	"erpnext.stock.doctype.serial_no.serial_no.update_maintenance_status",
	"erpnext.buying.doctype.supplier_scorecard.supplier_scorecard.refresh_scorecards",
	"erpnext.setup.doctype.company.company.cache_companies_monthly_sales_history",
	"erpnext.assets.doctype.asset.asset.update_maintenance_status",
	"erpnext.assets.doctype.asset.asset.make_post_gl_entry",
	"erpnext.crm.doctype.contract.contract.update_status_for_contracts",
	"erpnext.projects.doctype.project.project.update_project_sales_billing",
	"erpnext.projects.doctype.project.project.send_project_status_email_to_users",
	"erpnext.quality_management.doctype.quality_review.quality_review.review",
	"erpnext.support.doctype.service_level_agreement.service_level_agreement.check_agreement_status",
	"erpnext.crm.doctype.email_campaign.email_campaign.send_email_to_leads_or_contacts",
	"erpnext.crm.doctype.email_campaign.email_campaign.set_email_campaign_status",
	"erpnext.selling.doctype.quotation.quotation.set_expired_status"
],
"daily_long": [
	"erpnext.setup.doctype.email_digest.email_digest.send",
	"erpnext.manufacturing.doctype.bom_update_tool.bom_update_tool.update_latest_price_in_all_boms",
	"erpnext.hr.doctype.leave_ledger_entry.leave_ledger_entry.process_expired_allocation",
	"erpnext.hr.utils.generate_leave_encashment"
],
"monthly_long": [
	"erpnext.accounts.deferred_revenue.convert_deferred_revenue_to_income",
	"erpnext.accounts.deferred_revenue.convert_deferred_expense_to_expense",
	"erpnext.hr.utils.allocate_earned_leaves"
],
"cron": {
    "* * * * *": [
        "frappe.email.queue.flush"
    ]
}

}
Regards
Nofal

unfortunately, still issue not resolved.

Any idea how to start background jobs? why system keep queuing jobs?

thanks in advance

I guess the issue maybe due to your server specification being low. Usually a 2GB RAM is the minimum requirement with Intel Xeon processor.

Hi @Muzzy currently I am using a wsl with Ubuntu 18.04, production, local (no Network)

I am able to get email send by doing below action plan, plz share your thoughts:

final solution:

add cron on server side

frappe@LAPTOP-5F23F7J8:~/frappe-bench$ sudo crontab -e

add this line in the end


* * * * * cd /home/frappe/frappe-bench && bench execute frappe.email.queue.flush

save & exit

stop / start server
1 Like

Is there any way of dedicating 2 GB of RAM like a VM? Try it. Issue looks like there is not enough processing power.

Hi,
After long time spend digging still I am not able to solve it, any advice for above issues.

only solution I have now is to restart server.

Note:
I am using wsl ubuntu under windows 10 home.

rgds
Nofal

HI nmami,

you can resolve your problem? i am also using WSL under Windows 10… it unable to send email.

Regards,
Doni

Is there any way to stop a selected background jobs?

Was facing this issue after upgrading to v12.12.0 and this fixed my problem. Thanks.

hey @nmami facing same issue when I migrated V12 from another server to AWS. Did as you mentioned above but the background worker starts queuing job every 2 minutes. Any additional help would be appreciated.

Server is 8 core, 32 GB RAM on AWS.

Thanks

Hi,
No unfortunately I lost interest as usual.

Rgds
Nofal

Did you found a solution to stop a particular background job?

No I did not. But I’ve forgotten about it and see no issue until now.

@rahy Thank you for your reply and i found the solution to stop a selected background job with its id.(In V14, there is a doctype rq job)

for <v14, we can just copy the function.

1 Like