JobTimeoutException: Job exceeded maximum timeout value (300 seconds)

Frappe Framework Bug
12

Thanks Clarke, Yea I have tried searching the forum and this post was the closest to the issue I was facing.

I managed to partially resolve the issue by setting a default sent email account. Once that was set then the Admin could send the emails (notification emails were working fine). I think the user specific emails are not properly configured and I’ll check there was some mistake in setting it.

Best,
AJ

1 Like
13

Did you install erpnext from scratch or did use a prepaid image from google cloud or other sources?

14

HI Jermaine, I installed it from scratch.

15

Yea the individual ones are still not working.

16

I’m also getting this error for a custom background job. Is there a setting where we can extend the limit to more than 5 min?

17

Somewhere I read that you can do that in site_config.json. Let me see if I can find it.

18

With gmail my experience be advised: Less than 15 minutes between polls to check and pull mail, and their service will ignore your request for some interval

frappe@erpnext:~/frappe-bench$ bench set-config scheduler_interval 900

1 Like
19

I couldn’t find this setting anywhere. My payroll processing is still timing out. here is my trace -

Site: site1.local
Form Dict: {
 "cmd": "runserverobj",
 "docs": "{\"creation\":\"2016-06-16 11:34:07.761273\",\"doctype\":\"Process Payroll\",\"owner\":\"Administrator\",\"modified_by\":\"Administrator\",\"idx\":\"0\",\"docstatus\":0,\"name\":\"Process Payroll\",\"modified\":\"2016-06-16 11:34:07.761273\",\"salary_slip_based_on_timesheet\":0,\"__last_sync_on\":\"2017-09-04T04:13:50.592Z\",\"posting_date\":\"2017-09-04\",\"start_date\":\"2017-08-01\",\"end_date\":\"2017-08-31\",\"payroll_frequency\":\"Monthly\",\"company\":\"AH Private Limited\",\"__unsaved\":1,\"cost_center\":\"Main - AHPL\"}",
 "method": "create_salary_slips"
}
Request Error
Traceback (most recent call last):
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/app.py", line 55, in application
    response = frappe.handler.handle()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/handler.py", line 21, in handle
    data = execute_cmd(cmd)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/handler.py", line 52, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 907, in call
    return fn(*args, **newargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/handler.py", line 80, in runserverobj
    frappe.desk.form.run_method.runserverobj(method, docs=docs, dt=dt, dn=dn, arg=arg, args=args)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/desk/form/run_method.py", line 22, in runserverobj
    doc.check_if_latest()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 517, in check_if_latest
    where doctype=%s and field='modified' for update''', self.doctype)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/database.py", line 138, in sql
    self._cursor.execute(query, values)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 42, in defaulterrorhandler
    raise errorvalue
OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')```

Any help?
20

This might help you troubleshoot your issue - your timeout relates to a database update that is a separate context from a JobTimeException

frappe@erpnext:~/frappe-bench$ mysql -u root -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5263
Server version: 10.0.29-MariaDB-1~trusty mariadb.org binary distribution

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mysql]> show variables like 'innodb_lock_wait_timeout';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| innodb_lock_wait_timeout | 50    |
+--------------------------+-------+
1 row in set (0.11 sec)
21

Hi

thanks for the updates @clarkej John, this helped me to fix my same problem I received today with no email being sent or received.

I have not had any errors show up in normal usage or issues with scheduled processes in the web broswer.

But when i ran ‘show variables like ‘innodb_lock_wait_timeout’;’ in my server mariasql prompt, I got a result as so.

Database changed
MariaDB [mysql]> show variables like ‘innodb_lock_wait_timeout’;
±-------------------------±------+
| Variable_name | Value |
±-------------------------±------+
| innodb_lock_wait_timeout | 50 |
±-------------------------±------+
1 row in set (0.11 sec)

Obviously there is a lock in Mariasql which is stopping emails from being sent or received as per the forum post. I do not use Gmail, I have my own Exim server setup sensibly in the UK. But obviously something caused the innodb_lock still unknown, maybe spam .

I have now done:-

bench set-config scheduler_interval 600 instead of the default value of 300

to increase the timeout period, as I had another very similar problem about 3 months ago. At the time, I wasn’t clear what the issue was, but now I’m sure this was the same issue stopping email.

How should I go about calculating or estimating what a sensible scheduler interval may be?

I have a support contract with Frappe as this relates to my own server and asked if there’s anything they can do at all to help here, as this issue has occured for a few of us recently, and it’s not an easy one to debug and not all obvious what is stopping email. In the end just restarting mariadbsql fixed it for me, but that bit is easy .

22

Hi Julian,

“How should I go about calculating or estimating what a sensible scheduler interval may be?”

The provider decides on performance etc so I would say trial and error.

Here is the scoop with for example gmail

23

Hi John

Thanks for this. Just so I’m clear, you are saying basically that the email polling interval is affecting how the email scheduler via mysql in ERPNext works and thus if you have an email ie spam perhaps that is causing a timeout that this affects everything further down the chain ?

The mail server is wholly under my control as it uses Symbiosis https://symbiosis.bytemark.co.uk/docs/ch-mailreference.html so I can make changes in teh config if this will help.

Any thoughts or help you can offer are very gratefully accepted :wink:

24

scheduler_interval dictates how often the scheduler polls, to pull or push emails waiting in the queue at either client and server end.

So if the scheduler polls say every 60 seconds and the database has a lock timeout of say 30 or 50 seconds, then yes the ‘slow’ database might cause a bottleneck so that emails with say a large attachment do not transit within the lock timeout interval. You might need to add RAM or more cpu power or limit the size of emails.

Does this help answer your question?

My response to kirthi’s post # 19 did not relate to email poll frequency but instead to lock wait timeout…

1 Like
25

Hey, I’m struggling with the similar problem (unable to pull emails from gmail to erpnext) and mine is actually an instance created from an image. Do I need to change anything if this is the case?

Versions:
Frappe Framework: v8.0.61 (master)
ERPNext: v8.0.42 (master)

Error log list:
Title: pull_from_email_account
Error:
{‘retry’: 0, ‘log’: <function log at 0x7f0ba1a43140>, ‘site’: u’site1.local’, ‘event’: u’all’, ‘method_name’: u’pull_from_email_account’, ‘method’: <function pull_from_email_account at 0x7f0ba1202f50>, ‘user’: u’Administrator’, ‘kwargs’: {‘email_account’: u’Vishesh’}, ‘async’: True, ‘job_name’: u’pull_from_email_account|Vishesh’}
Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py”, line 65, in execute_job
method(**kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 698, in pull_from_email_account
email_account.receive()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 253, in receive
emails = email_server.get_messages()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 136, in get_messages
self.retrieve_message(message_meta, i+1)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 271, in retrieve_message
self.imap.uid(‘STORE’, message_meta, ‘+FLAGS’, ‘(\SEEN)’)
File “/usr/lib/python2.7/imaplib.py”, line 772, in uid
typ, dat = self._simple_command(name, command, *args)
File “/usr/lib/python2.7/imaplib.py”, line 1087, in _simple_command
return self._command_complete(name, self._command(name, *args))
File “/usr/lib/python2.7/imaplib.py”, line 909, in _command_complete
typ, data = self._get_tagged_response(tag)
File “/usr/lib/python2.7/imaplib.py”, line 1016, in _get_tagged_response
self._get_response()
File “/usr/lib/python2.7/imaplib.py”, line 928, in _get_response
resp = self._get_line()
File “/usr/lib/python2.7/imaplib.py”, line 1026, in _get_line
line = self.readline()
File “/usr/lib/python2.7/imaplib.py”, line 1188, in readline
return self.file.readline()
File “/usr/lib/python2.7/socket.py”, line 447, in readline
data = self._sock.recv(self._rbufsize)
File “/usr/lib/python2.7/ssl.py”, line 341, in recv
return self.read(buflen)
File “/usr/lib/python2.7/ssl.py”, line 260, in read
return self._sslobj.read(len)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/rq/timeouts.py”, line 51, in handle_death_penalty
‘value ({0} seconds)’.format(self._timeout))
JobTimeoutException: Job exceeded maximum timeout value (300 seconds)

26

“Do I need to change anything if this is the case?”

Yes, the default 300 is too frequent to poll and pull from gmail - try 900 as noted here JobTimeoutException: Job exceeded maximum timeout value (300 seconds) - #18 by clarkej

27

Hi,

I changed the scheduler_interval to 1800 using below command
bench set-config scheduler_interval 1800

Still I am facing same issue “Task exceeded maximum timeout value (300 seconds)”. My scheduled job is marked as Complete without any error because of this.

Can someone please help.

28

@GouthamM As frappe.enqueue method has some parameters which you can use for your job.
You can do it in two ways:-

  1. You can define custom parameter queue=‘long’

frappe.enqueue(method, queue=‘long’, **kwargs)

2.Or You can just pass a parameter timeout=1800

frappe.enqueue(method, timeout=1800, **kwargs)

29

Seems below code can change default timeout

try to set below code in your common_site_config

"workers": {
    "default": {
        "timeout": 900
    },
    "short": {
        "timeout": 900
    }
 }

Then your background job default timeout will be change to 900 seconds

30

get error: “unsupported operand type(s) for //: ‘str’ and ‘int’”
solution: “unsupported operand type(s) for //: ‘str’ and ‘int’”

31

This worked for me,

sudo nano site_config.json

Then add “scheduler_interval”: 600, without quote for value.