Issues are not created on email import, not all mails are being imported

ci2016 · April 3, 2017, 6:48pm

Hello there,

since approximately 1 month or more, we have this issue:

Not all mails are imported to the ERPNext database
Issues are not created when pulling mails
Leads / Opportunities are not created when pulling mails

The error is always like the following:

{'retry': 0, 'log': <function log at 0x7f332a865cf8>, 'site': u'erp.ci-commerce.com', 'event': u'all', 'method_name': u'pull_from_email_account', 'method': <function pull_from_email_account at 0x7f332a7429b0>, 'user': u'Administrator', 'kwargs': {'email_account': u'Lead'}, 'async': True, 'job_name': u'pull_from_email_account|Lead'}
Traceback (most recent call last):
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 65, in execute_job
    method(**kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 694, in pull_from_email_account
    email_account.receive()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 250, in receive
    emails = email_server.get_messages()
AttributeError: 'NoneType' object has no attribute 'get_messages'

OR

{'retry': 0, 'log': <function log at 0x7f332a865c80>, 'site': u'erp.ci-commerce.com', 'event': u'all', 'method_name': u'pull_from_email_account', 'method': <function pull_from_email_account at 0x7f332a742938>, 'user': u'Administrator', 'kwargs': {'email_account': u'Support'}, 'async': True, 'job_name': u'pull_from_email_account|Support'}
Traceback (most recent call last):
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 65, in execute_job
    method(**kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 694, in pull_from_email_account
    email_account.receive()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 250, in receive
    emails = email_server.get_messages()
AttributeError: 'NoneType' object has no attribute 'get_messages'

I thought its some problem of us, but since now we are on the master branch and have the latest ERPNext and Frappe installed, I think there is a problem in the email_account.py file.

Any help?

Thank you in advance.

makarand_b · April 7, 2017, 9:04am

@ci2016, this issue has been fixed please check [minor] minor fixes in email_account and added has_attachment in comm… by mbauskar · Pull Request #2997 · frappe/frappe · GitHub

ci2016 · April 9, 2017, 1:32pm

@makarand_b Thank you. The errors are gone, but unfortunately there are still not issues created when a new mail comes. There are not even all mails in the Inbox App.

ci2016 · April 19, 2017, 8:45am

I still don’t receive any mails with ERPNext. That might be the reason for that no tickets are created.

makarand_b · April 19, 2017, 10:15am

@ci2016,

can you share the error traceback ?

ci2016 · April 19, 2017, 10:40am

There are no errors or tracebacks at all. The mails just don’t import. Before this fix [minor] minor fixes in email_account and added has_attachment in comm… by mbauskar · Pull Request #2997 · frappe/frappe · GitHub I had the error message above.

By the way, I don’t know if its a real fix or it just avoids the error message.

ci2016 · April 20, 2017, 4:12pm

Is there any way to investigate this?

ci2016 · April 24, 2017, 3:50pm

I am still trying to fix this issue. I’ve deleted all “Unhandled Mails”. After that I tried to save the E-Mail Domain without changing anything. I get the following traceback(s):

Traceback (most recent call last):
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/desk/form/save.py", line 22, in savedocs
    doc.save()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 231, in save
    return self._save(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 281, in _save
    self.run_post_save_methods()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 790, in run_post_save_methods
    self.run_method("on_update")
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 667, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 892, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 875, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 661, in 
    fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/email/doctype/email_domain/email_domain.py", line 78, in on_update
    frappe.throw(e)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 316, in throw
    msgprint(msg, raise_exception=exc, title=title, indicator='red')
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 305, in msgprint
    message_log.append(json.dumps(out))
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: AuthenticationError('Password not found',) is not JSON serializable

Traceback (most recent call last):
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/app.py", line 56, in application
    response = frappe.handler.handle()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/handler.py", line 19, in handle
    execute_cmd(cmd)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/handler.py", line 42, in execute_cmd
    ret = frappe.call(method, **frappe.form_dict)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 907, in call
    return fn(*args, **newargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/desk/form/save.py", line 22, in savedocs
    doc.save()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 231, in save
    return self._save(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 281, in _save
    self.run_post_save_methods()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 790, in run_post_save_methods
    self.run_method("on_update")
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 667, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 892, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 875, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 661, in 
    fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/email/doctype/email_domain/email_domain.py", line 78, in on_update
    frappe.throw(e)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 316, in throw
    msgprint(msg, raise_exception=exc, title=title, indicator='red')
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 305, in msgprint
    message_log.append(json.dumps(out))
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: AuthenticationError('Password not found',) is not JSON serializable
Report this issue

Any ideas?

ci2016 · April 30, 2017, 2:01pm

By now, I found out what the issue was and is. Unfortunately I’ve bin misleaded by several bugs, which I have documented in this thread.

In the IMAP account there were more than 50 unread mails.

I found the following lines in the file /apps/frappe/frappe/email/receive.py

# WARNING: Hard coded max no. of messages to be popped
if num > 50: num = 50

These lines cause the issue. If there are more than 50 unread mails, the rest will be ignored. I understand, that importing more than 50 mails at once obviously is not a solution. So in theory every run there should be imported 50 other mails, until all the mails are done.

I realized, at least on my production and my development server, that the theory is not working.

I can’t tell you about the reason. Maybe the mails are not marked as read properly, so it keeps to import the same 50 mails every time. Does anyone know more about this issue?

EDIT: After investigating more, I found, that the IMAP Inbox is selected as ReadOnly! The mails were not marked as read. I opened a github issue for this to discuss:

https://github.com/frappe/frappe/issues/3195