Email inbox fetch error

[Oskar]

Hi,

We are getting some errors when ERPNext trying to fetch emails from IMAP account.

The errors listed in Error Logs are the nexts:

email_account.receive

Traceback (most recent call last):
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 280, in receive
communication = self.insert_communication(msg, args=args)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 337, in insert_communication
email = Email(raw)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 372, in init
self.parse()
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 393, in parse
self.process_part(part)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 445, in process_part
self.text_content += self.get_payload(part)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 489, in get_payload
charset = self.get_charset(part)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 484, in get_charset
charset = chardet.detect(cstr(part))[‘encoding’]
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/chardet/init.py”, line 34, in detect
‘{0}’.format(type(byte_str)))
TypeError: Expected object of type bytes or bytearray, got: <class ‘str’>

pull_from_email_account

{‘e’: InternalError(1366, “Incorrect string value: ‘\xE1tico,…’ for column _1bd3e0294da19198.tabUnhandled Email.raw at row 1”), ‘method_name’: ‘pull_from_email_account’, ‘log’: <function log at 0x7fea1c37f378>, ‘retry’: 0, ‘is_async’: True, ‘user’: ‘Administrator’, ‘kwargs’: {‘email_account’: ‘Oskar’}, ‘job_name’: ‘pull_from_email_account|Oskar’, ‘event’: ‘all’, ‘method’: <function pull_from_email_account at 0x7fea1c325ea0>, ‘site’: ‘erp.loonfy.com’}
Traceback (most recent call last):
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 280, in receive
communication = self.insert_communication(msg, args=args)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 337, in insert_communication
email = Email(raw)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 372, in init
self.parse()
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 393, in parse
self.process_part(part)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 445, in process_part
self.text_content += self.get_payload(part)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 489, in get_payload
charset = self.get_charset(part)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/receive.py”, line 484, in get_charset
charset = chardet.detect(cstr(part))[‘encoding’]
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/chardet/init.py”, line 34, in detect
‘{0}’.format(type(byte_str)))
TypeError: Expected object of type bytes or bytearray, got: <class ‘str’>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/utils/background_jobs.py”, line 99, in execute_job
method(**kwargs)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 724, in pull_from_email_account
email_account.receive()
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 289, in receive
self.handle_bad_emails(email_server, uid, msg, frappe.get_traceback())
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 323, in handle_bad_emails
unhandled_email.insert(ignore_permissions=True)
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/model/document.py”, line 244, in insert
self.db_insert()
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 324, in db_insert
), list(d.values()))
File “/home/frappe-user/frappe-bench/apps/frappe/frappe/database/database.py”, line 156, in sql
self._cursor.execute(query, values)
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 170, in execute
result = self._query(query)
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/cursors.py”, line 328, in _query
conn.query(q)
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 732, in _read_query_result
result.read()
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 1075, in read
first_packet = self.connection._read_packet()
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/connections.py”, line 684, in _read_packet
packet.check_error()
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/protocol.py”, line 220, in check_error
err.raise_mysql_exception(self._data)
File “/home/frappe-user/frappe-bench/env/lib/python3.6/site-packages/pymysql/err.py”, line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1366, “Incorrect string value: ‘\xE1tico,…’ for column _1bd3e0294da19198.tabUnhandled Email.raw at row 1”)

Anyone have some suggest about?

Thanks for all.

[clarkej]

"pymysql.err.InternalError: (1366, “Incorrect string value: ‘\xE1tico,…’ for column _1bd3e0294da19198 . tabUnhandled Email . raw at row 1”)
"

An email you received is 4-byte encoded while your database is possibly set to accept a maximum 3-byte encoding? So you need to set your database to accept utf8mb4.

Check these:

https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html

What is the difference between utf8mb4 and utf8 charsets in MySQL?

This other thread should help too

Pull_from_email_account + email_account.receive errors

Hello! I have a weird email error, could you help me out? Should I open a github issue? I have this email problem: pull_from_email_account Error {'e': InternalError(1366, "Incorrect string value: '\\xE9b \\xDCl\\xF5...' for column `_1bd3e0294da19198`.`tabUnhandled Email`.`raw` at row 1"), 'method_name': 'pull_from_email_account', 'log': <function log at 0x7f1ba1523e18>, 'retry': 0, 'is_async': True, 'user': 'Administrator', 'kwargs': {'email_account': 'Replies'}, 'job_name': 'pull_from_ema…

V12.1 EMAIL pull from server not working properly

Thank you , make my today happy. Finally updated my system to UBUNTU 18.04, tough it helps. Nothing changed. Dig it to charset in mysql database. I had to allow in mariadb.cnf the # out lines and let set the charset to [mysqld] charser-set-server=utf8mb4 (it was utf8) collation-server=utf8mb4_unicode_ci (it was general_ci) Now it works! No more error in list. Please close this subject.

[krnkris]

@clarke
Getting understand thank you!

It seem back in time I was not aware to convert all my databases to UTF8mb4.

Now I search how to do it correctly.
Will inform …

At least now the reason is clarified.

[clarkej]

@krnkris

It seem back in time I was not aware to convert all my databases to UTF8mb4.

Yes the current install advises that The Hitchhiker's Guide to Installing Frappe on Linux · frappe/frappe Wiki · GitHub

[krnkris]

@clarkej
The problem is if install a new instance and
restore an „old“ database consist of regular UTF8 character encoding.

It seems is my case now.

[clarkej]

From my reading UTF8 is 3-byte so to restore that data to a database set to utf8mb4 should ‘automatically’ work but best use a test environment to validate that theory!

This may help? How to easily convert utf8 tables to utf8mb4 in MySQL 5.5 - Database Administrators Stack Exchange

[krnkris]

@clarkej

Thank You!
I alter badly handled TABLE in this case it is solved on my side.