Solved - Email Issues - Outgoing Not working in Certain Cases

rmehta · September 27, 2016, 5:09pm

Seems like GMail has started dropping support for transaction email:

arghya.basu · September 28, 2016, 7:45am

Thanks @rmehta.

Is it safe to assume that the current way forward would be the following for self hosted instances:

Signup for Sparkpost.
Configure the Setup->Email Accounts section in the following way:
2.1 Service field is left blank.(Is this correct?)
2.2 Enable Incoming (Configure the server details)
2.3 Enable Outgoing (Configure the server details)

On a separate issue with Gmail (Google Apps for me)

I have noticed that the “Enable Incoming” gets unchecked on its own, for my current configuration with Gmail as the service after a while.

Heres the scheduler log output:

Error 1

Method
receive.connect_pop

Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 68, in connect_pop
self.pop = Timed_POP3_SSL(self.settings.host, timeout=frappe.conf.get(“pop_timeout”))
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 422, in init
self._super.init(self, *args, **kwargs)
File “/usr/lib/python2.7/poplib.py”, line 346, in init
for res in socket.getaddrinfo(self.host, self.port, 0, socket.SOCK_STREAM):
gaierror: [Errno -2] Name or service not known

Method
pull_from_email_account

Error 2

Method
pull_from_email_account

{‘retry’: 0, ‘log’: <function log at 0x7fc3abed57d0>, ‘site’: u’site1.local’, ‘event’: u’all’, ‘method_name’: u’pull_from_email_account’, ‘method’: <function pull_from_email_account at 0x7fc3ac040500>, ‘user’: u’Administrator’, ‘kwargs’: {‘email_account’: u’Support’}, ‘async’: True, ‘job_name’: u’pull_from_email_account|Support’}
Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py”, line 61, in execute_job
method(**kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 427, in pull_from_email_account
email_account.receive()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 202, in receive
raise Exception, frappe.as_json(exceptions)
Exception: [
“Traceback (most recent call last):\n File "/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 184, in receive\n communication = self.insert_communication(raw)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 225, in insert_communication\n self.set_thread(communication, email)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 330, in set_thread\n parent.insert(ignore_permissions=True)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 213, in insert\n self._validate()\n File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 391, in _validate\n self._validate_length()\n File "/home/frappe/frappe-bench/apps/frappe/frappe/model/base_document.py", line 515, in _validate_length\n .format(reference, (df.label), max_length, value), frappe.CharacterLengthExceededError, title=(‘Value too big’))\n File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 299, in throw\n msgprint(msg, raise_exception=exc, title=title, indicator=‘red’)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 292, in msgprint\n _raise_exception()\n File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 265, in _raise_exception\n raise raise_exception, encode(msg)\nCharacterLengthExceededError: Issue ISS-00297: ‘Subject’ (Exclusive Research Fellowship Grants By Govt of India, Rs 8 Lakh\n Salary @ ITC, PhD Admissions Open @ IISER, Jobs @ Syngene, NISER, Chai\n Point, CUK, IISER + Much More) will get truncated, as max characters allowed is 140\n”
]

Error 3

Method
email_account.receive

Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 184, in receive
communication = self.insert_communication(raw)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 225, in insert_communication
self.set_thread(communication, email)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 330, in set_thread
parent.insert(ignore_permissions=True)
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 213, in insert
self._validate()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 391, in _validate
self._validate_length()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 515, in _validate_length
.format(reference, (df.label), max_length, value), frappe.CharacterLengthExceededError, title=(‘Value too big’))
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 299, in throw
msgprint(msg, raise_exception=exc, title=title, indicator=‘red’)
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 292, in msgprint
_raise_exception()
File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 265, in _raise_exception
raise raise_exception, encode(msg)
CharacterLengthExceededError: Issue ISS-00297: ‘Subject’ (Exclusive Research Fellowship Grants By Govt of India, Rs 8 Lakh
Salary @ ITC, PhD Admissions Open @ IISER, Jobs @ Syngene, NISER, Chai
Point, CUK, IISER + Much More) will get truncated, as max characters allowed is 140

rmehta · September 28, 2016, 11:25am

Maybe make Subject should be truncated before it is saved!

arghya.basu · September 29, 2016, 6:14am

Yes that’ll be helpful as we don’t have control over the subject sizes of incoming mails. How do i do this? Do i have to update a specific file?

arghya.basu · September 29, 2016, 8:00am

Its finally solved (as of now, will do rigorous testing to check).

Here’s what worked for me and what didn’t work:

My existing configuration:

I am not using a generic domain like @gmail, I am using my own domain name (e.g. @company1.com), and my current mail server is hosted with Google Apps. (I do not think this would effect the configuration)

For sending mails:

Sending Mails Configuration

1.1 Configure Sparkpost
1.2. Create an account at sparkpost.com and sign in (The username domain, etc used to sign up doesn’t matter in the configuration later on)
1.3 Go to https://app.sparkpost.com and click “Account” Then “Sending Domain”
1.4 Click New Domain
1.5 Add a domain of your choice (This step is very important, this will be the domain from which all your emails from ERPnext would be sent) - For this example, lets assume the domain is company1.com.
1.6 Once your domain is created, you’ll see three records which are awaiting aproval, Click settings and note down the details.

1.7 Log onto your hosting provider where company1.com’s DNS files are present.
1.8 Create new TXT records in the DNZ zone file (This is specific to Godaddy, for other service providers please go through - https://support.sparkpost.com/customer/portal/articles/1933318-create-and-verify-sending-domains)
1.9 Update the TXT record and create one TXT record for SPF and one for DKIM.
1.10 Once done, come back to sparkpost, and click test under SPF and DKIM. If everything went correctly, you’ll see a green tick against each record.
1.11 Verify Email - To do this , you need to have access to abuse@company1.com and postmaster@company1.com. If these accounts don’t exist, you need to create them in your mail server. Since i am using Google Apps hosted Mail server, abuse@ and postmaster@ are mail ids which we cannot create as an email account. The workaround is to go to your google apps admin page, and create two groups abuse@company1.com and postmaster1@company1.com. Add your existing mail id as a member in each of these two groups.
1.12 Once done, go back to sparkpost and click “Verify Email” and click “Send Email” next to abuse@ and postmaster@
1.13 You’ll receive the mails in your inbox, and click them to verify your account. Once done, this is what your complete setup within sparkpost should look like:

1.14 Now Click “API Keys” and generate a new key. (This is your password for configuring ERPnext). Note the key, sparkpost just shows it ONCE, and once you close the tab/window,there’s now way to retrieve it.
1.15 Click "SMTP RELAY’ - This has all the information which you require to configure ERPNExt

1.14 ERPNext → Setup-> Email Accounts → New

Service Field - Blank
Email id - Give an email id (email1@comapny1.com) (Remember we configured company1.com as the sending domain in step 1.5
Click Login id is different
Login Id - Check this data from 1.15 (Mine was SMTP_Injection)
Password - Your generated API key from 1.14
Do not check “Enable Incoming” (Sparkpost is just to send emails not receive them)
Click “Enable Outgoing”
SMTP Server - Check this data from 1.15 (Mine was smtp.sparkpostmail.com)
Use TLS - Check it
Port - Check this data from 1.15 (Mine was 587)
Default Outgoing - Check it
Always use Account’s Email ID as Sender - Check it ( The email id of users who would be using ERPNext should be @company1.com, else you’ll get an error saying

SMTPDataError: (550, '5.7.1 Unconfigured Sending Domain)

1.15 Enjoy ERPNext

All the best.

eirua · September 30, 2016, 5:56am

I tried this and it doesn’t work for me. Here’s the error I get from the Sparkpost config above:

Traceback (most recent call last):
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/desk/form/save.py", line 20, in savedocs
    doc.save()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/model/document.py", line 250, in save
    self._validate()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/model/document.py", line 369, in _validate
    self._save_passwords()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/model/base_document.py", line 582, in _save_passwords
    set_encrypted_password(self.doctype, self.name, new_password, df.fieldname)
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/utils/password.py", line 25, in set_encrypted_password
    { 'doctype': doctype, 'name': name, 'fieldname': fieldname, 'pwd': encrypt(pwd) })
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/utils/password.py", line 95, in encrypt
    cipher_suite = Fernet(encode(get_encryption_key()))
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/env/lib/python2.7/site-packages/cryptography/fernet.py", line 32, in __init__
    backend = default_backend()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/env/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 35, in default_backend
    _default_backend = MultiBackend(_available_backends())
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/env/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 22, in _available_backends
    "cryptography.backends"
AttributeError: 'EntryPoint' object has no attribute 'resolve'

Traceback (most recent call last):
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/app.py", line 58, in application
    response = frappe.handler.handle()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/handler.py", line 19, in handle
    execute_cmd(cmd)
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/handler.py", line 36, in execute_cmd
    ret = frappe.call(method, **frappe.form_dict)
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/__init__.py", line 875, in call
    return fn(*args, **newargs)
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/desk/form/save.py", line 20, in savedocs
    doc.save()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/model/document.py", line 250, in save
    self._validate()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/model/document.py", line 369, in _validate
    self._save_passwords()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/model/base_document.py", line 582, in _save_passwords
    set_encrypted_password(self.doctype, self.name, new_password, df.fieldname)
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/utils/password.py", line 25, in set_encrypted_password
    { 'doctype': doctype, 'name': name, 'fieldname': fieldname, 'pwd': encrypt(pwd) })
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/apps/frappe/frappe/utils/password.py", line 95, in encrypt
    cipher_suite = Fernet(encode(get_encryption_key()))
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/env/lib/python2.7/site-packages/cryptography/fernet.py", line 32, in __init__
    backend = default_backend()
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/env/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 35, in default_backend
    _default_backend = MultiBackend(_available_backends())
  File "/opt/bitnami/apps/erpnext/htdocs/frappe-bench/env/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 22, in _available_backends
    "cryptography.backends"
AttributeError: 'EntryPoint' object has no attribute 'resolve'

Tried a bunch of other stuff too so must be my machine. But seriously, Gmail, Sendgrid, Cpanel Mail, O365, SES, Mailgun and now Sparkpost. Haven’t got a single one authenticated with ERP next mail settings. I did just manage to send with Sparkpost via a terminal API call.

I have the most current version of the Bitnami AWS ERPNext stack (Ubuntu 14). I’ve tried following ERPNext documentation. Tried Bitnami docs to modify the site-config.json. Tried the Amazon SES documentation extensively (including stunnel & postfix).

Also tried some stuff around this error like: yt target x86-linux-native fails with 'EntryPoint' object has no attribute 'resolve' · Issue #764 · ARMmbed/yotta · GitHub

I must be daft. Probably done more harm than good by now.

EDIT: Just created my own thread for this: Unable to configure outgoing mail

arghya.basu · September 30, 2016, 8:08am

@rmehta - The issue of the mail box (default inbox) getting auto unselected (Enable incoming gets unselected on its own) is still continuing. This in turn is switching off all incoming mails.

Here are the scheduler logs:

Method
receive.connect_pop

Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 68, in connect_pop
self.pop = Timed_POP3_SSL(self.settings.host, timeout=frappe.conf.get(“pop_timeout”))
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 422, in init
self._super.init(self, *args, **kwargs)
File “/usr/lib/python2.7/poplib.py”, line 346, in init
for res in socket.getaddrinfo(self.host, self.port, 0, socket.SOCK_STREAM):
gaierror: [Errno -2] Name or service not known

Method
pull_from_email_account

{‘retry’: 0, ‘log’: <function log at 0x7fabd6a917d0>, ‘site’: u’site1.local’, ‘event’: u’all’, ‘method_name’: u’pull_from_email_account’, ‘method’: <function pull_from_email_account at 0x7fabd6bfd500>, ‘user’: u’Administrator’, ‘kwargs’: {‘email_account’: u’Support’}, ‘async’: True, ‘job_name’: u’pull_from_email_account|Support’}
Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py”, line 61, in execute_job
method(**kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 427, in pull_from_email_account
email_account.receive()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 179, in receive
incoming_mails = email_server.get_messages()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 101, in get_messages
if not self.connect():
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 42, in connect
return self.connect_pop()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 68, in connect_pop
self.pop = Timed_POP3_SSL(self.settings.host, timeout=frappe.conf.get(“pop_timeout”))
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/receive.py”, line 422, in init
self._super.init(self, *args, **kwargs)
File “/usr/lib/python2.7/poplib.py”, line 346, in init
for res in socket.getaddrinfo(self.host, self.port, 0, socket.SOCK_STREAM):
gaierror: [Errno -2] Name or service not known

My incoming mails is configured with the pop.gmail.com for my G-Suite (Gogle Apps) account.

arghya.basu · October 1, 2016, 8:49am

I changed my settings of incoming mail server from pop to imap, and tested it for 24 hours. Its working. So that’s good news.

In total email issues solved, for now!

arghya.basu · October 2, 2016, 6:42pm

Hi after 48 hours, incoming mail server isn’t working anymore.

Scheduler log:

{‘retry’: 0, ‘log’: <function log at 0x7fabd6a917d0>, ‘site’: u’site1.local’, ‘event’: u’all’, ‘method_name’: u’pull_from_email_account’, ‘method’: <function pull_from_email_account at 0x7fabd6bfd500>, ‘user’: u’Administrator’, ‘kwargs’: {‘email_account’: u’Support’}, ‘async’: True, ‘job_name’: u’pull_from_email_account|Support’}
Traceback (most recent call last):
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py”, line 61, in execute_job
method(**kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 427, in pull_from_email_account
email_account.receive()
File “/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py”, line 202, in receive
raise Exception, frappe.as_json(exceptions)
Exception: [
“Traceback (most recent call last):\n File "/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 184, in receive\n communication = self.insert_communication(raw)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 225, in insert_communication\n self.set_thread(communication, email)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py", line 330, in set_thread\n parent.insert(ignore_permissions=True)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 213, in insert\n self._validate()\n File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 391, in _validate\n self._validate_length()\n File "/home/frappe/frappe-bench/apps/frappe/frappe/model/base_document.py", line 515, in _validate_length\n .format(reference, (df.label), max_length, value), frappe.CharacterLengthExceededError, title=(‘Value too big’))\n File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 299, in throw\n msgprint(msg, raise_exception=exc, title=title, indicator=‘red’)\n File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 292, in msgprint\n _raise_exception()\n File "/home/frappe/frappe-bench/apps/frappe/frappe/init.py", line 265, in _raise_exception\n raise raise_exception, encode(msg)\nCharacterLengthExceededError: Issue ISS-00348: ‘Subject’ (Apply for PhD/MS @ IIT Madras, SRF/JRF Positions @ BITS Pilani,\r\n SRM University, IIAR, BHU, R&D Vacancies @ Syngene, Dupont, Liva\r\n Pharmaceuticals and Much More) will get truncated, as max characters allowed is 140\n”
]

@rmehta - This seems to be a message length issue, which is by default unchecking “Enable Incoming” and all incoming mails stop. It has to be checked again for the mails to start coming in.

Could you suggest where I need to make those changes.

arghya.basu · October 2, 2016, 6:54pm

I have done the following change now.

Issue - > Customise Form - > Subject Field - > Length - I changed length from 0 to 256 based on a few threads mentioned on similar issue.

If this was the solution, that’s great, I’ll check it for 48-72 hours.

arghya.basu · October 4, 2016, 7:07am

The issue still continues. Anybody has a solution for incoming mails?

arghya.basu · October 7, 2016, 11:48am

@rmehta - Any solution, workaround, which can be made to work, for incoming mails. Default inbox keeps getting unselected and mails coming to that account don’t show up in ERPNext anymore.

This is impacting resolving the issues as the mails stop showing up.

rmehta · October 7, 2016, 12:09pm

@arghya.basu Works alright for us.

Edit: Maybe you can just push a fix to truncate the subject before saving, if that is the only error you are facing.

arghya.basu · October 8, 2016, 10:10am

@rmehta. Thanks for the suggestion.

This is what I have come up with, could you suggest if this is good to go?

File : /home/frappe/frappe-bench/apps/frappe/frappe/email/doctype/email_account/email_account.py

def insert_communication(self, raw):
email = Email(raw)

  if email.from_email == self.email_id:
  	# gmail shows sent emails in inbox
  	# and we don't want emails sent by us to be pulled back into the system again
  	raise SentEmailInInbox

  communication = frappe.get_doc({
  	"doctype": "Communication",
  	**"subject":  email.subject[0:130],**
  	"content": email.content,
  	"sent_or_received": "Received",
  	"sender_full_name": email.from_real_name,
  	"sender": email.from_email,
  	"recipients": email.mail.get("To"),
  	"cc": email.mail.get("CC"),
  	"email_account": self.name,
  	"communication_medium": "Email"
  })

I have zero knowledge of python coding, so this might be absolutely wrong, but will this ensure that the email subject length is always set to a max of 130 chars?

Could you suggest an alternative change if this is wrong.

Thanks.

rmehta · October 8, 2016, 10:11am

Seems fine, try it for sometime. I also pushed a fix to develop branch!

arghya.basu · October 8, 2016, 10:13am

Ok, great thanks. I’ll keep track of this and post the feedback soon. Thank you.

arghya.basu · October 12, 2016, 12:23pm

@rmehta - The fix you pushed, worked. Thanks . https://github.com/frappe/frappe/pull/2157

imllc · October 26, 2016, 10:21am

we have the problem with incoming becoming unchecked after validated and working for a period of time will this fix the problem. we are on production how do we apply?

arghya.basu · October 26, 2016, 10:47am

Try it. It worked for me.

imllc · October 26, 2016, 10:50am

how do we apply it to production?