Email Account Configuration OAuth from Microsoft Office 365

revant_one · February 22, 2023, 5:08pm

the logic doesn’t poll.

whenever get_active_token() is called,

it’ll fetch token cache
check if access token is expired
- if access token is expired, it’ll request new bearer token using refresh token, new bearer token updates the token cache and expiry of token cache
- if valid returns access token from cache

frappe/frappe/blob/3406b6d7527be04dc170d6fd0572ef786a5be144/frappe/integrations/doctype/connected_app/connected_app.py#L107-L124


      
          	def get_active_token(self, user=None):
          		user = user or frappe.session.user
          		token_cache = self.get_token_cache(user)
          		if token_cache and token_cache.is_expired():
          			oauth_session = self.get_oauth2_session(user)
          
          			try:
          				token = oauth_session.refresh_token(
          					body=f"redirect_uri={self.redirect_uri}",
          					token_url=self.token_uri,
          				)
          			except Exception:
          				self.log_error("Token Refresh Error")
          				return None
          
          			token_cache.update_data(token)
          
          		return token_cache

Jecintha · February 23, 2023, 5:06am

@Jiri_Sir
Can you please share the full error log

Jiri_Sir · February 23, 2023, 5:35am

Of course @Jecintha,
here are two errors in one time, reapeting every 4th minute. But exist time gaps without errors.

First error:

  File "apps/frappe/frappe/email/oauth.py", line 44, in connect
    self._connect_imap()
      self = <frappe.email.oauth.Oauth object at 0x7efd8a6f6710>
  File "apps/frappe/frappe/email/oauth.py", line 72, in _connect_imap
    self._conn.authenticate(self._mechanism, lambda x: self._auth_string)
      self = <frappe.email.oauth.Oauth object at 0x7efd8a6f6710>
  File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
    raise self.error(dat[-1].decode('utf-8', 'replace'))
      self = <frappe.email.receive.Timed_IMAP4_SSL object at 0x7efd8a6f5c90>
      mechanism = 'XOAUTH2'
      authobject = <function Oauth._connect_imap.<locals>.<lambda> at 0x7efd8b717760>
      mech = 'XOAUTH2'
      typ = 'NO'
      dat = [b'AUTHENTICATE failed.']
imaplib.error: AUTHENTICATE failed.

And second one:

  File "apps/frappe/frappe/email/doctype/email_account/email_account.py", line 494, in get_inbound_mails
    email_server = self.get_incoming_server(in_receive=True, email_sync_rule=email_sync_rule)
      process_mail = <function EmailAccount.get_inbound_mails.<locals>.process_mail at 0x7efd909edf30>
      email_sync_rule = 'UNSEEN'
      mails = []
      self = <EmailAccount: Jiri Sir>
  File "apps/frappe/frappe/email/doctype/email_account/email_account.py", line 208, in get_incoming_server
    self.check_email_server_connection(email_server, in_receive)
      self = <EmailAccount: Jiri Sir>
      in_receive = True
      email_sync_rule = 'UNSEEN'
      oauth_token = <exception while printing> Traceback (most recent call last):
          File "env/lib/python3.10/site-packages/traceback_with_variables/core.py", line 222, in _to_cropped_str
            raw = print_(obj)
        TypeError: _get_traceback_sanitizer.<locals>.<listcomp>.<lambda>() takes 0 positional arguments but 1 was given
        
      args = {'email_account_name': 'Jiri Sir', 'email_account': 'Jiri Sir', 'host': 'outlook.office365.com', 'use_ssl': 1, 'use_starttls': 0, 'username': 'jiri.sir@beeinside.com', 'use_imap': 1, 'email_sync_rule': 'UNSEEN', 'incoming_port': 993, 'initial_sync_count': '250', 'use_oauth': True, 'access_token': 'eyJ0eXAiOiJKV1QiLCJub25jZSI6IjYyd29Wc0dkYmo1WFd6R2ljTGs3aUxuVldMUVJkREh3clpBa3k2U1pMV1EiLCJhbGciOiJSUzI1NiIsIng1dCI6Ii1LSTNROW5OUjdiUm9meG1lWm9YcWJIWkdldyIsImtpZCI6Ii1LSTNROW5OUjdiUm9meG1lWm9YcWJIWkdldyJ9.eyJhdWQiOiJodHRwczovL291dGxvb2sub2ZmaWNlLmNvbSIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0Lzc4NmJjMGZmLTlmODUtNDc1Mi1hOTM3LTBiZjY3Mzc2MDY2NS8iLCJpYXQiOjE2NzcxMjA5MTksIm5iZiI6MTY3NzEyMDkxOSwiZXhwIjoxNjc3MTI1MjYwLCJhY2N0IjowLCJhY3IiOiIxIiwiYWlvIjoiQVZRQXEvOFRBQUFBZ3FDeFMzRjlJQVJubzNMUlNRL1ZaUzV1bHQwUis0RFpQNjhqSzg2aGVxV096Q3hKZjdEUzhjRXRTSE4vb3d4ZGtaRDM3ZGFBczhHRUNWS0gxcm05ZnZKSWtiWGRBVXEyNFFoN2dRb3grYWs9IiwiYW1yIjpbInB3ZCIsIm1mYSJdLCJhcHBfZGlzcGxheW5hbWUiOiJlcnBuZXh0X2pzIiwiYXBwaWQiOiJjMjZkM2Yw...
      email_server = <frappe.email.receive.EmailServer object at 0x7efd8aa55300>
  File "apps/frappe/frappe/email/doctype/email_account/email_account.py", line 246, in check_email_server_connection
    frappe.throw(cstr(e))
      self = <EmailAccount: Jiri Sir>
      email_server = <frappe.email.receive.EmailServer object at 0x7efd8aa55300>
      in_receive = True
      auth_error_codes = ['authenticationfailed', 'loginfailed']
      other_error_codes = ['err[auth]', 'errtemporaryerror', 'loginviayourwebbrowser']
      all_error_codes = ['authenticationfailed', 'loginfailed', 'err[auth]', 'errtemporaryerror', 'loginviayourwebbrowser']
      message = 'authenticatefailed.'
  File "apps/frappe/frappe/__init__.py", line 525, in throw
    msgprint(
      msg = 'AUTHENTICATE failed.'
      exc = <class 'frappe.exceptions.ValidationError'>
      title = None
      is_minimizable = False
      wide = False
      as_list = False
  File "apps/frappe/frappe/__init__.py", line 493, in msgprint
    _raise_exception()
      title = None
      as_table = False
      as_list = False
      indicator = 'red'
      alert = False
      primary_action = None
      is_minimizable = False
      wide = False
      sys = <module 'sys' (built-in)>
      out = {'message': 'AUTHENTICATE failed.', 'title': 'Message', 'indicator': 'red', 'raise_exception': 1}
      _raise_exception = <function msgprint.<locals>._raise_exception at 0x7efd8a5e9870>
      _strip_html_tags = <functools._lru_cache_wrapper object at 0x7efd8a498460>
      inspect = <module 'inspect' from '/usr/lib/python3.10/inspect.py'>
      msg = 'AUTHENTICATE failed.'
      raise_exception = <class 'frappe.exceptions.ValidationError'>
      strip_html_tags = <function strip_html_tags at 0x7efd920a3370>
  File "apps/frappe/frappe/__init__.py", line 442, in _raise_exception
    raise raise_exception(msg)
      inspect = <module 'inspect' from '/usr/lib/python3.10/inspect.py'>
      msg = 'AUTHENTICATE failed.'
      raise_exception = <class 'frappe.exceptions.ValidationError'>
frappe.exceptions.ValidationError: AUTHENTICATE failed.

Jecintha · February 23, 2023, 7:02am

@Jiri_Sir
We had the same problem before we fixed it.
Can you please share the below screen shots?

Setup of email accounts
On your app registration, users and groups in the enterprise application.

Jiri_Sir · February 23, 2023, 8:06am

Here you are @Jecintha:
1.

Users and groups in the enterprise application:

Snímek obrazovky 2023-02-23 v 9.02.392642×592 53.9 KB

Thank you very much.

Jecintha · February 23, 2023, 8:40am

@Jiri_Sir

Change the password on your Office 365 email account.
Remove your token cache.
After 15 minutes, click API-authorize to create the token.

Jiri_Sir · February 23, 2023, 9:03am

@Jecintha, thank you.
I will do that and I will let you know.

Unfortunately, errors are here again. What type of the Token URI do you use, please? v1 or v2?

Jecintha · February 23, 2023, 12:48pm

@Jiri_Sir
We are used to OAuth 2.0 token endpoint (v1) token URI

Patrick.St · February 23, 2023, 1:13pm

Hi @Jecintha ,
I guess you are also using v1 for authorization URI? Have you set anything at “Revocation URI” and “Introspection URI”?

Jiri_Sir · February 23, 2023, 1:21pm

Hi @revant_one ,
thanks for the code preview, it seems clear.
What do you thing about no error time gaps, please?
Here error gaps with ca. 50 minutes duration and no error gaps with ca. 1,5 hours duration:

50 minutes - errors
1,5 hours - no errors
50 minutes - errors
1,5 hours - no errors
50 minutes - errors
1,5 hours - no errors

etc…

Thanks for your opinion in advance.

Jiri_Sir · February 23, 2023, 1:22pm

Thanks @Jecintha, I will try it.

Can you share screen shot from Azure → Users → “your user” → Assigned roles, please?

revant_one · February 23, 2023, 11:06pm

introspection endpoint depends on provider.

I think Microsoft doesn’t have introspection, they recommend to verify and use id_token.

Patrick.St · February 24, 2023, 6:32am

Another thing that came to my mind: As I understand the offline_access (token refresh) is working well for @revant_one and @Jecintha , right? In this discussion it seem that @Jiri_Sir and we have issues with the setup.

What kind of Azure AD license do you have? You can see that at Microsoft Azure
We are using right now “Azure AD Free” tier. Based on the documentation, that should be ok (10 apps are included). But who knows, what in-detail difference there might be in addition…

Jiri_Sir · February 24, 2023, 9:07am

Hello all @Patrick.St , @avc, @Jecintha, @revant_one, @amjithlal,
I have found temporary solution for my issue.
I have added “small” server side script, that is hour scheduled. It get new tokens for all emails in array without taking care about expiration time of the previous tokens.

email_users = ['jiri.sir@bee.com','xyz@bee.com','abc@bee.com','xyt@bee.com'];
for email_user in email_users:
    connected_app = frappe.get_doc('Connected App', 'f28440895f');
    oauth_session = connected_app.get_oauth2_session(email_user);
    token_cache = connected_app.get_token_cache(email_user)
    token = oauth_session.refresh_token(body=f"redirect_uri={connected_app.redirect_uri}",token_url=connected_app.token_uri,);
    token_cache.update_data(token);

It is inspired by get_active_token function in connected_app.py in @revant_one comment: Email Account Configuration OAuth from Microsoft Office 365 - #21 by revant_one

I think there is a bug in frappe/token_cache.py at develop · frappe/frappe · GitHub
in:

	def get_expires_in(self):
		modified = frappe.utils.get_datetime(self.modified)
		expiry_utc = modified.astimezone(pytz.utc) + timedelta(seconds=self.expires_in)
		now_utc = datetime.utcnow().replace(tzinfo=pytz.utc)
		return cint((expiry_utc - now_utc).total_seconds())

	def is_expired(self):
		return self.get_expires_in() < 0

My actual tokens has been created in 8:48:29 and has expires in 4578 second, it means it will expire at 10:04:47. Now is ca. 10:05:00 but "is_expired() function gives me still “false” and the get_expires_in function gives me 3600 seconds, that is next one hour. But real token is expired.
This is the reason why exist time gaps, they took 52 minutes in error logs + plus 2 x 4 minutes (internal timer) and it is exactly 1 hour.

Is it problem of the time zones? My timezone is UTC + 1, Czech Republic.

avc · February 25, 2023, 9:16pm

Sorry! I have this thread silenced until now (i don’t know why …)
Connected app is working for us without issues. In this tenant there is a free AzureAD suscription (associated to M365 plan …). We are using v2 OAUTH endpoint …

I remember some kind of fight with scopes.
Connected app scopes:

Token scopes (automatic setted)

Hope this helps.

revant_one · February 26, 2023, 5:56am

for me it worked during development and manual testing.

can you share your timezones?

system timezone as well as timezone from frappe/ERPNext > System Settings?

anyone actively using the feature can share the timezone, it may help in configuration and debugging.

Jiri_Sir · February 26, 2023, 7:55am

Hi @revant_one,
I am “Europe/Prague”.

revant_one · February 26, 2023, 8:44am

That seems to be from “System Settings”

What is contents of /etc/localtime or output of date command.

@avc can you confirm yours? is “System Settings” and “local timezone” same or not same?

Jiri_Sir · February 26, 2023, 9:02am

@revant_one, our erpnext is hosted on the frappecloud. How can I get important informations here?

avc · February 26, 2023, 10:08am

Hi. Yes, we have the same timezone on Frappe System Settings and OS.