Custom apps for cloud users [extend ERPNext with microservices]

revant_one · September 21, 2017, 7:56am

Traditional Custom App

It is part of the same server, database, site, environment, assets.
Upgrades along with other apps.
Migration and retirement of the app is complex and endangers whole system
Can compromise data integrity or system security
Privileges of app to access system and user resource cannot be restricted by user
Based on Frappe Framework only and communicates via Python and JS api

Microservice Custom App

It is on different server with own database, site, environment, assets etc.
Can have its own release cycle
Migration and retirement of the app is independent of production ERP
In case of compromise or failure, ERP Data and system is safe
User has the control to allow or disallow app from using ERP resource. App access can be revoked
Can be based on any framework including Frappe on any device and communicates via OAuth 2.0 and REST API

Why

Frappe ERPNext is fast changing
Not upgrading ERPNext as per the release schedule means missing out on lot of features and fixes
Custom app blocks upgrade and may have migration complexities
Custom app fails and takes down whole system
What is not worth going into core as a pull request may be unreviewed patch work or hack

How

Setup Social Login such that the primary ERPNext server is in your Frappe Server URL. e.g. SaaS Server URL
The microservice always signs in using primary ERPNext account
OAuth2Authenticator for android, ERPNext Connector or equivalent OAuth 2.0 Client is installed on Microservice app
If microservice is a frappe app, relevant Link fields must show upstream doctypes. ERPNext Connector is hacking this in with Connector DocTypes which query upstream server db for instances saved there instead of showing local data
Whenever user enters such link field the upstream docnames start showing and latest docnames are saved locally for saving data in link field
If there is any change on upstream server, Webhook can inform it to app
Even Android app can use login via Frappe. Check out MN Technique’s App Showcase for Android app using this approach

Note: Read Documentation Frappe > Integration for more information

Signed in to Primary ERPNext Server with nothing installed e.g. SaaS Server

Item List on ERPNext

Custom App login with Frappe

Signed into Custom App

Upstream Items in Custom App Query

Service Providers can now deliver customization / integration to cloud users and promote cloud usage.
Anyone interested can send PR to erpnext_connector and extend it as needed.

chdecultot · September 21, 2017, 1:37pm

Thanks a lot @revant_one, you did an amazing job!

I have a silly question: when I try to add the Frappe Server URL in my Primary Server, I get the following error:

Unable to make request to the Frappe Server URL

If my understanding is correct, this URL should be the URL of the Primary Server ?

When I test:
requests.get("http://primary-server-url/api/method/frappe.handler.version", timeout=5) in the console I correctly get {"message":"8.10.8"}

But when I add http://primary-server-url in the UI and try to save it doesn’t work.

I also configured the “host_name” in site_config.json, but with no success…

And when I add the URL of another server, it works fine, so the issue is only when it is it’s own URL.

Do you, by any chance, have any idea of what I have missed ?

revant_one · September 21, 2017, 1:42pm

I tried it on trial account on erpnext.com and it is working for the subdomain configured there.
I was about to remove that validation, but it is working on erpnext cloud. So I kept it.

What I found was, If I add the hostname to /etc/hosts it doesn’t timeout on validation.
It works on appropriately configured domain/hostname I guess.
frappe.db.set_value your way forward if you just need to test! e.g. https://github.com/frappe/frappe/blob/develop/frappe/tests/ui/test_oauth20.py#L18

The url is required in id_token. Decrypted id_token has the url of issuer. iss, Standard Fields

chdecultot · September 21, 2017, 2:13pm

Thanks a lot for your answer.

Indeed it works fine on erpnext.com so it must be a configuration issue on my remote development server.

The strange thing is that it works fine in the console but not from Frappé directly…

I haven’t been able to find it yet, but I will post the solution once I have it.

And thanks for the frappe.db.set_value tip

chdecultot · September 21, 2017, 6:44pm

Still haven’t found the solution to connect via Oauth on my dev server, but I have been able to make it work through erpnext.com account.

I have another question regarding erpnext_connector: how are we supposed to setup our doctype to get the document list from the master server.

I tried something like:

But it doesn’t work automatically.

I also saw in your code that there is a hook called “erpnext_connector_search_fields”. Can you please tell me what it is supposed to do ?

Thanks a lot for your help!

revant_one · September 22, 2017, 7:18am

check this code, It has 3 examples,

All fields for ERPNext Connector doctypes have special query
Specific Link field with query and filter
Child table query example

github.com

mntechnique/insider_frappe/blob/develop/insider_frappe/insider_frappe/doctype/event_commission/event_commission.js

// Copyright (c) 2017, MN Technique and contributors
// For license information, please see license.txt

frappe.ui.form.on('Event Commission', {
	refresh: function(frm, cdt, cdn) {
		//All Link Fields with ERPNext Connector Doctypes
		$.each(frm.fields_dict, function(fieldname, field ){
			if (field.df.fieldtype == "Link" && (field.df.options.indexOf("ERPNext ") == 0)){
				frm.set_query(fieldname, function(){
					return {
						query:"erpnext_connector.api.upstream_doc_query"
					}
				});
			}
		});

		// Specific Field with query (upstream)
		frm.set_query("company", function(){
			return {
				query:"erpnext_connector.api.upstream_doc_query",

This file has been truncated. show original

Intended hook is for showing upstream searchfield for microservice app, because in the ERPNext Connector doctype there is only name field. Thus you cannot add search fields like normal doctype/customize form

"erpnext_connector_search_fields" = {
	"Contact": ["first_name", "last_name"],
	"DocType" ["field_1", "field_2"]
}

The connector is evolving, if I find a way to not use Conntector Doctypes at all then it will be ideal.

chdecultot · September 25, 2017, 6:08pm

Hi @revant_one,

I’ve managed to validate the “Social Login Keys” on the server. I just needed to increase the number of gunicorn workers (Bench is in Production Mode)…

If anyone is interested: https://discuss.frappe.io/t/how-to-increase-number-of-gunicorns-workers

Now I encounter another issue when trying to access the upstream document from my custom app (with an expired token):

Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 62, in application
    response = frappe.api.handle()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/api.py", line 53, in handle
    return frappe.handler.handle()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 22, in handle
    data = execute_cmd(cmd)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 53, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 923, in call
    return fn(*args, **newargs)
  File "/home/frappe/frappe-bench/apps/erpnext_connector/erpnext_connector/api.py", line 14, in login_via_frappe
    info = get_info_via_oauth("frappe", code, json.loads)
  File "/home/frappe/frappe-bench/apps/erpnext_connector/erpnext_connector/account_manager.py", line 25, in get_info_via_oauth
    session = flow.get_auth_session(**args)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/rauth/service.py", line 556, in get_auth_session
    session = self.get_session(self.get_access_token(method, **kwargs))
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/rauth/service.py", line 542, in get_access_token
    access_token, = process_token_request(r, decoder, key)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/rauth/service.py", line 24, in process_token_request
    raise KeyError(PROCESS_TOKEN_ERROR.format(key=bad_key, raw=r.content))
KeyError: 'Decoder failed to handle access_token with data as returned by provider. A different decoder may be needed. Provider returned: {"error":"unsupported_grant_type"}'

Here is my app advanced setup:

So grant_type is identical to the one hard-coded in the connector:

And token setup in erpnext_connector:

Do you know if I’m missing something again ?

revant_one · September 25, 2017, 7:30pm

Try with query token Always. It is working on one of the server I’ve setup. In this case it validates access token everytime get_auth_token is calleby making a request to openid_profile endpoint.

Have you successfully logged in via Frappe?

I’ll check on expiry part.

Edit: both seem to work when I tested. Always and On Expiry.

Are you logging in with Administrator user? It won’t work with admin

chdecultot · September 26, 2017, 8:40am

Thanks @revant_one,

Actually that’s the strange thing: I was able to login successfully via Frappe on Friday, but when trying again yesterday I got the above issue.

I’m trying to login with a system manager like last Friday.

I am going to check with Postman to understand where it comes from.

revant_one · September 26, 2017, 11:10am

Increasing gunicorn workers to 2 solved the problem.

chdecultot · September 26, 2017, 11:54am

I think there is an issue with the Oauth Bearer Token:

I have just deleted the existing Token and reconnected and now the connector is working fine.

The token registered on my primary server was expired, as expected, but maybe it caused an issue during the next token request.
I’ll let you know how it behaves after the next expiration.

revant_one · September 26, 2017, 12:15pm

Take a look at account_manager.py.

I’m overriding whitelisted function frappe.www.login.login_via_frappe

github.com

mntechnique/erpnext_connector/blob/develop/erpnext_connector/hooks.py#L119


      
          # }
          
          
# Testing
          # -------
          
          
# before_tests = "erpnext_connector.install.before_tests"
          
          
# Overriding Whitelisted Methods
          # ------------------------------
          #
          override_whitelisted_methods = {
          	"frappe.www.login.login_via_frappe": "erpnext_connector.api.login_via_frappe"
          }

During override the bearer token recieved with rauth is stored in Connector User Data DocType

revant_one · October 9, 2017, 5:30pm

Custom Scripts on cloud server that call endpoints on custom server will result in Access-Control error

On custom app server, setup wide open cors or restrict it to user’s cloud domain
e.g
add_header 'Access-Control-Allow-Origin' '*';
or
add_header 'Access-Control-Allow-Origin' 'user.erpnext.com';

full gist

revant_one · November 1, 2017, 5:07pm

Continue discussion here Custom apps for cloud users [extend ERPNext with microservices]