Microservice vs Monolith mindset

1

This post is for those doing code in Frappe / ERPNext, and NOT for those who are users.

It is really ironic that RMehta loves monolith.

But Frappe is an extremely powerful tool for Microservices development.
You create an app. Modules within the app. Doctypes within modules. And voila! You are ready to go. You may do javascript, python, html, css, and testing using the boilerplate templates provided. User permissions are built in. If fact, Frappe has a built-in OAuth server which you may use for user authentication in other applications. Virtual Doctypes even permit you to use other databases - even MongoDB.

I mean, Frappe is very powerful.

However, it is easy to mess up if you do not have the Microservices mindset - especially if you come from a monolith - Ruby type mindset where DRY - do not repeat yourself reigns.

When you do Microservices - your application should be self-contained. It should be able to run without reaching out to other apps.

For example, if you are developing an app for Property Management, it is better to have Tenant, and not use ERPNext’s Customer immediately. This way, the functionality of individual Microservices app is preserved and you do not need to think about the nuances and intricacies of other apps. So you are Agile.

When you need to use the functionalities of other apps, like if you need to use produce Sales Invoice for your Property App, you include a Linking field in the Tenant DocType to link the Tenant to a Customer DocType.

This breaks the DRY mindset - but this is very much in line with Microservices and is prevalent in the NoSQL world - which is the more recent paradigm in software development.

For example, School needs Student DocType and Health needs Patient DocType. To access the accounting functionality of ERPNext, Student may link to Customer and Patient may link to Customer.

With this mindset - Microservices mindset - development using Frappe / ERPNext can be very fast and agile.

Given the Docker / Kubernetes capability which allows Frappe / ERPNext to scale according to need, this setup has almost no equal.

What do you think?

8 Likes
2

thoughtful idea, I am interested in how to re-use other apps existing feature like what you mentioned, possible to provide sample realization based on use case?

thanks for your sharing of the ideas.

3

“I am interested in how to re-use other apps existing feature like what you mentioned,”

Each app should focus only on the functionality that it provides. Reuse of other apps existing feature is achieved by link fields to relate a DocType in one app to another DocType in the other app. Which is very easy in Frappe.

For example, in an Education app, the DocType Student would be very much concerned with grades, schedules, and the DocType Teacher would be very much concerned with Classes, Student Attendance, Exames, etc.

But in the ERPNext App, the finance department is not concerned with the grades of Students, it is concerned with the Customer (receivables and payments of fees). To reuse the functionality of the ERPNext app, the Education app only needs to link the Student DocType to the Customer DocType.

In the case of the Teacher, the HR is concerned with the Employee to produce the payroll. So, the Teacher only needs to have a link field from the Teacher DocType to the Employee DocType.

ERPNext itself is designed as a monolith. But if it is redesigned within the Microservices mindset, it will catch a lot of attention because the Microservices paradigm is very much in today, and Frappe is a very powerful Microservices tool.

1 Like
4

Thanks for your detailed explanation, then what is your idea how to redesign frappe to support micro service?

I am so interested in this new concept, even though I have thorough understanding of the whole frappe and erpnext framework so far, but I am not clear how frappe can be adapted to support micro service you envisioned so far, if I can understand more in detail, I would like to contribute as needed.

5

Frappe is Microservices ready.

If you need to have any specific functionality, you just need to create an app and put in the DocTypes and code for that specific functionality and you are ready to go.

This is what Microservices is all about. It cuts the development time, and allows the developer to deliver the functionalities to the users in a very short time. Within days. – With Frappe, it is within minutes. Hahaha. This is Agile to the max.

However, ERPNext is a jumble. A huge mix of several domains because of the outdated monolith mindset. Agriculture, Health, Education, Manufacturing, etc. (Apologies).

What if, ERPNext is redesigned as with the Microservices mindset?

Domains need not be jampacked into ERPNext.

It can be separated into different focused apps.
It will be very flexible, and developers will find it easy to contribute specific focused areas of functionalities when needed.

4 Likes
6

or have the School/Health hosted in a different Frappe server without ERPNext and communicate with ERPNext through REST API?

7

With this PR feat(hooks): auth hooks by revant · Pull Request #12895 · frappe/frappe · GitHub Custom authorization is now possible.

Example custom app / architecture GitHub - castlecraft/castlecraft_auth

Diagram:

Usage Diagram

Access Frappe Framework resources with Google’s access_token or id_token Use with Google access_token or id_token · castlecraft/castlecraft_auth Wiki · GitHub

This is just RAD / Agile way to start apps. Nothing about database decoupling. Event Streaming feature can be used to duplicate relevant data. I haven’t tried Event Streaming feature.

I’m planning to improve OAuth 2.0.
Now that we’re on v13 and I can commit to py3 completely, I’ll also implement OpenID Connect. After that the central authorization server can be any frappe framework site instead of Google or any third party FOSS/Non FOSS.

9 Likes
9

These new powers of Frappe / ERPNext are a clear reason why Frappe’s Microservices super powers are woefully smothered by ERPNext’s monolith approach.

10

Sometimes I feel using the framework should be even modular.

  • I start with “hello world” with frappe. No db, no cache, no queue, just api runtime.
  • I add ui, enables the frontend
  • I add db only, Get the power of DocType when db gets added.
  • I add user sessions and cache, enables frappe.cache()
  • I add queue, enable frappe.enqueue
  • I add realtime, enable socketio.
  • If I use anything that is not enabled I get error, asks me to enable feature first.

This will make it easy for deployment of simple stateless or minimal db interaction based app.

Right now 1 app means 1 bench needs to be deployed.

I don’t mind that if 1 bench is only minimal parts used by my app.

7 Likes
11

And for microservice, it is different for different people.

Example 40 line nodejs "micro" service listens to MQTT to fire Twilio Messages 
const mqtt = require('mqtt');
const twilio = require('twilio');

// Twilio Client
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_ACCOUNT_SID;
const twilioClient = new twilio(accountSid, authToken);

// MQTT Client
const quitMessage = 'Ctrl + C to quit';
const mqttUri = process.env.MQTT_URI || 'mqtt://user:changeit@localhost:1883';
console.info("Connecting to : " + mqttUri);
const mqttClient = mqtt.connect(mqttUri);
mqttClient.on('connect', () => {
  console.info(`MQTT client connected. ${quitMessage}`);
});

// Listen to events
mqttClient.subscribe('+', { qos: 0 });
mqttClient.on('message', (topic, message) => {
  console.info(`Topic: ${topic}`);
  console.info(message.toString());

  try {
    const payload = JSON.parse(message);
    twilioClient.messages.create({
      body: payload?.body,
      to: payload?.to,
      from: payload?.from,
    })
    .then((msg) => console.log(msg.sid))
    .catch((err) => console.error(err));
  } catch (error) {
    console.error(error);
  }

  console.info(quitMessage);
  console.info();
});

Microservice > Service Mesh > DDD > CQRS > Event Sourcing > Stateless > Serverless > Chaos Engineering > “next buzzword here”!

All this for what?

Not really worth for anyone looking for $3 VPS installations and freelancers to add custom doctypes, custom scripts and print formats!

No one needs all this unless things are deadlocked due to success!

And if you are one of the startups founded by hackers who reached success, you’ll find out about all the needed technology magic anyway!

5 Likes
12

For me, Microservices is primarily about DevOps. Being able to put out functionalities required by Users within days of specification. Agile.

Springboot / Java has JHipster that produces Server-side Java code and Front-end code based on Angular, React, or Vuejs based on your Data definitions.

For me, Frappe - in terms of being Agile, beats JHipster PLUS it has so many built-in goodies.

The only thing that stops Frappe from being completely Microservices DevOps is its reliance on SQL database which requires server downtime when patching up the database structures.

If you look at Frappe, it is actually very ideal for NoSQL databases, particularly because it is natively JSON in the first place. So, it is very natural for NoSQL.

With the right mindset. Frappe can really be very powerful in TODAY’S software milieu.

It is like a child who is so brilliant and talented in Piano, but is forced by the parent to be a burly football player.

2 Likes
13

for Frappe Apps, we’re there. Always room for improvement. (Not talking about ERPNext)

I tried. I figured out it needs NodeJS v14, v16 didn’t work. Installation is as “easy” as frappe development setup.

For building models it was not as easy as Frappe Framework DocType. My opinion.

With frappe framework even a business user (non developer) can build models and commit code to a branch if given appropriate tools. With minimal knowledge of python and sql, users can even put validations on models and build reports.

I don’t think SQL or NoSQL matters.

Code and other abstractions can manage to keep things running while database is patched.

And in case of multiple dbs for multiple services, we’ve better control to patch databases with no downtime.

Sometimes patching is not even necessary. I observed last two v13 releases (<v13.1.2) of frappe framework (NOT ERPNext) didn’t require patching.

I say we’ve some power of NoSQL in SQL!

We’ve Redis for fast, schema-less key-value store. If pure python + mongodb or any other db is sufficient, it can even work with current frappe framework app. (with no doctype magic)

In general, for developers and hackers everything is possible and available.

I think, we just need to build a cookbook of all such recipes for others to try out.

5 Likes
14

With SQL database, Frappe has to do mini acrobatics to place JSON structured DocTypes to Relational structured (Row-Column) models required by SQL. With NoSQL like MongoDB, JSON DocTypes are an almost 1-1 match! In fact, MongoDB calls its entries Documents. (DocTypes <==> Documents - got it!)

The reason why Frappe on NoSQL is powerful is, with Frappe on MariaDB (SQL), you have to patch the database because changes in DocType require changes in MariaDB tables. With MongoDB, you do not need this.

SAP has moved to NoSQL with SAP Hana (I am not sure I am correct).

Frappe on NoSQL would be natural, and it will make Frappe an even more exciting product to look into.

1 Like
15

Correct! With Frappe, DevOps is within minutes. Not even days.

16

Not easy task. In frappe codebase “frappe.db.sql()” is used 694 times in 263 files!

My attempt (closed PR) to add mongodb backend in frappejs Mongodb Backend by revant · Pull Request #48 · frappe/frappejs · GitHub

17

This is what I mean by Frappe being forced to do acrobatics to suit SQL, like it has to do getchildren acrobatics.

With MongoDB NoSQL, the entire frm.doc in javascript (including children) , can be saved hook line and sinker – natural in MongoDB without need for looking at table structures. frm.doc can be saved to MongoDB and reloaded from MongoDB in its native JSON form.

When I look at the frm.doc at the Chrome Debugger, I smile at how brilliantly natively JSON Frappe is. In other frameworks, they need tools like GraphQL to go beyond the limits of REST API to get to JSON. And here is Frappe doing JSON like its no big deal. (And Frappe has to do acrobatics to suit the old-age SQL while its communication to NoSQL would have been so natural).

18

SQL itself isn’t the limiting factor here. There are good reasons why NoSQL hasn’t entirely replaced it in the ERP world. SQL’s ACID features makes it rather fantastic for storing ERP data. There’s no reason it couldn’t work in a Microservices architecture equally well.

The challenges are with the current design:

  • The current “ORM” schema and design could be much improved via a redesign.
  • The SQL design could be made modular, just like their services. Today many things are lumped together. Purchase Orders rolling up into Buying Controller, rolling up into Stock Controller. When you need them to behave differently, it’s awkward.
  • Support for the concept of “display methods”, to avoid redundantly storing values in SQL.
  • Support for overriding name without use of Prompt during POST.
  • The acrobatics you describe could be much faster/easier/better.

(finally, generally speaking, components need to be separated and broken apart (installation, upgrade, SQL sync, Frontend, Backend, APIs)

Anyhow, with time & effort, the SQL backend could be hardened and better-support modular and micro services. Those acrobatics wouldn’t bother people as much. I’ve seen other ERPs make the SQL maintenance rather seamless.

Though I feel SQL is fantastic for ERP tables, there are absolutely use cases for other storage solutions too. Would be fun if DocType A could have a SQL backend. But DocType B and C have MongoDB and Redis. All while largely obfuscating the technical differences from the average developer.

3 Likes
19

This is no longer true. NoSQL databases like MongoDB have become transactional and more.

In version 4.2 , MongoDB introduces distributed transactions, which adds support for multi-document transactions on sharded clusters and incorporates the existing support for multi-document transactions on replica sets.

In the first place, unlike SQL databases which forces you to structure your data in rows and columns, so that you have to deal with multiple tables when you have parent - child relationships, you store everything in a single document in NoSQL database like MongoDB - Just like Frappe.doc is using JSON in memory at the front end. Which is why Frappe’s front end JSON structured DocType could have a 1-1 mapping to MongoDB Document.

SQL databases are becoming passe. Frappe will benefit from NoSQL.

20

Could be possible when Virtual DocTypes mature. With Frappe team’s attention, it is a matter of time.

1 Like
21

This is Microservices mindset precisely. A Microservices app focuses on providing specific functionality to a user’s business domain, and then reach out to other available Microservices apps through linkages.

It does not matter if you have to duplicate data from one app to another, as long as you can get started with beginning software development on user specification ASAP. (This duplication also ensures that a problem in one App will not affect the other apps.)

With a monolith approach – especially with a DRY mindset, you cannot begin software development immediately because you have to dig up what are the functionalities provided by the monolith - and also provide the monolith with all sorts of possiblities. This is how you end up with having party_types on ledgers.