Microservices and Bezos 2002 Mandate

1

In 2002, while many big software companies were enamored with Monolith Enterprise approach and repeated the Do Not Repeat Yourself (DRY) mantra, Jeff Bezos issued a mandate that broke these rules. Many look back to this as the breakthrough point for AWS which permitted the company to speed ahead of everyone else:

  1. All teams will henceforth expose their data and functionality through service interfaces.
  2. Teams must communicate with each other through these interfaces.
  3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
  4. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter.
  5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
  6. Anyone who doesn’t do this will be fired.
  7. Thank you; have a nice day!

If ERPNext can follow this guidelines (which it can because it stands on Frappe), it could be the breakthrough moment for our beloved Elephant.

1 Like
2

An interesting read: Goodbye Microservices: From 100s of problem children to 1 superstar | Twilio Segment Blog

Microservices works for 1% of the use cases where each action needs to be infinitely scalable. There are several ways of scaling Frappe / ERPNext as it is without using microservices.

3 Likes
3

There may be advantages in breaking up ERPNext into Accounting, Sales, HR, into separate apps along the lines of the Bezos mandate.

4

Yeah, is on the cards (no ETA) but that is still not “microservices”. Just better repository management.

3 Likes
5

I think an advantage of “Microservices” is no suprises, no side effects. Each module will contain well-defined “connectors” for other modules to pick up. Therefore, each team may be able to focus on its functionality and not worry if members from other teams messed up with their code. Also, errors in one team would not impact other teams.

6

Do frappe.get_doc() and REST/RPC endpoints count as a service interfaces? If so, are these conditions not already met?

7

No. Service Interfaces provide a well-defined “connection” between application functionalities. Kubernetes makes this quite clear. Also, knowing how to manage cloud infrastructure of AWS / Google / or Alibaba through API code makes one realize the importance of having well-defined (and enclosed) functionalities which communicate with others through service interfaces. Data may be duplicated on both ends, but it prevents surprises were one functionality may create invisible side effects on the other functionality. It forces the developer to become a software archeologist - because you have to dig through the code to uncover the puzzling side effect. This “Bezos” approach prevents these and allows quick and safe code development among multiple teams.

8

So, each module would have to implement all its own query protocols, CRUD methods, REST endpoints, etc? If so, no thanks. :slight_smile:

2 Likes
9

That’s ok. Discussions eventually surface the best ideas among many. Thanks.

10

Mission accomplished. Thanks @rmehta. You are a wonderful leader.

11

What about network failure, retries, circuit breaking, tracing and debugging requests, logging and monitoring?

Also, installation, maintenance and upgrades need to be easy.

ERPNext is not expensive to self-host, VPS prices go down and hosting ERPNext becomes even cheaper.

Even if we manage to decompose endpoints, add a gateway, add service mesh, tracing, logging and monitoring with “easy” scripts. All this setup will need more resource than the cheapest single vps installation people commonly choose.

5 Likes
12

One small step at a time.

Because you are familiar with Kubernetes, you tend to see things that Frappe may lack - but is already available technology. This is why no-batteries included frameworks like React are getting more traction compared to batteries included Frappe.

If code that is already available in Frappe and ERPNext are separated into self-contained modules, implement APIs as Bezos envisions, individual parts may be very valuable.

DocType is a powerful concept. You are right in saying Frappe provides NoSQL power on SQL databases.

13

Interesting old stuff. No disrespect. Just frank assessment.

During the time of writing, containerization technology - specifically Kubernetes was not that mature. Today, with mature containerization, much of these “difficulties” and “pitfalls” have been resolved. Cookbook recipes (revant’s term) are available.

Many companies are breaking up their monolith systems into containerized microservices. The work done by Revant on Kubernetes for Frappe / ERPNext is brilliant and difficult. Revant is a valuable resource for the community. I’m sure he has a lot of ideas.

The write-up is a monstrous misunderstanding of microservices. It is like a horror story of DevOps gone wrong. The fact that they had a miss mash of library versions shows tight coupling which is not what microservices is all about. The monolith the writer is taking about IS THE MICROSERVICE. It serves a particular area of business functionality.

1 Like
14

99% ERPNext users don”t need Kubernetes. I am not sure what problem you are trying to solve other than falling for the latest fad.

3 Likes
15

The mention of Kubernetes is with regard to the write-up and the issues they were trying to resolve. I agree that 99% of ERPNext users don’t need Kubernetes. But fortunately, Frappe / ERPNext on Kubernetes is available for those whose loads would require it. This makes Frappe / ERPNext future-proof in a way. For those who are knowledgeable enough, deploying via containers is simple and powerful.

A benefit Kubernetes brings to Frappe / ERPNext is High Availability - meaning, you don’t care if your server breaks down. You have the option of three or more (as long as it is odd numbered) “master” replicating servers - if one goes down, the other two is there to take up the slack. You can place there servers in different locations so that fire and other natural disasters would not affect the server operations.

If you are using cloud, you can even place your servers in different zones and different providers. This way you are assured of total stability. With the cloud, adding servers is a click of a button away.

Another benefit Kubernetes brings to Frappe / ERPNext is scalability. Suppose the load on the servers is growing - because probably, more users are using the system. No problem. You just add more “slave” servers. On the cloud, Kubernetes can automatically do this for you - spin up additional compute engines when load gets high and spin down compute engines to save money when load gets down.

Other containerized applications may run alongside Frappe / ERPNext, and you can monitor how each application is doing. Which server is experiencing trouble, which server is down, which server is doing well. All in the comfort of your terminal.

This is what Kubernetes can do for Frappe / ERPNext. And this is available to the community. Thanks to Revant.

If components of Frappe / ERPNext can be separated so that the separate functionalities is contained in individual containers, Frappe / ERPNext will even become more powerful because when the user load requires an increase in power, only those containers will be auto-scaled.

This is just the surface. Storage can be kept in multiple computers which contain duplicate copies of data. If one computer goes down, you just replace it, and the team of computers will welcome the new addition and place duplicated data on it. This is how you achieve massive scale. Fortunately this technology is available in ERPNext / Frappe, and would become even more powerful if the community understands the full potential of Revant’s work.

To illustrate, we have one company whose CTO has never heard of Kubernetes and containerization and microservices. So, he makes the company buy a very expensive and powerful server whose components are expensive or perhaps internally replicated. Then, he stays up late at night worrying when the expensive server may go down. His disaster prep is doing backups. For DevOps, he has to take down the server, install the new app, bring up the server, and hope the new app has no bugs.

Then, we have another company whose CTO knows about kubernetes, containerization, and microservices. He makes the company buy six ordinary, cheap, easily replaceable servers (three master and three slave). He also subscribes to virtual compute engines on the cloud. He connects these using Kubernetes. He installs apps using Helm. Then, He designs the DevOps in a continuous delivery mode where, when the developer team brings out a new release, the deployment serves only a portion of the users, and gradually increases the users until all the users use the new version. In case of any problem, Kubernetes quietly rolls back and serves the old version. The users don’t notice anything, nor experience any downtime. This is the technology that is now available with Kubernetes.

This shows the value of Open Source. While other ERP systems, may still be trying to know how to containerize their apps into Kubernetes, here, we have a member who did the hard stuff for all.

Note: of course you can start with one server and experience the ease of installing containerized application. Then you can elevate to High availability or add more slave servers when required.

16

We too had experience problems in scalability. Kubernetes saved us. Using the usual method went too costly for us.

I posted this before What is the proper configuration for a server with 3000 users? - #25 by johnskywalker

2 Likes