We use frappe extensively for microservices. It’s a great platform for that, batteries included. There have been some particularly positive developments along those lines recently, like virtual doctypes, server scripts, event streaming, etc. @revant_one’s work on oauth over the years has been incredible too.
On the SQL/NoSQL debates, I have a hard time getting invested. To my mind, the whole point of a framework like Frappe is to abstract away from that kind of stuff. In information theory, the debate over more or less explicit structure is as old as the hills, predating anything we would think of today as “programming”. I’m sure there are performance benefits to be had in each approach in different circumstances, and the question of which to use comes down to benchmarking and maintainability.
One thing I’d really like to see to further microservice flexibility is “virtual fields”. These would be computed values that behave indistinguishably from stored fields at the API level. Simple use cases would be table joins, but more complicated applications could certainly exist as well. I’ve been poking around at the code a little bit to see what would be necessary to make this work, but if anyone has had thoughts along these lines I’d love to discuss. This kind of functionality would do a lot, I think, to better encapsulate connected microservices.
I think being SQL or NoSQL, each one have they own problems!
Like @Joseph_Marie_Alba1 have mention before, it’s truth that the ORM need to do some acrobatics around the child tables
But in the other hand, it save a lot of time of computing, because, you can query the data much more easily. For example, produce the same reports ERPNext have, without a solid ORM layer that support NoSQL and SQL will be a nightmare, that’s also why the support to Postgres there’s not too much traction, ERPNext and Frappé relies a lot on SQL features, like functions and other things that are available in a way under MariaDb and in another way on Postgre.
This image is quite old, but explain the cost a NoSQL introduce just for quering the data
We always can precompute the reports, and build data collections during the insert on new data, but again, why do that, when SQL is okay, and much more stable for quering?
Microservices are great for frappé, and I do agree the framework should support and have improvements in this area, but, in the other hand, can it allow an simplification of the whole project, or just will create new levels of abstractions to deal with the same problem?
On top of microservices, there’s a lot of data replication and redundancy, also the fact that data is distribute in many places, cilos or warehouses, make it much more complicate while you need to make a serious analysis around data to extract knowledge.
I have heard a lot that micro-services are good, are future proof, etc, etc, etc. But I didn’t have saw someone explaining how micro-services can simplify the management of a monolithc codebase, without introduce new challenges, like data quering, sincronization, distribution, and in case that an single service is offline, how to ensure that all data will be send to this service when it came up, in a transparent manner for the developer.
Just as a complementary information, Gregor Hohpe & Others explain all the problems Microservices introduce about data, in the book “Enterprise Integration Patterns” (EIP),
Also, Python as language is far, far a way, to have a great support to EIP, like Java does, with Apache Camel.
The unique project that can simplify this, is ZatoESB https://zato.io/
But again, it’s another crazy layer, to deal and resolve things
To dont mention, that if you are willing to make a serious development for Enterprise solutions, atleast an API Orchestrator will be needed, to guarantee redundancy, availability and load distribution around microservices.
@peterg, I do have this code, that can be easily adapted to support virtualfields
The get_meta method need to be patched to support the discovery of virtualfields, and a VirtualFieldDescriptor need to be defined, passing the value getter function and the fieldtype to send to the UI
Frappe’s JSON structured DocType (especially frm.doc of front end javascript) can have 1 to 1 mapping with MongoDb Documents.
I mean, Frappe has this natively (unlike other frameworks which need add-ons like GraphQl). Yet instead of the natural - and currently on the rise - NoSQL database, Frappe only uses the incompatible SQL databases.
I agree. I also write code in other setups like React and GraphQL, Flask, Django, and Springboot (Java / SpringTools).
I like the way Frappe permits me to get things done. There is a saying that a system is worth more than the individual value of its parts.
Honestly, for me, it is the reverse for Frappe and ERPNext. Lumping up the parts into one system seems to devalue the individual component.
For instance, like you say, you can very easily spin up a web page in Frappe with the data from the partner .py code file, and you don’t need the user / permission part.
However, at the same time, in another application, the Frappe user permission system can provide powerful authentication / authorization.
Frappe and ERPNext components are powerful. But together, they bloat.
One of the motivations behind building Frappe the way it is, is that you don’t want to use any other system. Some people love working on different dev environments for different pieces, but that is massive context switching. Frappe was designed to be “battries included”. Think of it as some kind of extreme DRY. You can support 800+ DocTypes in ERPNext because of this mindset.
On the topic of having a NoSQL backend, it is tough but not unthinkable. Anyone suitably motivated should be able to pull it off.
I think Frappe should maintain frappe/ERPNext as it is. This is the base for any fork scenario.
As we can see in the benchless instance, docker based, etc.
So… we can expect to see the forks on the low-level (system, framework, db, etc), not only on features and modules
I think it is nice to shake things up a bit, especially at the beginning of a new version (14.0).
Respectful discussion is a value that the Open Source community bring.
I think when you see how Kubernetes brings things together by having clearly defined boundaries for needed functionalities, one may realize that Frappe / ERPNext has built in so many elements of Kubernetes. Yet, the value of these elements become devalued because they are quite tightly coupled with the other elements.
I think what @revant_one observed (especially coming from a Kubernetes background), is that Frappe includes batteries like ID (Authentication and Authorization), Web, etc.
If these elements became distinct in Frappe, someone who wants just the powerful web (bootstrap + jinja context) would not need the ID part. Thus, Frappe becomes even lighter.
This totally makes sense, especially that microservices/Kubernetes/Docker things may be too complex for us the target audience.
However, I also think in the context of ERPNext the monolithism is being pushed to the extreme to the point of becoming a burden to some users. Why should a school make an update when the hotel module has changed? My point here is to move domains from the core and define a clear interface of how domains interact with the core. This hopefully can give domain-focused communities room to flourish independently?
I have just a few notes about the NoSQL / SQL topic:
There were some concerns about MongoDB licensing recently, so even if an alternative NoSQL backend is developed, I would suggest to deeply evaluate the different options (Mongo, Cassandra … etc.)
PostgreSQL has great JSON support using its JSONB data type (that is very similar to how MongoDB stores its document in BSON). It has built-in SQL operators for handling JSON data, it can also index JSON documents for speeding up queries.
PostgreSQL experts recommend to use regular columns for data where the structure is not likely to change often and JSONB based columns for dynamic data.
There is also a SQL/JSON standard extension of SQL which is implemented by PostgreSQL
For large systems, where traditional SQL database clusters are no longer enough, there are at least 2 upgrade paths to cloud native, horizontally scalable databases, similar to Google Spanner: CockroachDB and YugabyteDB.
YugabyteDB even uses PostgreSQL code for its SQL support, it also has a Cassandra compatible mode and both frontends are backed by its own NoSQL engine: yugabytedb.com
I think it would be great to explore the different storage options in more detail and see how different solutions can be supported for different use cases. I think the new virtual doctype is definitely a step in the right direction.
I’m very interested in a cut-down version of Frappe for use with backend services, and would be happy to contribute both time and money as needed.
We use ERPNext exclusively and are a small team, however we want to launch a product and use Frappe/ERPNext as the backend with hundreds and hopefully one day thousands of concurrent users. Hence our front-end or UI requirements can easily be handled with a singular monolithic architecture, our API requirements would be more suited to a microservice architecture. We can’t be the only ones.
Ideally I would love to see a version of Frappe/ERPNext or better yet a command like bench microservice that strips out the backend elements of the product (with apps) and can be deployed to AWS Lambda via the CDK (I suppose in a virtualenv), it would have no cache or queue, just API runtime and DB connectivity. That way those of us addicted to the DocType framework can build customised API frontends and share a database with the monolithic ERPNext.
From my point of view it solves a medium term need, but also long term it provides a stepping stone to splitting up ERPNext into modular components that would be well suited to microservice architecture. I’m not a fully-fledged convert to microservices, in fact for the majority of users (myself included) the existing structure is perfect. However in looking at this approach it does provide an opportunity to re-work ERPNext for each use-case, for example the education module just simply doesn’t need to be installed for most businesses, there are so many doctypes in our database with no rows and so many modules that are never used. Imagine how great ERPNext would be if it focused on the installation of a bare-bones featureset upon installation, with a catalog of additional modules (or bundles) that can be installed that encompass those elements you require. A choose-your-own-adventure open source ERP, a world beater!
Enough of me ranting (day-dreaming). If anyone would like to chat about putting together a microservice backend just reply below.
