I find it very hard to create a holistic development lifecycle around changes in ERPNext.
We only want to create some new DocTypes and use existing ones, but we haven’t found any way to extract those changes after development, manage them with git and introduce them into a ci pipeline.
We are now following the custom apps approach because from what we’ve read we believe this might the feature we have to use. However, we still want to use default ERPNext DocTypes as well.
What if I want to create new Item Groups, or new Item Templates and Variants? They will all live in the Stock module. But that’s not part of my custom app, it is part of the ERPNext app. How could I get those chances out of there? And sadly, this brings us back to square one.
To increase complexity even more, our desired configuration and workflow should be docker-based. In order to have reproducible builds and a defined software pipeline, our approach needs to be based on docker.
You are not alone. These questions need some diggin’. There are also ambiguities in the UI.
If you mark “[ ] Custom”, it’s counterproductive, or not, depending on your use case and understanding. But basically this type of “[x] Custom” stuff lives in the database, which you normally ingest after an update.
“[ ] Custom” stuff lives in the file system, and you must be Administrator and in dev_mode to use it properly. But then it won’t get transported by your database backup/restore Operations. You have to transport these by your version control system or so (e.g. git).
So this technique creates files when you change DocTypes.
You can also have data exported into json files (e.g. docs living in the doctypes).
And you can choose the module where these doctypes should live, which means the place in the file system folder hierarchy of your installation. You can also create an entirely new module, which becomes a new directory on the module level of your installation.
And lastly you can create another app alltogether to be installed in the bench where your frappe and erpnext live.
With docker there is some additional stuff to master: volumes, immutability, build cycle, etc. indeed.
But this might be the lesser learning in case you already bring knowledge of docker with you.
Thank you for responding and sharing your knowledge. I feel like the UI is sometimes misleading indeed. And docker knowledge wouldn’t be a problem. I am a very seasoned DevOps and Backend Engineer. But this is my first project using ERPNext.
Today, I created some Item Groups and made a tiny addition to the Item DocType, but unfortunately, the only file that got created was a database.log file, no json files
I am also not sure anymore, if I really want to adapt default ERPNext DocTypes, or if I should create my own, specifically for my use case. But, it feels like, building all things from scratch kinda negates the whole approach of using ERPNext. Then I could have been much more productive by just building my own microservice, data model and CRUD API.
Maybe you can help me figuring this out as well, this is all very confusing to me.
Our use-case is so simple:
we have a set of suppliers
we have a special kind of packaging material
we have a very specific list of materials
and we want to order them using Purchase Orders
and we want to book the delivery with a Purchase Receipt
How would you tackle this very simple use-case?
Should Materials be an Item Template with multiple Item Variants based on their Item Attributes?
Someone suggested to us, that our packaging material should also be an Item Group with Variants. However, this would mean “Items are packed in Items” and then the Item DocType would need a mandatory “Link” to another Item?
Yes, it sounds quite standard.
Why do you want Item Variants instead of simply using Items?
I don’t see a problem at all in using packaging material as an Item in itself (w/ or w/o Variants). That’s just a mind thing:
Of course you pack your things in some kind of packaging things, and you need to purchase all of them from suppliers, so warehousing in some form is necessary for everything, unless you can create packaging ex nihilo or get it Schroedinger’d into your warehouse.
So you probably want automatic reorders if stock level goes down for your packaging material. You just don’t sell it (unless you do), and you sell it anyway in some form or another because you have it as shipping cost part or as indirect expenses if nothing else.
You can create Work Orders, BOMs and Operations which could take the quantity of your packaging into account to get a good grip on everything’s turnover (if needed, maybe it’s not necessary in your use case, depends on other factors).
So why not just take a standard installation and try things out for some test data along such lines?
All business data (Item Groups, Items, Variants, Customers, Suppliers) exists exclusively in the SQL database. Pretty standard practice for ERPs, in my experience.
To move that business data between environments (test, staging, production), you can either:
Migrate the entire database (SQL backup + restore)
Write your own scripts to export and import your data.
The DocType metadata is stored in 1 of 2 places:
If “Custom” is flagged? Then the DocType/DocField metadata is only stored in SQL records.
Otherwise if not Custom (Standard is true), then metadata is stored both in SQL records and JSON files.
If you have your own App, and you want that App to create Custom fields in the Frappe/ERPNext DocTypes? For example, add a new column to Item Group, or Supplier?
In that case, you can use the fixtures feature to migrate those metadata changes.
This is personal judgement call we all have to make.
The more complex and extensive your modifications become? The more I’d recommended creating your App and DocTypes, instead of modifying the existing ones.
Other Thoughts
The Forum Community has historically recommended that everyone:
Do not edit Frappe/ERPNext code directly,
When possible, use the “Customize” feature to achieve your goals.
If the above isn’t sufficient, you can always just fork everything, and maintain all your changes in git. That’s what I’ve been doing for a very long time.
First of all, @brian_pond, thank you very much for sharing your experience. It feels like your response helped me understand some things better.
However, I would really like to take another follow up round and it would be really teriffic if you would find the time again to get back to me
For me personally, only option 2 feels good. Maybe I don’t understand bench and the frappe framework not well enough, but when I think about “restoring development changes to a production database”, I get chills
However, my experiences with the Data Import/Export feature are not quite good When I tried this earlier this week, the Data Import would show me an error message, that it doesn’t recognise the files I just created with the Data Export feature Very weird but ok.
I don’t know if this is just frappe are typical for the ERP-world, but I find this approach very cumbersome and als misleading.
This is some advice I find very interesting. To me, it feels like this comes closest to “normal” software development. Please help me out and tell me if I understood you correctly, unfortunately, the documentation about this feature is not very elaborate.
Your are saying:
I could develop my own custom app
this app could contain some custom DocTypes
but, more importantly, this app could also “contain” changes I made to ERPNext DocTypes
and by using bench update, I could rollout changes I developed locally to testing, staging and eventually to production environment
this would not put any production data in danger
If this is true, to me, this sounds kinda similar to data model and schema changes in “normal” software development. Hence, I don’t understand what would happen if changes I’d want to rollout made “migrations” necessary. For example, if I would change the data type of a field from number to text. How would bench update/migrate existing records in production?
I understand the general rational I guess, but what still confuses me is the scope and possible side effects of my changes.
For example:
We’ve established, that the packaging material I’ve mentioned could be just a normal item. @Peer argument about this was pretty on point. I can be more specific and say that the packaging I was referring to are just simple Big Bags.
When we order or handle them internally, I need to store the information which “material” is inside a every Big-Bag. I implemented “materials” also as Items, more specifically, as Item Variants, so far so good I guess.
Hence, I thought, I would need to add a custom field to the DocType Item which is a Link to the corresponding “material” Item Variant. Why? Because we want to ensure, that Big-Bags only contain the “materials” we have defined.
However, if I would make this field mandatory - because I want to ensure that every Big-Bag record has a “material” linked, this would become mandatory for all Items in ERPNext, correct?
Those are the potential side effects I mean. Even though our use-case seems so trivial, I feel like I cannot estimate how big the impact and side effects are on the rest of the system.
But let’s add the rest …
But before I ask you about your suggestion, let’s just add the other 3 requirements we have:
eventually, we want to track the amount and location of Big-Bags
and we want to track the total mass of materials, stored in Big Bags
but most importantly, when we send thos materials to subcontractors later in the process, we need unique IDs for our Big-Bags which enable us to trace back Big-Bags and materials from our suppliers through our subcontractors and back
The big question
Assuming you would like to apply software engineering best practices with different environments, testing and CI/CD pipelines… Which approach would you pick to implement something like this?
Thank you for pointing this out (the “and”) and getting me to think/express this on my trying to get the first principles clear in my head:
This hints at the concept of “ownership” (of some piece of data), aka which data of several clones of a piece of info is “authoritative”. This might depend not onely on the storage location (RAM, HD, dev’s paper trail), but also on the moment, because a manipulation of the developer can imply the intent to transfer the authoritativeness to another clone (“restore backup”).
So this also implies syncing aspects: e.g. during development, roll-out, update and possibly data recovery.
What I find so interesting in this dialogue is that your “accessory” “wrapping” or “envelope” which came along incognito at first turns out to be a mobile warehouse with a location and possibly it’s own individuality (ID) to be tracked.
Pending more information about it reuse/reusability in your organization’s application: possibly also inspection of further usability state, possibly accounting need of it’s current value against external clients and/or suppliers of the transported materials (just like pallets can be new, old, scrappy, thermorecycling or fertilizer material in the end, or transformable as the new trend in interior design or even housebuilding depending on the local trends or requirements, bigbags could be transformed into trendy clothing, kid’s swimming pool, or need to be disposed of as a liability due to what they contained or how such waste might be taxed in the comprehensively administrated sustainable future).
So this mobile warehouse (or container) is itself being subjected to be positioned into or onto some other warehouse: weather-proof building or transport vehicle, open air terrain, intermediate scrap-yard, selling location, depending on your use case.
Your organization could well be be a logistics company (organisationally speaking for what concerns configuring/adapting ERPNext), or a trading company reselling demolition material, a material transformation (recycling) enterprise (grinding stuff to particles or powders to be reused in construction for instance) – or any kind of combination of these, exploiting verticality integration benefits.
I’m not sure what is the best way forward. You could tack a “location” field onto the warehouse leafs and update it as you move them around, or maybe such is not necessary because they are more like grouped into bigger bigbag collections which are treatable like batches, so this influences the effort for handling of the bags. You might need some specialised controller logic, or maybe the low-code features are enough to tackle your intended use.
But after all, maybe the bags end up to be just the purely accessory item as presented in the beginning.
The beauty of all this is that the Framework and also ERPNext allows for a lot of flexibility, which implies a creation effort to use it as desired, and of course one needs to know one’s IT building blocks.
And docker knowledge wouldn’t be a problem. I am a very seasoned DevOps and Backend Engineer. But this is my first project using ERPNext.
