We’re two people (Functional Implementer and a Developer). This is a learning experience for both of us. We’re currently doing an implementation for a client.
The functional implementer was working on his instance at first. In the beginning, he created some custom DocType. When there was a development requirement, I took a backup of his instance and replicated it locally before I wrote a custom app. I set some of those new Doctypes under this module (the ones that made logical sense), so I have the JSON files in my custom app. Meanwhile on the other instance, a custom app is not yet installed.
As I was developing the app, the functional implementer has continued work on the system and made some changes. to those Doctypes. My worry at this time is that if I install my custom app on his instance, it might overwrite some of his changes, whereas if I take a new backup of his instance and clone it again in my local instance, I am not sure of any potential side effects.
We discussed whether it makes sense to just work on a common instance, but that doesn’t sound right to me. I mean, at some point we may need to add a junior developer also. So I think this is a workflow and coordination issue that we’re facing, due to not having experience working as a team.
When we look at the documentation or forum posts, any information we find is from the perspective of a single user / developer, rather than for a team. Am I overthinking this or are there recommended workflows that I’m not aware of?
2 Likes
Here’s what has worked for me:
-
Development Environment = Each developer’s laptop, desktop, or whatever.
- Use git to push and pull changes. Create branches as needed, merge as needed.
- No one can see this environment except the individual owner.
- Can download the latest SQL database from other environments (Test, Production) as needed.
-
Rule 1: Don’t use the “Customize” button for custom Apps. If a change needs to be made, change it via the App’s code, hooks, etc.
-
Rule 2: If you use “Customize” button for Frappe/ERPNext Apps, you have to document everything you did. So those changes can be repeated in Test, Staging, and Production.
-
Test Environment
- Lives in the cloud.
- Only a few people have access (people who are developing/testing)
- All code changes are pulled down from GitHub, based on what developers have previously pushed.
- Data can be refreshed from Production, if necessary. But usually this only happens with the consent of everyone who uses TEST. So people don’t lose their test data.
-
Rule 1: Testers should avoid use “Customize” button for custom Apps. If a change needs to be made, try to change via the App’s code.
-
Rule 2: If anyone uses “Customize” button for Frappe/ERPNext Apps, they must document everything they did. So those changes can be repeated in Staging and Production.
-
Staging
- Lives in the cloud.
- 95% of the time, is an almost-clone of Production. Both in terms of code and data.
- Most employees have access.
- Most-often used for reproducing bugs previously found in Production, running experiments, testing things you cannot do in Production, because you’d harm the data.
- Sometimes used to “stage” changes that are heading to Production very soon, and you want to run a final set of tests.
- Refreshed from Production data frequently, so they behave similarly.
-
Production
There’s a ton more to it. Especially when it comes to pulling Production data into lower environments. That can be very dangerous. You don’t want automation to trigger that impacts real-world customers, suppliers, employees, etc.
But hopefully this helps a bit. Unless your functional person is actually writing code, they would spend all their time in the Test and Staging environments.
I have a mixed relationship with framework features like Customize, Server & Client Scripts, hooks, and anything that “patches” code. Those are double-edged swords. They can be useful. But can also make things like troubleshooting and code comparison a real pain.
10 Likes
I realize the mistake we’ve made. Early on in our implementation, we overlooked the potential issues with using Customize Form and have not documented those changes, so we need to check each one but we’re gonna keep track of it from now on.
In our two-member team, only one of us (me) is a developer and writing the custom app, while the other person (Functional implementation only) is configuring a separate instance on which my custom app is not yet installed (but will install soon). Is there a way to retrieve only the settings out of the other non-development instance without the mock data in it so that we can set up a “clean” instance on which the custom app is installed, or is that also going to be a mostly-manual process? As I understand, there is a partial backup process. So is that what I need to be doing table-by-table, so I can fetch only settings, and then restore DocTypes manually?
Moving forward, when some option is set in the system (For example: Enabling a checkbox, or setting some value, or changing some of the default settings, etc.), is it possible to log just those so they can be replicated on a clean-system? Or is this also something that needs to be documented when a system is being configured?
The customized settings are stored in various SQL tables. I’ll list the ones I know about below:
-
tabUser Permission - Contains data when you assign special permissions for a specific User + specific Document number. I’ve never used this feature.
-
tabCustom DocPerm - Rows are added here whenever you use the “Role Permissions Manager” web page to change the default Role permissions, for a DocType.
-
tabCustom Role - This contains some of the data behind the “Role Permission for Page and Report” web page.
-
tabCustom Field - This is an important one. When you use the “Customize” button to add a new DocField? It is recorded in this SQL table.
-
tabProperty Setter - Another important one. Rows are added here when you use the “Customize” button to change a DocField’s metadata. For example, if you add a new Option to a Select field? That new DocField metadata is stored here.
-
tabReport - This is an interesting table. It contains both Standard and Custom reports. So, to find Reports you’ve created that are not part of your App? You have to filter like this:
SELECT
name, report_name, is_standard, module, report_type, query, json
FROM
my_database_name.`tabReport`
WHERE
is_standard = 'No'
ORDER BY
module
-
tabClient Script - This is where custom Client Scripts (JavaScript) are stored, from the web page with the same name. I’ve rarely used these, and modify the JS on disk instead. But many, many developers here on the forums use these.
-
tabServer Script - This is where custom Server Scripts (Python) are stored, from the web page with the same name. I’ve rarely used these, and modify the Python on disk instead.
-
__UserSettings - This is a bit different. This table holds User’s most recent filter and sort settings, for each DocType. Not something you probably need to worry about migrating. But…I thought I’d mention it anyway. Since we’re talking about specialty SQL tables.
All of these tables are part of the Frappe/ERPNext backup process. The complete MariaDB database is in the backup files (mysqldump)
As for migrating between environments. It’s all SQL, so there’s nothing preventing us from copy and inserting rows. However…the tricky part is when there are data conflicts.
For example, assume you have customized a DocField’s metadata. That customization is written to 'tabProperty Setter'. Assume your functional partner also made their own customization to the same DocField? Perhaps you added a new “Widget Status” named “Foo”, and the other person added a new “Widget Status” named “Bar”? Those separate modifications will have the same SQL surrogate key. You cannot copy and insert theirs into your SQL database. Because then you’ll have 2 rows for the same customization. Only one of them can be applied: Foo or Bar.
You’ll have to identify those types of conflicts, and handle manually.
The above illustrates my personal reluctance to use the Customize button, hooks, client and server scripts, etc. Yes, they’re incredibly flexible and prevent you from touching the official code. But it’s also a bit messy, settings are specific to each database/environment, and it introduces possible conflicts. I prefer just running a git diff and knowing whatever I see is the truth, and the whole truth.
But everyone is different. It’s always nice to have options, and I’m glad Frappe Framework has them. Choose whatever works best for you.
The one place I’m most-comfortable using Custom features is with reports. The ability to write/edit SQL in Production by using a Custom query report is pretty handy. This has enabled me to quickly answer (some) of my customer’s report requests, without making them wait until the next code deployment window. However. It still requires me to be disciplined. When I modify a Custom Query Report in Production, I must to remember to update DEV, TEST, and STAGING with whatever changes I just made. And vice versa. If I don’t, it will bite me someday.
6 Likes
How about workflow that setup and test in dev environment. What’s the best practice to move from one to another?
Could you please give me the idea?