Beta Testing Process

At the speed at which we are moving, it would be great if service providers can spend 15-20% of their time contributing features. This would greatly reduce collisions and also speed up development. This also includes test cases :slight_smile:

@rmehta at the conference I suggested you guys need to develop less and guide the community to get more involved.

With your current approach I feel you are creating a barrier instead and your speed is not sustainable for a long term plan.

For the Beta Testing Process, which is part of the community involvement, I would like to see some facts now as I’m not longer interested in the discussion if it doesn’t lead to anything.

As IAG, we’ll keep contributing as usual either during our migrations to your latest stable master branch (I set a budget once/twice migrations per year), plus any our development which is interesting for the community. Next one will be the E-mail Inbox app, as soon as we are able to finish the migration to 7.1. Then we’ll be back to the bar code scanner, and so on.

Hope my feedback helps. We cannot do more than this.

1 Like

@IAGAdmin Thanks for the feedback and leadership!

If you are contributing your features, you should be good. I took your advice and we are clearly very responsive for giving feedback on PRs and merging issues.

I am really happy people think ERPNext is good as is, but I still see so many things could be better. As long as your features are in the core, contributed, it will be our responsibility to migrate them. On the ERPNext cloud we manage close to 5000 ERPnext instances and most of the changes are bug reported by the users.

I think in the medium run, the best strategy would be to keep contributing and staying on latest. A lot of good stuff is yet to come!

1 Like

@rmehta We’ll be able to stay close to the latest if you guys introduce the community beta testing process.

I’m not interested in the new stuff if the old ones have been compromised or got worst, as we need to fix them first before migrating. Secondly, we need to train and help users to get back to normal after migration. Then we can explore the new stuff.

This is one reason why I don’t feel comfortable to host our data in the ERPNext cloud. It’s too risky for us.

2 Likes

How about discussing this on the Developer Call Tomorrow?

@nabinhait is our release manager - he as already committed to starting a release-candidate branch !

Nabin - please lead the discussion tomorrow!

I am not sure whether I am misreading anything but somehow I don’t get the feeling you are on board with this generally. Can’t really think of any reason why though

@nabinhait I’ll try to be present. what time will it be held?

should be 1pm UTC (easily checked here https://time.is/UTC)

URL for the meeting should be https://www.gotomeet.me/erpnext

Yes, it will be on 1 PM UTC

trying to summarize a little the status based on the community hangout that just took place … (which is a bit difficult because I hardly couldn’t understand what @nabinhait said due to audio transmission being shattered to bits … anyway this is what I took from it:

  1. For the time being there will be not automated test-routines because they have not been set up yet
  2. beta testing process will be applied to point releases 7.1.x > 7.2
  3. release of 7.2 beta is scheduled for December 15 (I suggest to confirm or postpone that date around Dec 10)
  4. Two-phase testing (as suggested) seems to be accepted but the length will be much shorter
  5. For the moment cloud users will not be provided with a beta branch of their actual database (as suggested in my DRAFT) but should conduct testing on the beta.erpnext.com

I’ll release a new branch for this procedure here in the coming days and suggest to stick to it for the first beta test and THEN see whether it needs to be adjusted

@nabinhait can you add the details which branched you will create for the Beta Testings (one of the things I didn’t quite catch in the hangout)

1 Like

I am sure if you will be able to cater to the needs of that group (companies selling ERPNExt services [maybe including hosting] who will have technical and operational know-how), such contributions will come automatically. I think the Odoo Community is a phantastic example for that (especially given the fact the Odoo S.A may not even have taken the needs of their community into consideration, or even acted against them at many points really)

One thing that I think should be considered is the following scenario.

We have a testing server. We will always bench update on the testing server and test properly. After we test, then we will update our production server.

However, we frequently run into an issue where lets say we upgrade our testing server from 7.1.25 to 7.1.32. By the time we finish testing, master may be on 7.1.36 or something like that. So, we are constantly running behind in our bench updates.

It would be good it it would be possible to bench update up to a particular release point. In that scenario, if we’re running 7.1.25, we would be able to run a command like “bench update --release 7.1.32” and update to 7.1.32 rather than the current 7.1.36. After we reach 7.1.32, we can test 7.1.36 and update appropriately.

I’m not sure how exactly something like this would be possible, this would really help us meet the rigors of keeping up with the rapid releases currently happening while still maintaining quality.

1 Like

I think the first release this applies to (7.2.0) is scheduled to start on Dec 15th, right?

@nabinhait is that still the schedule?

Furthermore can you specify again here which branches you will create for the Beta Testings (one of the things I didn’t quite catch in the hangout about this)

@felix I totally agree with this idea.:+1:

@vrms Yes, release date is still 15th December for v7.2.0 beta. We will create a branch named “v7.2.0-beta”, develop will be merged into that branch.

@felix The upgrade from 7.1.25 to 7.1.32 is based on hot (critical) fixes, which you should apply as soon as possible. This fixes is generally to solve some critical issues which are there in v7.1.25. Your point is totally valid for minor releases, but there is generally more than a month gap between 2 minor release. The problem in adding such options is the dependency to the correct version of frappe, for any specific release for erpnext.

1 Like

so, I assume all problems should be reported int the issues of that ‘v7.2.0 beta’ branch, right?
Furthermore I guess assume ‘develop’ will be inactive during the beta procedures.

Can you please take a look and confirm the updated timeline here?

1 Like

hi Nabin,

I think we may be the ones steering the upcoming beta process for 7.0.2 release. Looking forward to work with you closely on this. Generally my approach is to get the administrantional things off your plate a little so you can focus on code

As I haven’t heard anyting about the updated schedule here erpnext-beta-testing-process/timeline.md at master · vrms/erpnext-beta-testing-process · GitHub I’ll assume you agree

I suggest we handle this as below

  1. you are going to announce beta release in the forum
  2. I will add the concrete schedule (with dates) in the same topic and will also push updates once a new step is starting.
  3. Also I will try to guide people and comments to the issues of the ‘v7.0.2 beta’ branch on github (away from the forum) and try to help moderate the overall procedure

I think it’ll be helpful to concentrate the discussion on one location and would think the github issues are the ideal place rather then the forum.

maybe it would be helpful to get a direct line open between you and me to being able to converse quickly about whatever might come up? I’ll send you my skype in a PM

FYI. I am currently in Germany (UTC+1)

3 Likes

We are ready for this testing. Please let me know when it’s available.

2 Likes

Made a post on the forum about the updates on the v7.2.0 release and testing process.

2 Likes

Hi Nabin, are you aware of any definitive guide(s) on testing for frappe apps?

UPDATE

  1. Youtube links
    Writing & Running Unit Tests in Your Frappe Apps! - YouTube
    Session 9: Testing and Debugging - YouTube
    Barista - Automate Frappe apps testing - Karthikeyan Y | ERPNext Conference 2022 - YouTube