Version 11 is a very rough release for us. We are constantly fixing bugs and other bugs are popping out from nowhere. Most of these bugs are related to UI interactions.
That’s why we have decided to bring back UI tests. We have tried writing UI tests before but they have failed for multiple reasons.
Tests were directly coupled to Frappe API
Tests would sometime fail (not deterministic)
Tests took a long time to run
Solutions
Write tests that have no knowledge of how frappe works, we just give simple commands like visit a page, or click a button.
Write tests that don’t depend on each other and can be tested in isolation. Also tests should not depend on explicit timeouts.
Larger test suites will take a long time. This is inevitable. But if we implement the 2nd solution correctly, we can run tests in parallel.
Cypress makes it possible to achieve all of the above and also makes writing UI tests fun. Cypress has its own dashboard which records videos and screenshots of the tests that ran. You can take a look at it here.
Running UI tests in Cypress:
All of these tests run independently. None of them uses any frappe api, nor uses any timeouts for things to complete. These are the core principles of testing with Cypress, and I think they are important for robust test suites.
Since these tests do not depend on the frappe api, we will have to re-write any UI tests that we wrote for the previous system. We will be doing that over the course of the next 6 months.
If you are interested in helping us write UI tests, please check out the documentation:
Dear All,
While understand the developer’s enthusiasm surrounding cypress, it comes at a huge price: The developers of cypress expressively refuse to support FreeBSD (which is realy not that far away from OSX, isn’t it?). This means that I cannot upgrade from version 10 to 11 on my servers, which have been running rock-solid and stable FreeBSD ever since.
A possible way out would be to remove cypress as requirement for a production environment and modify the frappe installation instructions accordingly. So, for folks like myself that run a production environment on superbly stable FreeBSD, we do not seem to need cypress (correct me, if I got this wrong), and therefore could happily do without it.
What do you think?
For now, I have to revert to version 10. There is just no way I can use version 11. This is a bit of a shame…
Chris
I have just used git to download the very latest stable version. Funny enough, the cypress error has disappeared, but year does not play ball:
I have been following the manual frappe bench installation instructions to the dot, running FreeBSD 11.2, with ERPNext inside a jail.
pip etc. work fine, and installs the required dependencies from the Internet without any issues.
yarn (or npm) has issues, please see the attached screenshot.
Dear All,
Another fresh install of the latest production git clone on FreeBSD 11.2. Here are the error messages I get caused by cypress:
Updating node packages...
INFO:bench.utils:yarn install
yarn install v1.13.0
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "rollup-plugin-vue > @vue/component-compiler@3.4.4" has unmet peer dependency "postcss@>=6.0".
[4/4] Building fresh packages...
[1/3] ⠂ uws
[2/3] ⠂ cypress
[1/3] ⠄ uws
[1/3] ⠈ uws
[1/3] ⢀ uws
[1/3] ⠠ uws
[2/3] ⠠ cypress
error /usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress: Command failed.
Exit code: 1
Command: node index.js --exec install
Arguments:
Directory: /usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress
Output:
/usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress/lib/tasks/state.js:39
throw new Error('Platform: "' + platform + '" is not supported.');
^
Error: Platform: "freebsd" is not supported.
at getPlatFormBinaryFolder (/usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress/lib/tasks/state.js:39:13)
at Object.getBinaryDir (/usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress/lib/tasks/state.js:65:44)
at Object.start (/usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress/lib/tasks/install.js:177:25)
at Object.<anonymous> (/usr/local/frappe_prd/frappe-bench/apps/frappe/node_modules/cypress/index.js:13:36)
at Module._compile (internal/modules/cjs/loader.js:734:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:745:10)
at Module.load (internal/modules/cjs/loader.js:626:32)
at tryModuleLoad (internal/modules/cjs/loader.js:566:12)
at Function.Module._load (internal/modules/cjs/loader.js:558:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:797:12)
at executeUserCode (internal/bootstrap/node.js:526:15)
[4/4] Building fresh packages...
success Installed "node-gyp@3.8.0" with binaries:
- node-gyp
[#######################################################################################################] 103/103Traceback (most recent call last):
File "/usr/local/frappe_prd/bin/bench", line 11, in <module>
load_entry_point('bench', 'console_scripts', 'bench')()
File "/usr/local/frappe_prd/bench-repo/bench/cli.py", line 40, in cli
bench_command()
File "/usr/local/frappe_prd/lib/python2.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/local/frappe_prd/lib/python2.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/frappe_prd/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/frappe_prd/lib/python2.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/frappe_prd/lib/python2.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/frappe_prd/bench-repo/bench/commands/make.py", line 30, in init
python = python)
File "/usr/local/frappe_prd/bench-repo/bench/utils.py", line 83, in init
update_node_packages(bench_path=path)
File "/usr/local/frappe_prd/bench-repo/bench/utils.py", line 449, in update_node_packages
update_yarn_packages(bench_path)
File "/usr/local/frappe_prd/bench-repo/bench/utils.py", line 462, in update_yarn_packages
exec_cmd('yarn install', cwd=app_path)
File "/usr/local/frappe_prd/bench-repo/bench/utils.py", line 159, in exec_cmd
raise CommandFailedError(cmd)
bench.utils.CommandFailedError: yarn install
Kindly note that I follow the manual installation instructions for Frappé, as per your Github documentation.
I think it would make sense to change the requirements to get rid off the need for cypress for production installations. What is you take?
Thank you!
Chris
@netchampfaris Can you please explain what type of ui tests we should use for developing our application on frappe framework that integrates with ERPNext - Cypress or QUinit? QUnits test files create for a new doctype by default in production vm image of ERPNext.
I brought up a discussion around UI tests as our team asked for this here:
I would like to know, what we can do to create UI tests? What is the status in the whole community? Is there any progress on this so far? How can we help?
