Run-tests --skip-test-records deprecated in v16: duplicate errors when running without the flag

Frappe Framework
,
1

We run tests with bench run-tests (by module or doctype). On an older Frappe version we used --skip-test-records so that test runs did not create (or re-create) test records and we could avoid duplicate-related issues.

What we see in v16

  1. With --skip-test-records

We get:

  • --skip-test-records is deprecated and without effect!

  • All records are loaded lazily on first use, so the flag is useless, now.

  • Simply remove the flag.

After that the command returns without running any tests. So we cannot use this flag anymore.

  1. Without --skip-test-records

Tests do run, but we hit duplicate errors (e.g. unique constraint violations or similar) when the test framework creates test records. We did not see these errors in the old version when we used --skip-test-records.

What we expected

  • Either:

  • A way to run tests without creating (or re-creating) test records, similar to the old --skip-test-records behaviour, or

  • A clear migration path: how we should run tests in v16 when we used to rely on --skip-test-records to avoid duplicate record creation.

  • If the new behaviour is “records are loaded lazily on first use”, we expected that:

  • Using the flag would not stop tests from running (e.g. only a deprecation warning), and/or

  • Running without the flag would not cause duplicate errors in our setup.

Questions

  1. What is the recommended way in v16 to run tests when we previously depended on --skip-test-records to avoid duplicate record creation?

  2. Is it expected that removing the flag leads to duplicate errors in some cases (e.g. due to unique constraints other than the primary key)? If yes, is the intended approach to fix test data / test record definitions so they don’t violate those constraints?

  3. Should we always run tests on a clean test site (or clear something like .test_records.jsonl) to avoid these duplicates?

Environment

  • Frappe v16

  • Running: bench run-tests --module “…” or bench run-tests --doctype “…”

  • Scripts that used to call bench run-tests --skip-test-records --doctype “…” now either show the deprecation and exit, or we remove the flag and get duplicate errors.

Thanks in advance for any guidance.