Bench run-tests leads to KeyError: u'introduction text'

I did a fresh install and ran bench run-tests. It ran well the first time, but after that started giving me this error.
Anyone has ideas on how to fix this?

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands.py", line 29, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands.py", line 790, in run_tests
    force=context.force, profile=profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 41, in main
    ret = run_all_tests(app, verbose, profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 76, in run_all_tests
    _add_test(path, filename, verbose, test_suite=test_suite)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 165, in _add_test
    make_test_records(doctype, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 182, in make_test_records
    make_test_records_for_doctype(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 227, in make_test_records_for_doctype
    frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 274, in make_test_objects
    d.insert()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 193, in insert
    self.run_before_save_methods()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 611, in run_before_save_methods
    self.set_title_field()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 303, in set_title_field
    self.set(df.fieldname, df.options.format(**get_values()))
KeyError: u'introduction text'

This seems to eb related to web form, but I dont know why it happens after the first test run. The first time runs smooth.
With verbose = true, here is the error trace -

Making for Email Alert
Making for Workflow Action
Please setup make_test_records for: Workflow Action
------------------------------------------------------------
Autoname: field:workflow_action_name
Mandatory Fields:
Workflow Action:workflow_action_name | Data |

Making for Workflow State
Making for Workflow
Making for Web Form
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands.py", line 29, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands.py", line 790, in run_tests
    force=context.force, profile=profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 42, in main
    ret = run_all_tests(app, verbose, profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 77, in run_all_tests
    _add_test(path, filename, verbose, test_suite=test_suite)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 166, in _add_test
    make_test_records(doctype, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 183, in make_test_records
    make_test_records_for_doctype(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 228, in make_test_records_for_doctype
    frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 275, in make_test_objects
    d.insert()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 193, in insert
    self.run_before_save_methods()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 611, in run_before_save_methods
    self.set_title_field()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 303, in set_title_field
    self.set(df.fieldname, df.options.format(**get_values()))
KeyError: u'introduction text'

Guys, I have the same problem.

Bench version: 4.1.0
ERPNext version: 7.0.21
Frappe version: 7.0.18

Step to reproduce:

I have run tests with our custom module and some tests are failed.(Sorry for IMG I have lost log)

bench run-tests

http://s017.radikal.ru/i437/1608/81/518e0801259e.png

Then I have restarted tests and get new error “KeyError: u’introduction text’”

bench run-tests

frappe@erpnext:~/frappe-bench$ bench run-tests
    Traceback (most recent call last):
      File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
        main()
      File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
        click.Group(commands=commands)(prog_name='bench')
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
        return self.main(*args, **kwargs)
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
        rv = self.invoke(ctx)
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
        return callback(*args, **kwargs)
      File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 24, in _func
        ret = f(frappe._dict(ctx.obj), *args, **kwargs)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/utils.py", line 302, in run_tests
        force=context.force, profile=profile)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py", line 39, in main
        ret = run_all_tests(app, verbose, profile)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py", line 74, in run_all_tests
        _add_test(app, path, filename, verbose, test_suite=test_suite)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py", line 164, in _add_test
        make_test_records(doctype, verbose)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py", line 181, in make_test_records
        make_test_records_for_doctype(options, verbose, force)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py", line 226, in make_test_records_for_doctype
        frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
      File "/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py", line 273, in make_test_objects
        d.insert()
      File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 212, in insert
        self.run_before_save_methods()
      File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 703, in run_before_save_methods
        self.set_title_field()
      File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 344, in set_title_field
        self.set(df.fieldname, df.options.format(**get_values()))
    KeyError: u'introduction text'

@rmehta, @Ben_Cornwell_Mott, All: have you ever faced with the issues like this one? Will you be able to suggest the right approach to fix/tackle it?

Error [quote=“Katerina_Romanchuk, post:3, topic:13068”]
“KeyError: u’introduction text’”
[/quote]
caused due to wrong field-name / links / status etc. in doctype.

my guess is that the field, called “introduction text”, that is set in the Options text box in the title for the doctype (not sure what doctype this is) is either incorrect or not provided.

Since it is setting up test records, it could be that you need to provide that field in the test records for that document.

The other thing I’d note (but I’m not sure is relevant) is that you have a space between introduction and text. Since it’s referencing a field, I think it should be introduction_text (with an underscore), given that field names can’t have spaces.

try with verbose mode and put up entire log in code block while sharing.

bench --verbose run-tests

I found this case is tests.

Source: frappe/frappe/blob/develop/frappe/custom/doctype/customize_form/test_customize_form.py

  # space in formatter
  df.options = """{doc_type} - {introduction text}"""
  self.assertRaises(InvalidFieldNameError, d.run_method, "save_customization")

I have run with verbose, this is log:

frappe@erpnext:~/frappe-bench$ bench --verbose run-tests
Running “before_tests” hooks
Making for Role
Message: None
Message: None
Message: None
Making for User
Message: None
Message: None
Message: None
Message: None
Message: None
Message: None
Message: None
Making for Blogger
Message: None
Message: None
Making for Module Def
Please setup make_test_records for: Module Def

Autoname: field:module_name
Mandatory Fields:
Module Def:app_name | Data |

Making for DocType
Please setup make_test_records for: DocType

Autoname: Prompt
Mandatory Fields:
DocType:module | Link | Module Def
DocType:fields | Table | DocField

Making for Web Form
Traceback (most recent call last):
File “/usr/lib/python2.7/runpy.py”, line 162, in _run_module_as_main
main”, fname, loader, pkg_name)
File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 79, in
main()
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 16, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in _func
ret = f(frappe._dict(ctx.obj), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/utils.py”, line 302, in run_tests
force=context.force, profile=profile)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 39, in main
ret = run_all_tests(app, verbose, profile)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 74, in run_all_tests
_add_test(app, path, filename, verbose, test_suite=test_suite)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 164, in _add_test
make_test_records(doctype, verbose)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 181, in make_test_records
make_test_records_for_doctype(options, verbose, force)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 226, in make_test_records_for_doctype
frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 273, in make_test_objects
d.insert()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 212, in insert
self.run_before_save_methods()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 703, in run_before_save_methods
self.set_title_field()
File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 344, in set_title_field
self.set(df.fieldname, df.options.format(**get_values()))
KeyError: u’introduction text’

Related links: [FAILED TEST] - TestCustomizeForm (KeyError: u’introduction text’) · Issue #6065 · frappe/erpnext · GitHub

@rushabh_mehta, @Ben_Cornwell_Mott, All:

We just wanted to take the chance to better understand the process/use of Travis (Travis CI - Test and Deploy Your Code with Confidence) in this particular case.

  1. How will it help to eliminate the issue with failed tests on the repeatable runs of the test suites? (where the probable issue is some of the test data break the consistency of the database without recovering the system to the state exactly like one it was in prior to the initial test run). We opened such issues in github per [FAILED TEST] - TestCustomizeForm (KeyError: u’introduction text’) · Issue #6065 · frappe/erpnext · GitHub and [FAILED TEST] - Tests are failed after repeated test execution · Issue #6066 · frappe/erpnext · GitHub, and now we see those issues simply closed there
  2. How will travis help on the simplier issues where ordinary tests fail immediately - like in the case of [FAILED TEST] - frappe.core.doctype.user.test_user.TestUser (AssertionError: 501 != 417) · Issue #6064 · frappe/erpnext · GitHub we posted as well?

Sorry if our questions sound in a lamming way. We still learn ERPNext/Frappe in order to become the effective consultants/contributors.

Since we could not find any documentation on how testing framework is tied back to Travis frontier for ERPNext frappe, we are a little confused now.

Please help us to understand this better. Please also explain the way the issues with repeatable test runs are going to be fixed soon.

Thank you.

George

https://github.com/frappe/erpnext/blob/develop/.travis.yml

@rmehta: thanks for sharing the code exempt :slight_smile:

I still wonder on

  1. How will it help to eliminate the issue with failed tests on the repeatable runs of the test suites? (I think it runs all of the tests once - in cases where the probable issue is some of the test data breaking the consistency of the database after the first run of the tests, it may need some human action :slight_smile: ). We opened such issues in github per [FAILED TEST] - TestCustomizeForm (KeyError: u’introduction text’) · Issue #6065 · frappe/erpnext · GitHub and [FAILED TEST] - Tests are failed after repeated test execution · Issue #6066 · frappe/erpnext · GitHub, and now we see those issues simply closed there - could you please let us know how travis handles these types of problems?
  2. How will travis help on the simpler issues where ordinary tests fail immediately - like in the case of [FAILED TEST] - frappe.core.doctype.user.test_user.TestUser (AssertionError: 501 != 417) · Issue #6064 · frappe/erpnext · GitHub we posted as well?

@rmehta: may I ask you to help us understand the points below?

  1. How will travis help to eliminate the issue with failed tests on the repeatable runs of the test suites? (I think it runs all of the tests once - in cases where the probable issue is some of the test data breaking the consistency of the database after the first run of the tests, it may need some human action :slight_smile: ). We opened such issues in github per [FAILED TEST] - TestCustomizeForm (KeyError: u’introduction text’) · Issue #6065 · frappe/erpnext · GitHub and [FAILED TEST] - Tests are failed after repeated test execution · Issue #6066 · frappe/erpnext · GitHub, and now we see those issues simply closed there - could you please let us know how travis handles these types of problems?
  2. How will travis help on the simpler issues where ordinary tests fail immediately - like in the case of https://github.com/frappe/erpnext/issues/60641 we posted as well?

@gvyshnya the whole point of automated testing is that you don’t need human intervention.

You need to setup your local environment in a way it is in the tests

@rmehta: thanks for the suggestion, this is really useful. Could you please review what we do correctly (or equally incorrectly) in terms of setting up our local environments and advise on possible changes/corrections there? Here is what we do locally

  1. Deploy a new local VM on Oracle Virtual box, using the production ERP Next VM image from https://erpnext.com/download
  2. When the new VM started, we do bench update command to raise the version up to the latest release branch you endorsed in the github repo
  3. Install the latest ubuntu package and security updates
  4. Simply start runnig the test suites using bench run-tests command

We then start seeing issues like those we depicted in
https://github.com/frappe/erpnext/issues/6041

https://github.com/frappe/erpnext/issues/6066

Please share your suggestions on possible corrections in our local setup/method of running the functional/unit tests.

Thank you.

@gvyshnya you should be able to debug them! I think setting the correct “hostname” should work

Also no need to run the full suite locally, just run your own tests and see if it works!

@rmehta: thanks for your comments and suggesting.

Eventually we grabbed your scripts for setting up the tests / testing framework for Travis and re-implemented them to fit to our environment and Continuous Integration strategy for the custom applications being developed by our team. In such a way, we stopped seeing such problems on generic doc types shipped with the ERP Next platform by default (although they are still present in if running tests in a straight-forward manner, via bench commands explained in https://github.com/frappe/frappe/blob/master/frappe/docs/user/en/guides/basics/writing-tests.md )

The bottom line is as follows

  • we no more suffer from the issues explained in this thread
  • the issues in the thread still exist, if trying to run the unit tests in a straight-forward way, via direct use of respective bench commands
  • it is your call as a product owner to decide whether there is any priority with the fixes to the issues above

thanks for the valuable discussion. We learned useful things while trying to find the work-around.