Custom App Installation breaks on Fixture Import due to wrong Order of Imports

tgraupne · February 5, 2024, 4:02pm

Hey everybody

I have a weird issue regarding my custom app and fixtures and I hope someone might be able to help me.

The Setting

Reading through the documentation and a lot of posts in this forum, I came to the conclusion that create a Custom App is the way to go for me. The basics are quite simple:

there are a couple of Custom Field I need to add to some DocTypes belonging to the Stock and Buying Module of ERPNext
I need to create a couple of Suppliers, Item Groups, Item Variants and Items
I want to make those changes portable and need to ability to install them in testing- and production-environments

So, working with bench export-fixtures, I found out I probably need to filter my fixtures inside hooks.py. Because if I don’t do this, my exported json files contain a lot of stuff from other apps and module. Ok, here is the important part of my hooks.py

fixtures = [
    "Address",
    {
        "doctype": "Supplier Group",
        "filters": [["name", "in", [
            "Primary Processor",
        ]]]
    },
    "Supplier",
    {
        "doctype": "Item Group",
        "filters": [["name", "in", [
            "Packaging",
            "Plastics"
        ]]]
    },
    {
        "doctype": "Item Attribute",
        "filters": [["name", "in", [
            "Material Form",
            "Plastic Type"
        ]]]
    },
    "Item",
    {
        "doctype": "Custom Field",
        "filters": {
            "module": ["=", "Bestandsmeldung EBH"]
        }
    },
]

So far so good. Executing bench export-fixtures created a couple of json files inside the fixtures subfolder and they look good.

The Problem

However, when I try to install the app in another environment, I get strange error:

builtins.AttributeError: 'NoneType' object has no attribute 'lft'

So, what I have found out is, that his happens here:

File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 912, in fn
    return method_object(*args, **kwargs)
      self = <Item: MAT>
      args = ()
      kwargs = {}
      method_object = <bound method Item.on_update of <Item: MAT>>
      method = 'on_update'
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 124, in on_update
    invalidate_cache_for_item(self)
      self = <Item: MAT>
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 1134, in invalidate_cache_for_item
    invalidate_cache_for(doc, doc.item_group)
      doc = <Item: MAT>
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/setup/doctype/item_group/item_group.py", line 203, in invalidate_cache_for
    for d in get_parent_item_groups(item_group):
      doc = <Item: MAT>
      item_group = 'Plastics'
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/setup/doctype/item_group/item_group.py", line 192, in get_parent_item_groups
    (item_group.lft, item_group.rgt),
      item_group_name = 'Plastics'
      from_item = False
      settings = <ECommerceSettings: E Commerce Settings>
      base_nav_page = {'name': 'All Products', 'route': '/all-products'}
      base_parents = [{'name': 'Home', 'route': '/'}, {'name': 'All Products', 'route': '/all-products'}]
      item_group = None
builtins.AttributeError: 'NoneType' object has no attribute 'lft'

Apparently, bench is importing my Item Template MAT and while doing this, it tries to “invalidate a cache”. For some reason, they need to query the Item Group of my Item MAT. But, its Item Group Plastics hasn’t been imported yet. This import would be part of importing the file: item_group.json.

So, the installation of my app broke, but somehow, if I try it again, bench tells me the app has been installed already

Subsequently executing bench migrate created the same error.

Workaround

But, what I was able to find out is, that things work out, if I manually manipulate the “order of imports” by moving json files out and in the fixtures folder.

The Big Question

So, my question would be, is it possible to enable bench to export and import everything in the correct order to prevent such errors?

Void_Moon · February 5, 2024, 4:42pm

Could you try placing your item_group fixture after the item fixture in the hooks file and try again?

tgraupne · February 16, 2024, 2:43pm

Hey @Void_Moon , thank you for your feedback.

I did change the order of entries in hooks.py:

fixtures = [
    "Address",
    {
        "doctype": "Supplier Group",
        "filters": [["name", "in", [
            "Primary Processor",
        ]]]
    },
    "Supplier",
    {
        "doctype": "Item Attribute",
        "filters": [["name", "in", [
            "Material",
            "Type"
        ]]]
    },
    "Item",
    {
        "doctype": "Item Group",
        "filters": [["name", "in", [
            "Packaging",
            "Plastics"
        ]]]
    },
    {
        "doctype": "Custom Field",
        "filters": {
            "module": ["=", "EBH"]
        }
    },
]

When executing bench export-fixtures I noticed, nothing changed in the exported json files. But I also did not know if I should have expected things to change.

However, things also did not change during app installation. I still get the same error:

  File "apps/erpnext/erpnext/stock/doctype/item/item.py", line 124, in on_update
    invalidate_cache_for_item(self)
  File "apps/erpnext/erpnext/stock/doctype/item/item.py", line 1134, in invalidate_cache_for_item
    invalidate_cache_for(doc, doc.item_group)
  File "apps/erpnext/erpnext/setup/doctype/item_group/item_group.py", line 203, in invalidate_cache_for
    for d in get_parent_item_groups(item_group):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "apps/erpnext/erpnext/setup/doctype/item_group/item_group.py", line 192, in get_parent_item_groups
    (item_group.lft, item_group.rgt),
     ^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lft'

However, the execution path to the error does go through invalidate_cache_for(doc, doc.item_group) anymore.

Does anybody have another idea maybe?

tgraupne · February 20, 2024, 1:15pm

While researching my issue I have found a lot of incidents where people all ofer over had similar issues.

There even was a PR merged on github introducing a deterministic import mechanism for fixtures:

Unfortunately, this fix hasn’t made it into v14 oder even v15 of the frappe framework yet

tgraupne · February 21, 2024, 3:33pm

After talking to the person who provided a fix and validating together, that this code never made it into v14 or v15. I opened a new issue on GitHub:

github.com/frappe/frappe

Deterministic fixture import order still missing in v14 and v15

opened 03:30PM - 21 Feb 24 UTC

tgraupne

bug

## Description of the issue Last year, @Tom-Finke raised a bug report about the import order of fixtures: #20852 He also linked this issue to open topics on discuss.frappe.io. He even provided a fix which was merged in December of 2023: #22210 While working with ERPNext on `Item Attributes`, `Items` and `Item Variants`, I noticed that this bug is still present in v14. After following up with @Tom-Finke , we were able to confirm that while this was merged to develop, this fix never made it into v14 or even v15. Are there any plans to bring this bug fix of the frappe fixtures feature to v14 or v15? **Output of `bench version`** ``` erpnext 14.61.3 frappe 14.64.0 ``` ## Steps to reproduce the issue 1. Create `Item Attributes` 2. Create an `Item Template` based on those attributes 3. Create actual `Item Variants` based on this template 4. export these changes as a fixture with `bench export-fixture` to you custom module 5. try to install this custom module in a clean installation of frappe and erpnext ### Observed result The installation fails like it is described here: https://discuss.frappe.io/t/custom-app-installation-breaks-on-fixture-import-due-to-wrong-order-of-imports/116780 `builtins.AttributeError: 'NoneType' object has no attribute 'lft'` ### Expected result The fixtures which just have been exported with bench should be installable. ### Stacktrace / full error message ``` File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 912, in fn return method_object(*args, **kwargs) self = <Item: MAT> args = () kwargs = {} method_object = <bound method Item.on_update of <Item: MAT>> method = 'on_update' File "/home/frappe/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 124, in on_update invalidate_cache_for_item(self) self = <Item: MAT> File "/home/frappe/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 1134, in invalidate_cache_for_item invalidate_cache_for(doc, doc.item_group) doc = <Item: MAT> File "/home/frappe/frappe-bench/apps/erpnext/erpnext/setup/doctype/item_group/item_group.py", line 203, in invalidate_cache_for for d in get_parent_item_groups(item_group): doc = <Item: MAT> item_group = 'Plastics' File "/home/frappe/frappe-bench/apps/erpnext/erpnext/setup/doctype/item_group/item_group.py", line 192, in get_parent_item_groups (item_group.lft, item_group.rgt), item_group_name = 'Plastics' from_item = False settings = <ECommerceSettings: E Commerce Settings> base_nav_page = {'name': 'All Products', 'route': '/all-products'} base_parents = [{'name': 'Home', 'route': '/'}, {'name': 'All Products', 'route': '/all-products'}] item_group = None builtins.AttributeError: 'NoneType' object has no attribute 'lft' ``` ## Additional information As a workaround, it is possible to manually move the `Items` fixture file `items.json` to a different folder. Install and migrate the custom module, move `item.json` back into the `fixtures` folder and execute `bench migrate` again.