Error in Amazon MWS integration: RuntimeError: dictionary changed size during iteration

Integration
1

I am attempting to integrate Amazon MWS with a new ERPNext installation. Install went fine with no issues… running in developer mode.

When I fill in the Amazon MWS details then hit ‘Synch Products’, the following error pops up:

RuntimeError: dictionary changed size during iteration

This is what I see on the bench terminal (masked IP as 0.0.0.0):

23:10:08 web.1 | 0.0.0.0 - - [07/Aug/2019 23:10:08] “POST /api/method/runserverobj HTTP/1.1” 500 -
23:10:08 web.1 | INFO:werkzeug:0.0.0.0 - - [07/Aug/2019 23:10:08] “POST /api/method/runserverobj HTTP/1.1” 500 -

Any ideas what is going on?

2

Hi welcome! This means probably an illegal operation.

To know what is going on you need to refer to the logs on either server.

You need to look for a error stack traceback for that RuntimeError.

3

Thanks - I am posting the log for this error below:

[ERROR] 2019-08-08 21:21:09,887 | /home/horballs/frappe-bench/apps/frappe/frappe/utils/error.py:
Could not take error snapshot: startswith first arg must be str or a tuple of str, not bytes
Traceback (most recent call last):
  File "/home/horballs/frappe-bench/apps/frappe/frappe/app.py", line 60, in application
    response = frappe.api.handle()
  File "/home/horballs/frappe-bench/apps/frappe/frappe/api.py", line 55, in handle
    return frappe.handler.handle()
  File "/home/horballs/frappe-bench/apps/frappe/frappe/handler.py", line 21, in handle
    data = execute_cmd(cmd)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/handler.py", line 56, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/__init__.py", line 1036, in call
    return fn(*args, **newargs)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/handler.py", line 84, in runserverobj
    frappe.desk.form.run_method.runserverobj(method, docs=docs, dt=dt, dn=dn, arg=arg, args=args)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/desk/form/run_method.py", line 36, in runserverobj
    r = doc.run_method(method)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py", line 786, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py", line 1055, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py", line 1038, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py", line 780, in <lambda>
    fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_settings.py", line 22, in get_products_details
    get_products_details()
  File "/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_methods.py", line 23, in get_products_details
    report_id = request_and_fetch_report_id("_GET_FLAT_FILE_OPEN_LISTINGS_DATA_", None, None, market_place_list)
  File "/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_methods.py", line 90, in request_and_fetch_report_id
    marketplaceids=marketplaceids)
  File "/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py", line 373, in request_report
    return self.make_request(data)
  File "/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py", line 164, in make_request
    extra_data = remove_empty(extra_data)
  File "/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py", line 74, in remove_empty
    for key in d.keys():
RuntimeError: dictionary changed size during iteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/horballs/frappe-bench/apps/frappe/frappe/utils/error.py", line 36, in make_error_snapshot
    snapshot = get_snapshot(exception)
  File "/home/horballs/frappe-bench/apps/frappe/frappe/utils/error.py", line 131, in get_snapshot
    if type(value)==str and value.startswith(b"u'"):
TypeError: startswith first arg must be str or a tuple of str, not bytes
4

ok the problem is in this loop

https://github.com/frappe/erpnext/blob/develop/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py#L69

Maybe try this: for i in list(d) that apparently works in python 2.7 or 3.x

This explains the problem python - How to avoid "RuntimeError: dictionary changed size during iteration" error? - Stack Overflow

5

Great - seemed to fix that issue. Brings us a new one:

AttributeError: module 'urllib' has no attribute 'quote'

Here is the trace:

ERROR] 2019-08-11 18:56:25,779 | /home/horballs/frappe-bench/apps/frappe/frappe/utils/error.py:
Could not take error snapshot: startswith first arg must be str or a tuple of str, not bytes
Traceback (most recent call last):
File “/home/horballs/frappe-bench/apps/frappe/frappe/app.py”, line 60, in application
response = frappe.api.handle()
File “/home/horballs/frappe-bench/apps/frappe/frappe/api.py”, line 55, in handle
return frappe.handler.handle()
File “/home/horballs/frappe-bench/apps/frappe/frappe/handler.py”, line 21, in handle
data = execute_cmd(cmd)
File “/home/horballs/frappe-bench/apps/frappe/frappe/handler.py”, line 56, in execute_cmd
return frappe.call(method, **frappe.form_dict)
File “/home/horballs/frappe-bench/apps/frappe/frappe/init.py”, line 1036, in call
return fn(*args, **newargs)
File “/home/horballs/frappe-bench/apps/frappe/frappe/handler.py”, line 84, in runserverobj
frappe.desk.form.run_method.runserverobj(method, docs=docs, dt=dt, dn=dn, arg=arg, args=args)
File “/home/horballs/frappe-bench/apps/frappe/frappe/desk/form/run_method.py”, line 36, in runserverobj
r = doc.run_method(method)
File “/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py”, line 786, in run_method
out = Document.hook(fn)(self, *args, **kwargs)
File “/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py”, line 1055, in composer
return composed(self, method, *args, **kwargs)
File “/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py”, line 1038, in runner
add_to_return_value(self, fn(self, *args, **kwargs))
File “/home/horballs/frappe-bench/apps/frappe/frappe/model/document.py”, line 780, in
fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
File “/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_settings.py”, line 22, in get_products_details
get_products_details()
File “/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_methods.py”, line 23, in get_products_details
report_id = request_and_fetch_report_id(“GET_FLAT_FILE_OPEN_LISTINGS_DATA”, None, None, market_place_list)
File “/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_methods.py”, line 90, in request_and_fetch_report_id
marketplaceids=marketplaceids)
File “/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py”, line 373, in request_report
return self.make_request(data)
File “/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py”, line 175, in make_request
request_description = ‘&’.join([‘%s=%s’ % (k, urllib.quote(params[k], safe=‘-.~‘).encode(‘utf-8’)) for k in sorted(params)])
File “/home/horballs/frappe-bench/apps/erpnext/erpnext/erpnext_integrations/doctype/amazon_mws_settings/amazon_mws_api.py”, line 175, in
request_description = ‘&’.join([’%s=%s’ % (k, urllib.quote(params[k], safe='-.~’).encode(‘utf-8’)) for k in sorted(params)])
AttributeError: module ‘urllib’ has no attribute ‘quote’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/horballs/frappe-bench/apps/frappe/frappe/utils/error.py”, line 36, in make_error_snapshot
snapshot = get_snapshot(exception)
File “/home/horballs/frappe-bench/apps/frappe/frappe/utils/error.py”, line 131, in get_snapshot
if type(value)==str and value.startswith(b"u’"):
TypeError: startswith first arg must be str or a tuple of str, not bytes

6

Using python 2.7.12 by the way

7

Glad to help! Just to give you a hint, next look up this for pointers :slight_smile:

8

I started by trying to figure out the issue - but that error is a python 3.0 error and I have 2.7. I added this as a workaround but it does not fix it:

try:
    from urllib import quote  # Python 2.X
except ImportError:
    from urllib.parse import quote  # Python 3+
9

Then try this:

from six.moves.urllib.parse import quote

as this suggests Python: Importing urllib.quote - Stack Overflow

10

Doesn’t fix it - same issue.

Seems like a really odd error, since I am using python 2.7 and urllib didn’t change until python 3.0.