[v11] Why am I getting a Unicode Error in Production Server but not on my test environment for special characters?

adityaduggal · January 10, 2019, 5:17am

Hi,
I am facing an issue basically I have code which would generate the description of the Item Code based on the variants of an item code, now the problem is that I have some characters which are causing issues in my production server and not in my test environment both are on v11-staging so the only difference which I can see is that the production server is on python 2.7 and test env is on python 3.0

The error is as below:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/utils.py", line 129, in execute
    ret = frappe.get_attr(method)(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/rigpl_erpnext/rigpl_erpnext/rigpl_erpnext/scheduled_tasks/variant_copy.py", line 13, in check_wrong_variants
    copy_from_template()
  File "/home/frappe/frappe-bench/apps/rigpl_erpnext/rigpl_erpnext/rigpl_erpnext/scheduled_tasks/variant_copy.py", line 45, in copy_from_template
    it_doc.save()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 260, in save
    return self._save(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 296, in _save
    self.run_before_save_methods()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 876, in run_before_save_methods
    self.run_method("validate")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 772, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1048, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1033, in runner
    add_to_return_value(self, f(self, method, *args, **kwargs))
  File "/home/frappe/frappe-bench/apps/rigpl_erpnext/rigpl_erpnext/rigpl_erpnext/item.py", line 23, in validate
    generate_description(doc,method)
  File "/home/frappe/frappe-bench/apps/rigpl_erpnext/rigpl_erpnext/rigpl_erpnext/item.py", line 357, in generate_description
    concat1 = concat1 + str(suffix[0][0][1:-1])
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201d' in position 0: ordinal not in range(128)

Now the 2 special characters I want to show in my item description which are causing this issue are:

and

So I have tried using the symbol directly from character map where it causes the above error and also I have tried &#216 or &#176 in there place but problem with using this instead of the symbol directly from the character map is that it would export the reports with the similar &#216 into excel.

Now I am kind of stuck between python unicode error and usability.
If I use the symbol Ø or º in description I am getting a unicode error in python 2.5 but it works fine with exporting of the reports into excel

But if I use &#216 or &#176 instead of symbols I am not getting the error in python 2.5 but it would then export these symbols as values in excel which causes a lot of confusion.

Also I am very much perplexed as to why this same code is working perfectly with my test env on my notebook with same DB and same code, the only reason I can think of is that it has python 3.0

Please any one can tell me what should I do get this error removed.

adityaduggal · January 10, 2019, 6:38am

Turns out that my fears were right, python 3 renamed unicode to str and str has been replaced by bytes

So this is the reason why my special characters are going fine in my test environment since the python there is updated as I installed a develop branch there and I guess there is python 3 by default but in my production server the python version being used is 2.7 or thereabouts.

Now the question turns to how do I upgrade my python in production server so that I can easily use code.

adityaduggal · January 10, 2019, 11:23am

Does anyone know how to update to python3 since I cannot figure out why my production server did not upgrade to python3 when I upgraded the bench to ver_11.

clarkej · January 10, 2019, 1:38pm

Try something like this ERPNext V11, Python 3 or 2.7 - #6 by clarkej

adityaduggal · January 11, 2019, 7:28am

Thanks @clarkej for the link but when I trying to migrate I am getting the below error:

[frappe@www frappe-bench]$ bench migrate-env python3
DEBUG:bench.commands:Clearing Redis Cache...
INFO:bench.utils:/usr/bin/redis-cli -p 13000 FLUSHALL
OK
DEBUG:bench.commands:Clearing Redis DataBase...
INFO:bench.utils:/usr/bin/redis-cli -p 13000 FLUSHDB
OK
DEBUG:bench.commands:Backing up Virtual Environment
DEBUG:bench.commands:Migration Error
Traceback (most recent call last):
  File "/usr/bin/bench", line 11, in <module>
    load_entry_point('bench', 'console_scripts', 'bench')()
  File "/home/frappe/bench-repo/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/bench-repo/bench/commands/__init__.py", line 139, in migrate_env
    os.rename(source, dest)
OSError: [Errno 2] No such file or directory

aleksas · January 11, 2019, 1:44pm

Same here:

DEBUG:bench.commands:Clearing Redis Cache...
INFO:bench.utils:/usr/bin/redis-cli -p 13000 FLUSHALL
OK
DEBUG:bench.commands:Clearing Redis DataBase...
INFO:bench.utils:/usr/bin/redis-cli -p 13000 FLUSHDB
OK
DEBUG:bench.commands:Backing up Virtual Environment
DEBUG:bench.commands:Migration Error
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 11, in <module>
    load_entry_point('bench', 'console_scripts', 'bench')()
  File "/home/frappe/.bench/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/.bench/bench/commands/__init__.py", line 139, in migrate_env
    os.rename(source, dest)
OSError: [Errno 2] No such file or directory

adityaduggal · January 11, 2019, 1:58pm

@aleksas Basically this means that the bench is now broken, most of the bench commands or I guess all of the bench commands would not work now, I am kind of stuck with this on my production server.

aleksas · January 11, 2019, 2:14pm

It’s probably just missing env folder.

Recover env dir from archived_envs (take the last one and copy one level up to frappe-bench, rename to env).
run bench migrate-env python3

Worked for me.

UPDATE

Also ran following after migration:
bench setup requirements
bench --site site-name migrate
bench restart

Actually I personally ran bench migrate-env python3.6 as I was interested speciffically in 3.6 version of python.

adityaduggal · January 11, 2019, 2:18pm

Well @aleksas pardon my limited knowledge about the linux file system, but I am unable to find the env folder let alone the archived_envs folder.

Would greatly appreciate if you could guide me where to find these folders.

PS: figured out that there is a folder archived_env in my frappe-bench folder and infact the env folder is gone now, so basically I should copy the latest folder inside the archived_env to env folder, if I am not wrong.

aleksas · January 11, 2019, 2:21pm

Sure no problem. It’s not a linux, it’s more of erpnext feature i’ve noticed today.
Probably when migrating they archive environment which is located under frappe_bench folder.
…
Yes, that’s exactly right. I mean copy latest to higher level (to frappe_bench folder) and rename it to env.

adityaduggal · January 11, 2019, 2:26pm

Just tried that and now I am getting another error:

[frappe@www frappe-bench]$ bench migrate-env python3
DEBUG:bench.commands:Clearing Redis Cache...
INFO:bench.utils:/usr/bin/redis-cli -p 13000 FLUSHALL
OK
DEBUG:bench.commands:Clearing Redis DataBase...
INFO:bench.utils:/usr/bin/redis-cli -p 13000 FLUSHDB
OK
DEBUG:bench.commands:Backing up Virtual Environment
DEBUG:bench.commands:Setting up a New Virtual python3 Environment
INFO:bench.utils:/usr/bin/virtualenv --python None /home/frappe/frappe-bench/env
The path None (from --python=None) does not exist
DEBUG:bench.commands:Migration Error
Traceback (most recent call last):
  File "/usr/bin/bench", line 11, in <module>
    load_entry_point('bench', 'console_scripts', 'bench')()
  File "/home/frappe/bench-repo/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib64/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/bench-repo/bench/commands/__init__.py", line 159, in migrate_env
    ), cwd = path)
  File "/home/frappe/bench-repo/bench/utils.py", line 159, in exec_cmd
    raise CommandFailedError(cmd)
bench.utils.CommandFailedError: /usr/bin/virtualenv --python None /home/frappe/frappe-bench/env

aleksas · January 11, 2019, 2:31pm

Do you have python3 installed?

adityaduggal · January 11, 2019, 2:36pm

I did install python3 on Centos following this guide.

This I did with the root login account and now when I am checking the python version its giving an output for v2.7.5, very strange indeed.

aleksas · January 11, 2019, 2:38pm

What matters is if running python3 shows 3.* version.

adityaduggal · January 11, 2019, 2:50pm

Well Thanks for the help and finally I was able to migrate the virtual-env the command that worked for me was bench migrate-env python3.6 instead of python3, but the issue is that out of the 3 sites 1 of the sites is unable to migrate and I am getting this error below:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py", line 227, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 31, in migrate
    clear_global_cache()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/cache_manager.py", line 39, in clear_global_cache
    clear_website_cache()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 289, in clear_cache
    for method in frappe.get_hooks("website_clear_cache"):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 917, in get_hooks
    hooks = _dict(cache().get_value("app_hooks", load_app_hooks))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 68, in get_value
    val = pickle.loads(val)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xde in position 1: ordinal not in range(128)

Any ideas what could be the cause for this?

aleksas · January 11, 2019, 2:58pm

Hopefully someone can help you with that. I haven’t seen this issue.

clarkej · February 4, 2019, 3:53pm

Maybe a hooks.py is the problem?

Clearing caches or a recompile may help

find . -name '*.pyc' -delete

lasalesi · March 7, 2019, 2:44pm

I have exactly the same issue.

Configuration: fresh ERPNext production install on a Debian stretch. The installer had puller Python 2, switched to Python 3 using

$ sudo bench migrate-env python3

Version

ERPNext: v11.1.13 (master)
Frappe Framework: v11.1.13 (master)

Traceback:

Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 61, in application
	response = frappe.handler.handle()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 21, in handle
	data = execute_cmd(cmd)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/handler.py", line 56, in execute_cmd
	return frappe.call(method, **frappe.form_dict)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1026, in call
	return fn(*args, **newargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/desk/form/load.py", line 33, in getdoc
	run_onload(doc)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/desk/form/load.py", line 203, in run_onload
	doc.run_method("onload")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 772, in run_method
	out = Document.hook(fn)(self, *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1048, in composer
	return composed(self, method, *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 1031, in runner
	add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 766, in <lambda>
	fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/selling/doctype/customer/customer.py", line 23, in onload
	self.load_dashboard_info()
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/selling/doctype/customer/customer.py", line 26, in load_dashboard_info
	info = get_dashboard_info(self.doctype, self.name, self.loyalty_program)
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/accounts/party.py", line 468, in get_dashboard_info
	current_fiscal_year = get_fiscal_year(nowdate(), as_dict=True)
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/accounts/utils.py", line 20, in get_fiscal_year
	return get_fiscal_years(date, fiscal_year, label, verbose, company, as_dict=as_dict)[0]
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/accounts/utils.py", line 23, in get_fiscal_years
	fiscal_years = frappe.cache().hget("fiscal_years", company) or []
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 177, in hget
	value = pickle.loads(value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 1: ordinal not in range(128)

Have tried

$ bench update
$ find . -name '*.pyc' -delete

Issue remains, error comes up every time a customer is opened… Does anyone have a solution?

Update a server reboot as removed the issue…