Doc Save error with possibly related database User error after update

Hi all,
After updating the latest changes on master branch (production env) everything seemed to work great. Someone then mentioned not being able to save changes to a document.
I found it first on saving a BOM, but then on more docs and also the file uploads.

The console error reckons it’s an invalid json character “<”, which means probably the json string contains an error message, but I can’t see what it is. Screenshot of chrome:

I found a few posts that seemed similar suggesting to bench update, bench clear-website-cache and the like. However, as usual my firt stop was running bench backup - or trying to. At this point a weird error was shown. Any bench command I tried of the above resulted in the script failing with a database error about being unable to connect my frappe user (frappie@localhost) using NO password. Where did it get this from? In my site config files the database and user is correctly configured from what I can see.

Here is full error from console:
[frappie@erp erp]$ bench clear-website-cache
Traceback (most recent call last):
File “/usr/lib64/python2.7/runpy.py”, line 162, in _run_module_as_main
main”, fname, loader, pkg_name)
File “/usr/lib64/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappie/erp/apps/frappe/frappe/utils/bench_helper.py”, line 97, in
main()
File “/home/frappie/erp/apps/frappe/frappe/utils/bench_helper.py”, line 18, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/home/frappie/erp/env/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappie/erp/apps/frappe/frappe/commands/init.py”, line 25, in _func
ret = f(frappe._dict(ctx.obj), *args, **kwargs)
File “/home/frappie/erp/apps/frappe/frappe/commands/utils.py”, line 63, in clear_website_cache
frappe.website.render.clear_cache()
File “/home/frappie/erp/apps/frappe/frappe/website/render.py”, line 283, in clear_cache
frappe.clear_cache(“Guest”)
File “/home/frappie/erp/apps/frappe/frappe/init.py”, line 561, in clear_cache
frappe.cache_manager.clear_user_cache(user)
File “/home/frappie/erp/apps/frappe/frappe/cache_manager.py”, line 22, in clear_user_cache
clear_notifications(user)
File “/home/frappie/erp/apps/frappe/frappe/desk/notifications.py”, line 173, in clear_notifications
config = get_notification_config()
File “/home/frappie/erp/apps/frappe/frappe/desk/notifications.py”, line 236, in get_notification_config
return frappe.cache().get_value(“notification_config”, _get)
File “/home/frappie/erp/apps/frappe/frappe/utils/redis_wrapper.py”, line 79, in get_value
val = generator()
File “/home/frappie/erp/apps/frappe/frappe/desk/notifications.py”, line 227, in _get
hooks = frappe.get_hooks()
File “/home/frappie/erp/apps/frappe/frappe/init.py”, line 940, in get_hooks
hooks = _dict(cache().get_value(“app_hooks”, load_app_hooks))
File “/home/frappie/erp/apps/frappe/frappe/utils/redis_wrapper.py”, line 79, in get_value
val = generator()
File “/home/frappie/erp/apps/frappe/frappe/init.py”, line 914, in load_app_hooks
for app in [app_name] if app_name else get_installed_apps(sort=True):
File “/home/frappie/erp/apps/frappe/frappe/init.py”, line 878, in get_installed_apps
installed = json.loads(db.get_global(“installed_apps”) or “[]”)
File “/home/frappie/erp/apps/frappe/frappe/database.py”, line 783, in get_global
return self.get_default(key, user)
File “/home/frappie/erp/apps/frappe/frappe/database.py”, line 795, in get_default
d = self.get_defaults(key, parent)
File “/home/frappie/erp/apps/frappe/frappe/database.py”, line 801, in get_defaults
defaults = frappe.defaults.get_defaults(parent)
File “/home/frappie/erp/apps/frappe/frappe/defaults.py”, line 77, in get_defaults
globald = get_defaults_for()
File “/home/frappie/erp/apps/frappe/frappe/defaults.py”, line 204, in get_defaults_for
where parent = %s order by creation"“”, (parent,), as_dict=1)
File “/home/frappie/erp/apps/frappe/frappe/database.py”, line 165, in sql
self.connect()
File “/home/frappie/erp/apps/frappe/frappe/database.py”, line 115, in connect
charset=‘utf8mb4’, use_unicode = True, conv = conversions, local_infile = self.local_infile)
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/init.py”, line 94, in Connect
return Connection(*args, **kwargs)
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/connections.py”, line 325, in init
self.connect()
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/connections.py”, line 599, in connect
self._request_authentication()
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/connections.py”, line 861, in _request_authentication
auth_packet = self._read_packet()
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/connections.py”, line 684, in _read_packet
packet.check_error()
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/protocol.py”, line 220, in check_error
err.raise_mysql_exception(self._data)
File “/home/frappie/erp/env/lib/python2.7/site-packages/pymysql/err.py”, line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, u"Access denied for user ‘frappie’@‘localhost’ (using password: NO)")

I thought at first I would just add that user to the database, but then, it’s not using a password, so that would not be safe.

Any ideas how this situation happened, and what to do about it?
Thank you kindly,
Ryan

Oops, forgot to add this:

image

Also - running on CentoOS7 4GB RAM, 3 processors - been running seemingly very well so far.

Hi, I have a similar issue when saving a doctype, did you find a solution?