Internal Server Error after server reboot

Install / Update Deployment
1

The server was rebooted, and since then I am getting Internal Server Error on ERPNext login page. I have already confirmed that mysql/MariaDB is running.

~$ bench run-tests
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/username/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/username/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/username/frappe-bench/apps/frappe/frappe/commands.py", line 29, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/username/frappe-bench/apps/frappe/frappe/commands.py", line 790, in run_tests
    force=context.force, profile=profile)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 25, in main
    frappe.clear_cache()
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 425, in clear_cache
    for fn in frappe.get_hooks("clear_cache"):
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 721, in get_hooks
    hooks = _dict(cache().get_value("app_hooks", load_app_hooks))
  File "/home/username/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 68, in get_value
    val = generator()
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 689, in load_app_hooks
    for app in [app_name] if app_name else get_installed_apps(sort=True):
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 669, in get_installed_apps
    installed = json.loads(db.get_global("installed_apps") or "[]")
  File "/home/username/frappe-bench/apps/frappe/frappe/database.py", line 642, in get_global
    return self.get_default(key, user)
  File "/home/username/frappe-bench/apps/frappe/frappe/database.py", line 654, in get_default
    d = self.get_defaults(key, parent)
  File "/home/username/frappe-bench/apps/frappe/frappe/database.py", line 660, in get_defaults
    defaults = frappe.defaults.get_defaults(parent)
  File "/home/username/frappe-bench/apps/frappe/frappe/defaults.py", line 72, in get_defaults
    globald = get_defaults_for()
  File "/home/username/frappe-bench/apps/frappe/frappe/defaults.py", line 179, in get_defaults_for
    where parent = %s order by creation""", (parent,), as_dict=1)
  File "/home/username/frappe-bench/apps/frappe/frappe/database.py", line 106, in sql
    self.connect()
  File "/home/username/frappe-bench/apps/frappe/frappe/database.py", line 52, in connect
    use_unicode=True, charset='utf8')
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2002, 'Can\'t connect to local MySQL server through socket \'/var/run/mysqld/mysqld.sock\' (2 "No such file or directory")')

also:

~$ bench start
Traceback (most recent call last):
  File "/usr/local/bin/honcho", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/honcho/command.py", line 266, in main
    COMMANDS[args.command](args)
  File "/usr/local/lib/python2.7/dist-packages/honcho/command.py", line 202, in command_start
    manager = Manager()
  File "/usr/local/lib/python2.7/dist-packages/honcho/manager.py", line 42, in __init__
    self.events = multiprocessing.Queue()
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue
    return Queue(maxsize)
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__
    self._rlock = Lock()
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 30] Read-only file system
2

I think this indicates a corruption or failure of the filesystem due to a VPS problem. If I backup /home/username/* to another machine, will I have everything I need to restore ERPNext to a new server?

3

FYI in case this happens to you, the problem was that after reboot, the drive mounted as read-only. I did:

sudo touch /forcefsck

followed by a reboot. That solved the read-only problem, but it still didn’t get ERPNext working. So I did:

bench update --patch (but no luck, so then)
bench update

And after a couple of refreshes in the web browser, I was OK again.

4

This is happening to me again. I am getting Internal Server Error and I can’t get ERPNext to start at all.

This time sudo touch /forcefsck did not work.

server:~$ bench start
Traceback (most recent call last):
  File "/usr/local/bin/honcho", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/honcho/command.py", line 266, in main
    COMMANDS[args.command](args)
  File "/usr/local/lib/python2.7/dist-packages/honcho/command.py", line 202, in command_start
    manager = Manager()
  File "/usr/local/lib/python2.7/dist-packages/honcho/manager.py", line 42, in __init__
    self.events = multiprocessing.Queue()
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue
    return Queue(maxsize)
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__
    self._rlock = Lock()
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 30] Read-only file system
server:~$

server:~$ bench update --build
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 9, in <module>
    load_entry_point('bench==0.92', 'console_scripts', 'bench')()
  File "/home/username/bench-repo/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/home/username/bench-repo/bench/commands/update.py", line 31, in update
    version_upgrade = is_version_upgrade()
  File "/home/username/bench-repo/bench/app.py", line 102, in is_version_upgrade
    upstream_version = get_upstream_version('frappe', bench=bench, branch=branch)
  File "/home/username/bench-repo/bench/app.py", line 150, in get_upstream_version
    return get_version_from_string(contents)
  File "/home/username/bench-repo/bench/app.py", line 206, in get_version_from_string
    return match.group(2)
AttributeError: 'NoneType' object has no attribute 'group'
server:~$

server:~$ bench run-tests
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/username/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/username/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/home/username/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/username/frappe-bench/apps/frappe/frappe/commands.py", line 29, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/username/frappe-bench/apps/frappe/frappe/commands.py", line 790, in run_tests
    force=context.force, profile=profile)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 39, in main
    ret = run_all_tests(app, verbose, profile)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 74, in run_all_tests
    _add_test(path, filename, verbose, test_suite=test_suite)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 156, in _add_test
    make_test_records(doctype, verbose)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 172, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 172, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 172, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 173, in make_test_records
    make_test_records_for_doctype(options, verbose, force)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 218, in make_test_records_for_doctype
    frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
  File "/home/username/frappe-bench/apps/frappe/frappe/test_runner.py", line 265, in make_test_objects
    d.insert()
  File "/home/username/frappe-bench/apps/frappe/frappe/model/document.py", line 194, in insert
    self._validate()
  File "/home/username/frappe-bench/apps/frappe/frappe/model/document.py", line 342, in _validate
    self._validate_links()
  File "/home/username/frappe-bench/apps/frappe/frappe/model/document.py", line 548, in _validate_links
    frappe.LinkValidationError)
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 264, in throw
    msgprint(msg, raise_exception=exc)
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 257, in msgprint
    _raise_exception()
  File "/home/username/frappe-bench/apps/frappe/frappe/__init__.py", line 242, in _raise_exception
    raise raise_exception, encode(msg)
frappe.exceptions.LinkValidationError: Could not find Parent Account: Stock Assets - _TC
server:~$
5

Your file permissions are messed up beyond imagination.
Make sure to share full details, when posting, such as OS, Build, Version, Bench version, ERPNext Version etc.
+
_Post Formating Help: https://meta.discourse.org/t/post-format-reference-documentation/19197/2_

Yes, Backup and Do Fresh install on Debian 7/8 Server with minimal image / uninstall web server, mysql server, etc. from server before you do bench installer tool for ERPNext.

6

Beyond imagination? Wow @yashodhan, that sounds bad.

I can’t find a link on how to take a manual backup hat includes both data and files… (This is a live production install with business implications.)

7

Why are you running bench start and bench run-tests? any custom module or customization also part of this erpnext instance you are experiencing issues with?

AND

Hire an Expert if you do not have idea and does not wish to have blunder. from https://community.erpnext.com/service-providers

8

I’m using those because I have seen them in related support requests. It is not clear to me what to do. Where can I find the documentation?

9

Based on the quotations I have seen from some Experts here, hiring them is cost prohibitive. One of the appealing aspects of open-source software is community support.

10

Seriously? You should NEVER go on running commands without reading and understanding its impact!
Though commands your ran does not cause any harm in general, but you should always make sure before throwing commands on CLI.

If your ERPNext installed in Production mode, then you dont run bench start because it used only for Development environment to kick start manually (only when ERPNext/Frappe is installed in Development mode).

11

As far as I can tell, the User Manual is only for end users, and the Documentation explains only how to install. So I am not finding anything documented about keeping ERPNext up and running or recovering from crashes.

This is my primary challenge in using ERPNext at the moment.

12

Sorry for confusion, but User Manual for ERPNext is talks about all lengths and depths of ERPNext.
Where as Developer Manual for Frappe (Framework in which ERPNext is built) does talks about Development side of it.

ERPNext is built with easy deployable option for users who do have basic understanding of OS and commands.
bench is official tool which does take care of installation, updates, patches, backups etc. and more.

For installation Instructions: Take a look at GitHub - frappe/bench: CLI to manage Multi-tenant deployments for Frappe apps
Keep this handy: https://frappe.github.io/frappe/user/en/bench/resources/bench-commands-cheatsheet.html

13

Thanks for the links

14

To be clear, bench run-tests can have negative effects on your database, as it will add a number of test entries to your database and might delete some entries. It is not meant to be run on a production server.

1 Like