Unable to solve Internal Server Error in erpnext

After server restart, I am getting internal server error.

Checked web.error.log in /home/frappe/frappe-bench/logs/

its shows the following error.

OperationalError: (2003, "Can't connect to MySQL server on u'localhost' 

Traceback (most recent call last):
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/gunicor$
    self.handle_request(listener, req, client, addr)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/gunicor$
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeu$
    return ClosingIterator(app(environ, start_response), self.cleanup)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/werkzeu$
    resp = f(*args[:-2] + (request,))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 88, in appli$
    response = handle_exception(e)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 191, in hand$
    http_status_code=http_status_code)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 2$
    path = resolve_path(path.strip('/ '))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 2$
    path = resolve_from_map(path)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 2$
    for r in get_website_rules()])
File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 2$
    return frappe.cache().get_value('website_route_rules', _get)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", l$
    val = generator()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 2$
    rules = frappe.get_hooks("website_route_rules")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 832, in$
    hooks = _dict(cache().get_value("app_hooks", load_app_hooks))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", l$
    val = generator()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 810, in$
    for app in [app_name] if app_name else get_installed_apps(sort=True):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 774, in$
    installed = json.loads(db.get_global("installed_apps") or "[]")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 723, in$
    return self.get_default(key, user)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 735, in$
    d = self.get_defaults(key, parent)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 741, in$
    defaults = frappe.defaults.get_defaults(parent)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/defaults.py", line 57, in $
    globald = get_defaults_for()
File "/home/frappe/frappe-bench/apps/frappe/frappe/defaults.py", line 177, in$
    where parent = %s order by creation""", (parent,), as_dict=1)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 136, in$
Internal Server Error    self.connect()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 86, in $
    charset='utf8mb4', use_unicode = True, conv = conversions)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql$
    return Connection(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql$
    self.connect()
  File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/pymysql$
    raise exc
1 Like

Is mysql server running ? Sounds like it isnt.

2 Likes

I have checked the status of MySQL using:

sudo service mysql status

and its return this error:

mariadb.service - MariaDB 10.2.16 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Tue 2018-07-31 09:53:03 PKT; 5min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 2017 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
  Process: 1722 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && system
  Process: 1717 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 1705 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 2017 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

Jul 31 09:53:00 ak systemd[1]: Starting MariaDB 10.2.16 database server...
Jul 31 09:53:00 ak mysqld[2017]: 2018-07-31  9:53:00 139750044420288 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xen
Jul 31 09:53:00 ak mysqld[2017]: 2018-07-31  9:53:00 139750044420288 [Warning] Could not increase number of max_open_files to more tha
Jul 31 09:53:00 ak mysqld[2017]: 2018-07-31  9:53:00 139750044420288 [Warning] Changed limits: max_open_files: 16364  max_connections:
Jul 31 09:53:03 ak systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 09:53:03 ak systemd[1]: Failed to start MariaDB 10.2.16 database server.
Jul 31 09:53:03 ak systemd[1]: mariadb.service: Unit entered failed state.
Jul 31 09:53:03 ak systemd[1]: mariadb.service: Failed with result 'exit-code'.

try sudo /etc/init.d/mysql restart

just another way of sudo systemctl restart mysql

@root13F

sudo /etc/init.d/mysql restart

is resulting in an error:

mysql.serviceJob for mariadb.service failed because the control process exited with error code. See “systemctl status mariadb.service” and “journalctl -xe” for details.
failed!

Have you recently made any changes to the my.cnf ?

@root13F

Nope. Just restarted the server and facing this problem.

Everything was working smooth until I restart the server.

kill mysqld and mysqld_safe and then restart server and try again restarting mysql via the above command

1 Like

@root13F,

killall -KILL mysql mysqld_safe mysqld

its says not process found.

mysql: no process found
mysqld_safe: no process found
mysqld: no process found

Still the problem is there.

try this

mysqladmin -u root -p shutdown

@root13F

mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

sudo /etc/init.d/mysql restart

@root13F

Restarting mysql (via systemctl): mysql.serviceJob for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

run this and check if you can find any errors and post them here or google search them

– The result is failed.
Aug 09 16:27:08 ak systemd[1]: mariadb.service: Unit entered failed state.
Aug 09 16:27:08 ak systemd[1]: mariadb.service: Failed with result ‘exit-code’.
Aug 09 16:27:08 ak sudo[6147]: pam_unix(sudo:session): session closed for user root
Aug 09 16:27:58 ak kernel: usb 1-1.8: reset high-speed USB device number 5 using ehci-pci
Aug 09 16:28:00 ak kernel: usb 1-1.8: USB disconnect, device number 5
Aug 09 16:28:00 ak kernel: usb 1-1.8: new high-speed USB device number 6 using ehci-pci
Aug 09 16:28:01 ak kernel: usb 1-1.8: New USB device found, idVendor=064e, idProduct=9202
Aug 09 16:28:01 ak kernel: usb 1-1.8: New USB device strings: Mfr=3, Product=1, SerialNumber=2
Aug 09 16:28:01 ak kernel: usb 1-1.8: Product: Integrated_Webcam_HD
Aug 09 16:28:01 ak kernel: usb 1-1.8: Manufacturer: SuYin
Aug 09 16:28:01 ak kernel: usb 1-1.8: SerialNumber: HF1316-T840-SS01-REV4122
Aug 09 16:28:01 ak kernel: uvcvideo: Found UVC 1.00 device Integrated_Webcam_HD (064e:9202)
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Extension 4 was not initialized!
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Extension 7 was not initialized!
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Processing 2 was not initialized!
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Camera 1 was not initialized!
Aug 09 16:28:01 ak kernel: input: Integrated_Webcam_HD: Integrate as /devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.8/1-1.8:1.0/input/input17
Aug 09 16:28:01 ak mtp-probe[6558]: checking bus 1, device 6: “/sys/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.8”
Aug 09 16:28:01 ak mtp-probe[6558]: bus: 1, device: 6 was not an MTP device
Aug 09 16:28:36 ak kernel: usb 1-1.8: reset high-speed USB device number 6 using ehci-pci

okay these are not relevant to our case we’ll filter results via grep then

journalctl -xe | grep 'mysql'

Aug 09 21:03:07 ak mysqld[1250]: 2018-08-09 21:03:07 140645876381888 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 1250 …
Aug 09 21:03:07 ak mysqld[1250]: 2018-08-09 21:03:07 140645876381888 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 21:03:07 ak mysqld[1250]: 2018-08-09 21:03:07 140645876381888 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:08:56 ak polkitd(authority=local)[956]: Operator of unix-session:c2 successfully authenticated as unix-user:ak to gain TEMPORARY authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.97 [/bin/systemctl --no-pager stop mysql.service] (owned by unix-user:ak)
Aug 09 16:09:13 ak polkitd(authority=local)[956]: Operator of unix-session:c2 successfully authenticated as unix-user:ak to gain TEMPORARY authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.121 [systemctl stop mysql.service] (owned by unix-user:ak)
Aug 09 16:09:47 ak sudo[4192]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:09:47 ak mysqld[4499]: 2018-08-09 16:09:47 140250964551872 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 4499 …
Aug 09 16:09:47 ak mysqld[4499]: 2018-08-09 16:09:47 140250964551872 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:09:47 ak mysqld[4499]: 2018-08-09 16:09:47 140250964551872 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:10:12 ak sudo[4627]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysql mysqld_safe mysqld
Aug 09 16:12:46 ak sudo[4658]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:12:46 ak mysqld[4964]: 2018-08-09 16:12:46 140329149847744 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 4964 …
Aug 09 16:12:46 ak mysqld[4964]: 2018-08-09 16:12:46 140329149847744 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:12:46 ak mysqld[4964]: 2018-08-09 16:12:46 140329149847744 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:13:06 ak sudo[5050]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/bin/systemctl restart mysql
Aug 09 16:13:06 ak mysqld[5343]: 2018-08-09 16:13:06 139815829424320 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 5343 …
Aug 09 16:13:06 ak mysqld[5343]: 2018-08-09 16:13:06 139815829424320 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:13:06 ak mysqld[5343]: 2018-08-09 16:13:06 139815829424320 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:13:52 ak sudo[5401]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysql mysqld_safe mysqld
Aug 09 16:14:04 ak sudo[5413]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysqld_safe mysqld
Aug 09 16:14:37 ak sudo[5429]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysql mysqld_safe mysqld
Aug 09 16:26:23 ak sudo[5689]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/mysqladmin -u root -p shutdown
Aug 09 16:26:56 ak sudo[5691]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:26:56 ak mysqld[5998]: 2018-08-09 16:26:56 140642653874368 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 5998 …
Aug 09 16:26:56 ak mysqld[5998]: 2018-08-09 16:26:56 140642653874368 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:26:56 ak mysqld[5998]: 2018-08-09 16:26:56 140642653874368 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:27:04 ak sudo[6147]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:27:05 ak mysqld[6453]: 2018-08-09 16:27:05 140315380984000 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 6453 …
Aug 09 16:27:05 ak mysqld[6453]: 2018-08-09 16:27:05 140315380984000 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:27:05 ak mysqld[6453]: 2018-08-09 16:27:05 140315380984000 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)

also what response do you get on sudo systemctl restart mariadb.service and then bench start

1 Like

@root13F Really appreciate your input and efforts.

sudo systemctl restart mariadb.service

is giving error:

Job for mariadb.service failed because the control process exited with error code. See “systemctl status mariadb.service” and “journalctl -xe” for details.

have you tried reinstalling mariadb-server ?