Internal Server Error - SQL not starting up

assamsiddibapa · October 3, 2018, 8:32pm

Hi,

I am not sure what happened but today i the erp started throwing “Internal Server Error”

After going through the posts on the forum and through google, i realized its the sql server not starting up.

root@erp:~# service mysql restart

Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core. See "systemctl status mariadb.service" and "journalctl -xe" for details.

root@erp:~# systemctl status mariadb.service

●     mariadb.service - MariaDB 10.2.13 database server
       Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/mariadb.service.d
               └─migrated-from-my.cnf-settings.conf
       Active: activating (auto-restart) (Result: core-dump) since Wed 2018-10-03 20:34:26 UTC; 3s ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 22488 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=dumped, signal=ABRT)
      Process: 22201 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
      Process: 22196 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 22192 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
     Main PID: 22488 (code=dumped, signal=ABRT)

Oct 03 20:34:26 erp systemd[1]: Failed to start MariaDB 10.2.13 database server.
Oct 03 20:34:26 erp systemd[1]: mariadb.service: Unit entered failed state.
Oct 03 20:34:26 erp systemd[1]: mariadb.service: Failed with result 'core-dump'.

root@erp:~# journalctl -xe

--
-- The result is failed.
Oct 03 20:35:44 erp systemd[1]: mariadb.service: Unit entered failed state.
Oct 03 20:35:44 erp systemd[1]: mariadb.service: Failed with result 'core-dump'.
Oct 03 20:35:49 erp systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Oct 03 20:35:49 erp systemd[1]: Stopped MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has finished shutting down.
Oct 03 20:35:49 erp systemd[1]: Starting MariaDB 10.2.13 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has begun starting up.
Oct 03 20:35:50 erp mysqld[26887]: 2018-10-03 20:35:50 140498210785472 [Note] /usr/sbin/mysqld (mysqld 10.2.13-MariaDB-10.2.13+maria~xenial-log) starting as process 26887 ...
Oct 03 20:35:50 erp systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
Oct 03 20:35:50 erp systemd[1]: Failed to start MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Oct 03 20:35:50 erp systemd[1]: mariadb.service: Unit entered failed state.
Oct 03 20:35:50 erp systemd[1]: mariadb.service: Failed with result 'core-dump'.
Oct 03 20:35:56 erp systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Oct 03 20:35:56 erp systemd[1]: Stopped MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has finished shutting down.
Oct 03 20:35:56 erp systemd[1]: Starting MariaDB 10.2.13 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has begun starting up.
Oct 03 20:35:56 erp mysqld[27232]: 2018-10-03 20:35:56 139783554398400 [Note] /usr/sbin/mysqld (mysqld 10.2.13-MariaDB-10.2.13+maria~xenial-log) starting as process 27232 ...
Oct 03 20:35:57 erp systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
Oct 03 20:35:57 erp systemd[1]: Failed to start MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Oct 03 20:35:57 erp systemd[1]: mariadb.service: Unit entered failed state.
Oct 03 20:35:57 erp systemd[1]: mariadb.service: Failed with result 'core-dump'.
Oct 03 20:36:02 erp systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Oct 03 20:36:02 erp systemd[1]: Stopped MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has finished shutting down.
Oct 03 20:36:02 erp systemd[1]: Starting MariaDB 10.2.13 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has begun starting up.
Oct 03 20:36:03 erp mysqld[27573]: 2018-10-03 20:36:03 139639055907008 [Note] /usr/sbin/mysqld (mysqld 10.2.13-MariaDB-10.2.13+maria~xenial-log) starting as process 27573 ...
Oct 03 20:36:03 erp systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
Oct 03 20:36:03 erp systemd[1]: Failed to start MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Oct 03 20:36:03 erp systemd[1]: mariadb.service: Unit entered failed state.
Oct 03 20:36:03 erp systemd[1]: mariadb.service: Failed with result 'core-dump'.
Oct 03 20:36:09 erp systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Oct 03 20:36:09 erp systemd[1]: Stopped MariaDB 10.2.13 database server.
-- Subject: Unit mariadb.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has finished shutting down.
Oct 03 20:36:09 erp systemd[1]: Starting MariaDB 10.2.13 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has begun starting up.
lines 3029-3120/3120 (END)

Appreciate some help .

Thank you

root13F · October 4, 2018, 4:19am

see if systemctl restart mysqld works for you

lokesh · October 4, 2018, 5:04am

Try this command
mysqld --tc-heuristic-recover=ROLLBACK

and then
service mysql start

assamsiddibapa · October 4, 2018, 5:41am

Thanks for your response .

@root13F
root@erp:~# systemctl restart mysqld

Job for mariadb.service failed because a fatal signal was delivered causing the                                                                                                                                                                                                                                              control process to dump core. See "systemctl status mariadb.service" and "journa                                                                                                                                                                                                                                             lctl -xe" for details.

@lokesh

root@erp:~# mysqld --tc-heuristic-recover=ROLLBACK
2018-10-04  5:44:30 140660998985920 [Note] mysqld (mysqld 10.2.13-MariaDB-10.2.1                                                                                                                                                                                                                                             3+maria~xenial-log) starting as process 4738 ...
Aborted (core dumped)
root@erp:~# service mysql start
Job for mariadb.service failed because a fatal signal was delivered causing the                                                                                                                                                                                                                                              control process to dump core. See "systemctl status mariadb.service" and "journa                                                                                                                                                                                                                                             lctl -xe" for details.

trentmu · October 4, 2018, 7:14am

You might have a corrupt system DB. To test, you could try creating a VM mimic and copy your DB files into it to test if it comes up.
It could also be a damaged mariadb installation, in which case, just reinstall mariadb - make backups before you do anything.
You might also want to check that you have disk space available

lokesh · October 4, 2018, 7:36am

use sudo before mysqld --tc-heuristic-recover=ROLLBACK like this,
sudo mysqld --tc-heuristic-recover=ROLLBACK
and then start it.

assamsiddibapa · October 6, 2018, 4:02pm

Hi ,

thanks for your reply . unfortunately this didn’t work either.

assamsiddibapa · October 6, 2018, 4:07pm

Hi ,

I did try and created a new VM of the same image. It didnt work.

I am unable to export the databases since i cannot connect it so cannot do that either.

Damaged mariadb … May be Could u please guide me on that .
Googling game me suggestion to run this:

apt-get --reinstall install mariadb-server mariadb-client

and i tried on the mirror copy. Sql seems to be running but i still have internal error and

ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (111)

Will be glad if you can comment.

PS: I have enough space available.

trentmu · October 6, 2018, 4:14pm

OK - I can only suggest trying something like this…

shutdown all the mysql services
make a copy of the data dir
create a new server VM with minimal services
- mysql/mariaDB at least
start the new VM
- stop the mysql services on the new VM
- remove its mysql/mariadb files and replace them with your copies of the DB from your live server

This way you will be able to at least test if the data files can be recovered, and make some informed decision about what to try next

Prachande_Nikhil · October 9, 2018, 1:36pm

Hello,

Screenshot%20from%202018-10-09%2019-02-47

As per the above screenshot, I will suggest you to kindly kill all the process id generating while restarting mysql.

Syntax :- sudo kill

After killing the process id stop & start mysql.

Hope this works for you…

lucholj · November 15, 2019, 6:53pm

Hi,I hope you are ok. Did fix the problem? I have the same problem.

jycob · May 17, 2020, 10:17am

this solved

antiquark100 · March 12, 2022, 5:47pm

Sometimes the old log files needs to be removed, especially after upgrade to MariaDB 10.3 -remove the ib_logfile* files in /var/lib/mysql. See Comments - Upgrading from MariaDB 10.1 to MariaDB 10.2 - MariaDB Knowledge Base