Unable to solve Internal Server Error in erpnext

[root13F]

@avaiskhatri

journalctl -xe

run this and check if you can find any errors and post them here or google search them

[avaiskhatri]

– The result is failed.
Aug 09 16:27:08 ak systemd[1]: mariadb.service: Unit entered failed state.
Aug 09 16:27:08 ak systemd[1]: mariadb.service: Failed with result ‘exit-code’.
Aug 09 16:27:08 ak sudo[6147]: pam_unix(sudo:session): session closed for user root
Aug 09 16:27:58 ak kernel: usb 1-1.8: reset high-speed USB device number 5 using ehci-pci
Aug 09 16:28:00 ak kernel: usb 1-1.8: USB disconnect, device number 5
Aug 09 16:28:00 ak kernel: usb 1-1.8: new high-speed USB device number 6 using ehci-pci
Aug 09 16:28:01 ak kernel: usb 1-1.8: New USB device found, idVendor=064e, idProduct=9202
Aug 09 16:28:01 ak kernel: usb 1-1.8: New USB device strings: Mfr=3, Product=1, SerialNumber=2
Aug 09 16:28:01 ak kernel: usb 1-1.8: Product: Integrated_Webcam_HD
Aug 09 16:28:01 ak kernel: usb 1-1.8: Manufacturer: SuYin
Aug 09 16:28:01 ak kernel: usb 1-1.8: SerialNumber: HF1316-T840-SS01-REV4122
Aug 09 16:28:01 ak kernel: uvcvideo: Found UVC 1.00 device Integrated_Webcam_HD (064e:9202)
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Extension 4 was not initialized!
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Extension 7 was not initialized!
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Processing 2 was not initialized!
Aug 09 16:28:01 ak kernel: uvcvideo 1-1.8:1.0: Entity type for entity Camera 1 was not initialized!
Aug 09 16:28:01 ak kernel: input: Integrated_Webcam_HD: Integrate as /devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.8/1-1.8:1.0/input/input17
Aug 09 16:28:01 ak mtp-probe[6558]: checking bus 1, device 6: “/sys/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.8”
Aug 09 16:28:01 ak mtp-probe[6558]: bus: 1, device: 6 was not an MTP device
Aug 09 16:28:36 ak kernel: usb 1-1.8: reset high-speed USB device number 6 using ehci-pci

[root13F]

okay these are not relevant to our case we’ll filter results via grep then

journalctl -xe | grep 'mysql'

[avaiskhatri]

Aug 09 21:03:07 ak mysqld[1250]: 2018-08-09 21:03:07 140645876381888 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 1250 …
Aug 09 21:03:07 ak mysqld[1250]: 2018-08-09 21:03:07 140645876381888 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 21:03:07 ak mysqld[1250]: 2018-08-09 21:03:07 140645876381888 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:08:56 ak polkitd(authority=local)[956]: Operator of unix-session:c2 successfully authenticated as unix-user:ak to gain TEMPORARY authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.97 [/bin/systemctl --no-pager stop mysql.service] (owned by unix-user:ak)
Aug 09 16:09:13 ak polkitd(authority=local)[956]: Operator of unix-session:c2 successfully authenticated as unix-user:ak to gain TEMPORARY authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.121 [systemctl stop mysql.service] (owned by unix-user:ak)
Aug 09 16:09:47 ak sudo[4192]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:09:47 ak mysqld[4499]: 2018-08-09 16:09:47 140250964551872 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 4499 …
Aug 09 16:09:47 ak mysqld[4499]: 2018-08-09 16:09:47 140250964551872 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:09:47 ak mysqld[4499]: 2018-08-09 16:09:47 140250964551872 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:10:12 ak sudo[4627]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysql mysqld_safe mysqld
Aug 09 16:12:46 ak sudo[4658]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:12:46 ak mysqld[4964]: 2018-08-09 16:12:46 140329149847744 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 4964 …
Aug 09 16:12:46 ak mysqld[4964]: 2018-08-09 16:12:46 140329149847744 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:12:46 ak mysqld[4964]: 2018-08-09 16:12:46 140329149847744 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:13:06 ak sudo[5050]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/bin/systemctl restart mysql
Aug 09 16:13:06 ak mysqld[5343]: 2018-08-09 16:13:06 139815829424320 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 5343 …
Aug 09 16:13:06 ak mysqld[5343]: 2018-08-09 16:13:06 139815829424320 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:13:06 ak mysqld[5343]: 2018-08-09 16:13:06 139815829424320 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:13:52 ak sudo[5401]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysql mysqld_safe mysqld
Aug 09 16:14:04 ak sudo[5413]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysqld_safe mysqld
Aug 09 16:14:37 ak sudo[5429]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/killall -KILL mysql mysqld_safe mysqld
Aug 09 16:26:23 ak sudo[5689]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/usr/bin/mysqladmin -u root -p shutdown
Aug 09 16:26:56 ak sudo[5691]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:26:56 ak mysqld[5998]: 2018-08-09 16:26:56 140642653874368 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 5998 …
Aug 09 16:26:56 ak mysqld[5998]: 2018-08-09 16:26:56 140642653874368 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:26:56 ak mysqld[5998]: 2018-08-09 16:26:56 140642653874368 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)
Aug 09 16:27:04 ak sudo[6147]: ak : TTY=pts/2 ; PWD=/home/ak ; USER=root ; COMMAND=/etc/init.d/mysql restart
Aug 09 16:27:05 ak mysqld[6453]: 2018-08-09 16:27:05 140315380984000 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log) starting as process 6453 …
Aug 09 16:27:05 ak mysqld[6453]: 2018-08-09 16:27:05 140315380984000 [Warning] Could not increase number of max_open_files to more than 16364 (request: 21011)
Aug 09 16:27:05 ak mysqld[6453]: 2018-08-09 16:27:05 140315380984000 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (was 500) table_cache: 7917 (was 10240)

[root13F]

also what response do you get on sudo systemctl restart mariadb.service and then bench start

[avaiskhatri]

@root13F Really appreciate your input and efforts.

sudo systemctl restart mariadb.service

is giving error:

Job for mariadb.service failed because the control process exited with error code. See “systemctl status mariadb.service” and “journalctl -xe” for details.

[root13F]

have you tried reinstalling mariadb-server ?

[avaiskhatri]

I just restarted the server and this error pops out.

Do you suggest to reinstall in this case?

[avaiskhatri]

@root13F

Also will reinstalling mariadb-server result in delete of data ?

[root13F]

probably let us not take that risk. I shouldn’t have suggested that.

[avaiskhatri]

Then what are suggestions to resolve this issue?

[root13F]

one thing what is this machine’s configuration ?

[avaiskhatri]

Let me attach htop picture.!

[root13F]

okay one thread it was suggested that you restart the server i.e. the machine

[avaiskhatri]

Already did couple of times.

Please note that I have installed this on my personal laptop with dual windows.

It was working fine last time I logged in.

[root13F]

Yes this is going in circles. Let me see if I can find something

[avaiskhatri]

Thanks brother.

Please post any solution here if you find any thing relevant.

[avaiskhatri]

@root13F, Just fyi, when I type this command:
sudo systemctl status mariadb.service

its popping out the following error.

Aug 09 17:11:29 ak systemd[1]: Starting MariaDB 10.2.16 database server…
Aug 09 17:11:30 ak mysqld[11324]: 2018-08-09 17:11:30 139697069971648 [Note] /usr/sbin/mysqld (mysqld 10.2.16-MariaDB-10.2.16+maria~xenial-log)
Aug 09 17:11:30 ak mysqld[11324]: 2018-08-09 17:11:30 139697069971648 [Warning] Could not increase number of max_open_files to more than 16364
Aug 09 17:11:30 ak mysqld[11324]: 2018-08-09 17:11:30 139697069971648 [Warning] Changed limits: max_open_files: 16364 max_connections: 500 (wa
Aug 09 17:11:33 ak systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 09 17:11:33 ak systemd[1]: Failed to start MariaDB 10.2.16 database server.
Aug 09 17:11:33 ak systemd[1]: mariadb.service: Unit entered failed state.
Aug 09 17:11:33 ak systemd[1]: mariadb.service: Failed with result ‘exit-code’.

[avaiskhatri]

I found the solution and would like to share it the community.

First I have removed the warning of max_open_files
by editing file using this command

sudo nano /etc/systemd/system/mysqld.service

Changed LimitNOFILE to 40000 from 16364

#LimitNOFILE=16364
LimitNOFILE=40000

Then I googled and found the same problem on this thread

[SOLVED] Mysql error after restart

Hello, I’ve SOLVED the issue by myself, I’d like to share the solution with the community. This post helped me to SOLVE the issue: I’ve found the following error in /var/lib/mysql/mysql-error.log 2018-04-16 16:22:15 140293027571904 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to comm…

Repeated the same procedure
checked mysql-error.log file and the error was same.

Then entered 2 commands and my app was up and functional

mysqld --tc-heuristic-recover=ROLLBACK
service mysql start