Database Issues: mysqld: Table ‘tabError@0020Log’ is marked as crashed and should be repaired

ERPNext
,
1

Hi,

A AWS hosted instance of ERPNext with InnoDB Engine and not configured for repairs keeps hitting very high CPU utilization frequently (like once a day). Today I looked at the Log Files and the relevant part of the log files is attached after my message.

Any DB experts can indicate what I need to do to fix this issue?

2019-10-21 12:48:29 139851132917952 [Note] Using unique option prefix ‘myisam-recover’ is error-prone and can break in the future. Please use the full name ‘myisam-recover-options’ instead.
2019-10-21 12:48:29 139851132917952 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See InnoDB File Format - MariaDB Knowledge Base
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Uses event mutexes
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Using Linux native AIO
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Number of pools: 1
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Using SSE2 crc32 instructions
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Initializing buffer pool, total size = 5G, instances = 8, chunk size = 128M
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Completed initialization of buffer pool
2019-10-21 12:48:29 139844930717440 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Highest supported file format is Barracuda.
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Starting crash recovery from checkpoint LSN=76908969084
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Trx id counter is 8500224
2019-10-21 12:48:29 139851132917952 [Note] InnoDB: Last binlog file ‘/var/lib/mysql/mysql-bin.000914’, position 300795
2019-10-21 12:48:32 139851132917952 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-10-21 12:48:32 139844897146624 [Note] InnoDB: Starting in background the rollback of recovered transactions
2019-10-21 12:48:32 139851132917952 [Note] InnoDB: Removed temporary tablespace data file: “ibtmp1”
2019-10-21 12:48:32 139851132917952 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-10-21 12:48:32 139851132917952 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
2019-10-21 12:48:32 139851132917952 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
2019-10-21 12:48:32 139844897146624 [Note] InnoDB: Rolled back recovered transaction 8499774
2019-10-21 12:48:32 139844897146624 [Note] InnoDB: Rollback of non-prepared transactions completed
2019-10-21 12:48:32 139851132917952 [Note] InnoDB: 5.7.21 started; log sequence number 76908969093
2019-10-21 12:48:32 139845063960320 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-10-21 12:48:32 139851132917952 [Note] Plugin ‘FEEDBACK’ is disabled.
2019-10-21 12:48:32 139851132917952 [Note] Recovering after a crash using /var/lib/mysql/mysql-bin
2019-10-21 12:48:32 139851132917952 [Note] Starting crash recovery…
2019-10-21 12:48:32 139851132917952 [Note] Crash recovery finished.
2019-10-21 12:48:32 139851132917952 [Note] Server socket created on IP: ‘127.0.0.1’.
2019-10-21 12:48:32 139851132917952 [Note] Reading of all Master_info entries succeded
2019-10-21 12:48:32 139851132917952 [Note] Added new Master_info ‘’ to hash table
2019-10-21 12:48:32 139851132917952 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘10.2.14-MariaDB-10.2.14+maria~xenial-log’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution
2019-10-21 12:48:47 139851048158976 [Warning] Aborted connection 21 to db: ‘a81126be1d0839c7’ user: ‘a81126be1d0839c7’ host: ‘localhost’ (Got an error reading communication packets)
2019-10-21 12:49:21 139845055567616 [ERROR] mysqld: Table ‘./a81126be1d0839c7/tabError@0020Log’ is marked as crashed and should be repaired
2019-10-21 12:49:21 139845055567616 [ERROR] mysqld: Table ‘tabError Log’ is marked as crashed and should be repaired
2019-10-21 12:49:21 139845055567616 [Warning] Checking table: ‘./a81126be1d0839c7/tabError@0020Log’
2019-10-21 12:49:57 139845054961408 [ERROR] mysqld: Table ‘./a81126be1d0839c7/__global_search’ is marked as crashed and should be repaired
2019-10-21 12:49:57 139845054961408 [ERROR] mysqld: Table ‘__global_search’ is marked as crashed and should be repaired
2019-10-21 12:49:57 139845054961408 [Warning] Checking table: ‘./a81126be1d0839c7/__global_search’
2019-10-21 12:50:20 139845063960320 [Note] InnoDB: Buffer pool(s) load completed at 191021 12:50:20

Thanks

Jay

2

Is it possible to show/share your mysql conf file?

I would try adding in something like this…

# the default here is 4MB?? in more recent versions, which sometimes allows for disconnects in the middle of transactions
max_allowed_packet=64M

PS: I am NOT an DB expert - just giving it a shot

3

Thanks for your message. It was 16M. Changed it to 64M. Let’s see if you need to continue to be modest about your database skills. :slight_smile:

Thanks

Jay

4

Hi @JayRam, I know this is from two years ago :slight_smile: but do you recall if this change fixed your issue? I’m facing the same issue in one of my instances.

5

Hi Chude,

I cannot remember. I think I tried a bunch of things including the one in this thread and that seemed to have solved the issue for a while. But that server is having performance issues. But please go ahead and try this. It cannot hurt! :slight_smile:

Thanks

Jay

6

@Chude_Osiegbu We are facing the same issue in ERPNext v12.

I know your query was 2 years ago. Is the issue solved? if so can you help us how to fix it?

Thanks,
Saravana

7

if it is a tabError problem - check that the table size is not too big and try deleting items in that table which are particularly old