Important MariaDB Version 10.3 Action Required before Update

dominik · June 22, 2018, 10:14am

Summary:

MariaDB version 10.3 has been released about half a year ago (January 18th 2018). If you are planning on updating from 10.2 to 10.3 please read this post, otherwise you may have a problem of your MariaDB server not starting. This short post explains on why you have to make small changes in your my.cnf and how to do it to update without downtime or bugfixing.

What happened?

I recently updated and upgraded my MacBook’s Homebrew packages as well as our servers apt packages. With this update MariaDB is updated to the latest stable 10.3.7. The problem was, that after this update my MariaDB server didn’t start up and threw an error mysqld: unknown variable 'innodb-file-format=barracuda’.

After checking the logfiles errors pointed at variables that are set for Bench which have been removed in 10.3. After removing those lines, the server starts without any issues.

What you need to do before MariaDB 10.3 update:

Stop Frappe Bench (bench stop)
Stop MariaDB Server (Linux: sudo service.mysql stop or MacOs: mysql.server stop)
Open you MariaDB config file (e.g. sudo vim /usr/local/etc/my.cnf)
Comment out the 3 innodb variables (innodb-file-format, innodb-file-per-table and innodb-large-prefix) which have been removed in 10.3

[mysqld]
# innodb-file-format=barracuda
# innodb-file-per-table=1
# innodb-large-prefix=1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4

Update your system (Linux: sudo apt-get update && sudo apt-get upgrade or MacOS: brew update && brew upgrade)
Check if MariaDB server is version 10.3+ (mysql -V)
Start MariaDB server (Linux: sudo service.mysql start or MacOs: mysql.server start)
Start and/or update bench (bench start)

More Information:

The Issue that explains the problem and the solution in more detail: MariaDB 10.3 causing mysqld: unknown variable 'innodb-file-format=barracuda’ · Issue #681 · frappe/bench · GitHub
MariaDB documentation listing the deleted variables: https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_file_format

Tropicalrambler · June 25, 2018, 1:27am

Thanks Dominik!

J0K3R · July 17, 2018, 4:04pm

Hi
could you help me with this,
I commented out the mentioned line , but when I run “Bench new-site site.name”

================================================================================
Creation of your site - site.name failed because MariaDB is not properly
configured to use the Barracuda storage engine.
Please add the settings below to MariaDB’s my.cnf, restart MariaDB then
run bench new-site site.name again.

[mysqld]
innodb-file-format=barracuda
innodb-file-per-table=1
innodb-large-prefix=1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4

================================================================================

I did restart mariaDB , after changing “/usr/local/etc/my.cnf”

Macos 10.13.5
MariaDB 10.3.8

clarkej · July 17, 2018, 6:16pm

What does this report - for example

frappe@ubuntu:~/frappe-bench$ sudo systemctl status mariadb
● mariadb.service - MariaDB 10.2.13 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: active (running) since Wed 2018-06-27 03:24:45 UTC; 2 weeks 6 days ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 1335 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1332 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Process: 943 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] && systemctl set-environm
Process: 913 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 894 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1262 (mysqld)
Status: “Taking your SQL requests now…”
CGroup: /system.slice/mariadb.service
└─1262 /usr/sbin/mysqld

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Check this for clues /var/lib/mysql/mysql-error.log

The ERPNext logs may offer clues too

frappe@ubuntu:~/frappe-bench$ find . -name ‘*.log’ | xargs ls -alt
-rw-r–r-- 1 frappe frappe 5472 Jul 17 18:18 ./logs/frappe.log
-rw-rw-r-- 1 frappe frappe 4662 Jul 17 10:00 ./logs/auto_update_log.log
-rw-r–r-- 1 root root 1617 Jul 17 06:00 ./logs/worker.error.log
-rw-r–r-- 1 root root 3144109 Jul 12 12:54 ./logs/web.error.log
-rw-r–r-- 1 root root 46754 Jul 12 12:54 ./logs/node-socketio.log
-rw-r–r-- 1 frappe frappe 2733 Jul 12 12:54 ./logs/bench.log
-rw-r–r-- 1 root root 99357 Jun 27 03:24 ./logs/redis-socketio.log
-rw-r–r-- 1 root root 101453 Jun 27 03:24 ./logs/redis-cache.log
-rw-r–r-- 1 root root 104597 Jun 27 03:24 ./logs/redis-queue.log
-rw-r–r-- 1 root root 7738 Jun 27 03:23 ./logs/schedule.error.log
-rw-r–r-- 1 root root 4628480 Mar 21 05:57 ./logs/schedule.log
-rw-r–r-- 1 root root 9581933 Mar 15 12:35 ./logs/node-socketio.error.log
-rw-r–r-- 1 root root 0 Mar 9 09:53 ./logs/worker.log
-rw-r–r-- 1 root root 0 Mar 9 09:53 ./logs/web.log
-rw-r–r-- 1 root root 0 Mar 9 09:53 ./logs/redis-socketio.error.log
-rw-r–r-- 1 root root 0 Mar 9 09:53 ./logs/redis-cache.error.log
-rw-r–r-- 1 root root 0 Mar 9 09:53 ./logs/redis-queue.error.log

J0K3R · July 17, 2018, 8:11pm

Hi clarkej
thank you for your quick respond,
I’m using macOs 10.13.5 (as Im mentioned above), this command “sudo systemctl status mariadb” works on ubuntu not macOs, similar command is “launchctl”

I tried “launchctl error mariadb”, print out with no errors

clarkej · July 17, 2018, 8:53pm

ok to troubleshoot you will need to identify where macos writes its logs?

J0K3R · July 18, 2018, 1:08am

macos has an application called “Console”, that includes all log files, I couldn’t find anything relevant.

in frappe.log:
…
[DEBUG] 2018-07-18 05:13:39,791 | /Users/username/Dev/erpnext/apps/frappe/frappe/utils/scheduler.py:
Site: foo.boo
Access denied for site foo.boo
…

clarkej · July 18, 2018, 1:56am

A web search turns up this mac osx - Where is my mysql log on OS X? - Server Fault

This refers to some macos environment issues
[SOLVED] Install ERPNext on OS X (w/o VM) - #30 by saidsl

Other options are to install VirtualBox and run ERPNext on that or to use Docker - but those can be a learning challenge.

J0K3R · July 18, 2018, 11:56am

too old, 2009, no working

[SOLVED] Install ERPNext on OS X (w/o VM)
I did see this topic before, nothing helpful.

Thanks, I know the alternatives. but It’s just erasing the problem. I already have installed ERPnext on my Ubuntu VM, I got this problem since about 2 months ago, I did some research those days and find out MariaDB 10.2 deprecated on macOs also After reading the MariaDB documentation, I found out MariaDB 10.3+ do not support barracuda , better to say the first 3 lines mentioned below (also alittle change in syntax of configuration):

=========================
[mysqld]
innodb-file-format=barracuda
innodb-file-per-table=1
innodb-large-prefix=1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4

=============================

but when I removed them + changing the syntax, still the same error. this was the scenario!!

so I did the @dominik 's solution before(about 2 months), but it doesn’t solve the problem. I think this solution works when you’re updating, not a fresh install. I did follow:

over and over, reinstall brew, MariaDB , etc. nothing works!!

maybe bench checks all of those lines in my.cnf before proceeding, and If my.cnf doesn’t match all of those lines (Including innodb-file-format=barracuda, innodb-file-per-table=1, innodb-large-prefix=1 ) just print out :

==================================================
Creation of your site - foo.boo failed because MariaDB is not properly
configured to use the Barracuda storage engine.
Please add the settings below to MariaDB’s my.cnf, restart MariaDB then
run bench new-site foo.boo again.

[mysqld]
innodb-file-format=barracuda
innodb-file-per-table=1
innodb-large-prefix=1
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4

========================================================

@rmehta
to conclude I still couldn’t Turn-My-Mac-Into-a-Frappe-Framework-Developer-Machine
in the following env (after 2 months):

Macbook Pro (Mid 2012, MD102)
macOs 10.13.5
MariaDB 10.3.8
Python 2.7.15

It’s blowing my brain…!!!

clarkej · July 18, 2018, 12:36pm

ok yes I grok you enjoy the learning challenge and of course YMMV.

In my case that obliged me to flee Windows for Linux circa 2000, then later take up KVM
to tame environment problems. Docker awaits but you would have to pay me!

clarkej · July 19, 2018, 8:41pm

“to conclude I still couldn’t Turn-My-Mac-Into-a-Frappe-Framework-Developer-Machine”

Ahh so you refer to the wiki and say these entries:
Turning a Mac into a Frappe Framework Developer Machine · frappe/bench Wiki · GitHub
Installing Bench Pre requisites on MacOSX · frappe/bench Wiki · GitHub
edit: Home · frappe/frappe Wiki · GitHub

Thank so much for your reminder - I generally overlook that advice gold mine - how unfortunate that the discuss forum search omits that resource.

I think anyone registered on github.com can edit and update these?

J0K3R · July 20, 2018, 10:13am

thanks m8, be sure when I find the right solution I’ll edit that
I think something is missing, the writer forgot to type

anyway, I’m trying to run frappe_docker, hope it could relieve my pain :))

Vinob_chander_Ramasw · August 6, 2018, 4:23am

try using MariaDB 10.2

J0K3R · August 6, 2018, 9:27pm

deprecated on macOs ,

Vinob_chander_Ramasw · August 7, 2018, 11:44am

nope…i have set it up successfully on my mac running 10.13.1

Vinob_chander_Ramasw · August 7, 2018, 12:42pm

Barracuda engine could be configured on mariadb 10.2 and u r all set for development on ur mac …

antiquark100 · March 12, 2022, 5:48pm

Also, sometimes the old log files needs to be removed, especially after upgrade to MariaDB 10.3 -remove the ib_logfile* files in /var/lib/mysql. See https://mariadb.com/kb/en/upgrading-from-mariadb-101-to-mariadb-102/+comments/2903