MariaDB kubernetes installation failed

Nader_Alabed · December 10, 2021, 5:10pm

following the instruction in the helm.erpnext.com to deploy erpnext on linode but mariadb failed to start:

kubectl -n mariadb get all
NAME READY STATUS RESTARTS AGE
pod/mariadb-0 1/2 CrashLoopBackOff 6 (36s ago) 6m41s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mariadb ClusterIP 10.128.32.149 3306/TCP,9104/TCP 6m41s

NAME READY AGE
statefulset.apps/mariadb 0/1 6m41s

Nader_Alabed · December 10, 2021, 11:07pm

I tried both repositories, bitnami and helm/stable the same issue is mariadb setup broken or is there a tested version, any contribution is appreciated.

kubectl -n mariadb get all
NAME READY STATUS RESTARTS AGE
pod/mariadb-0 1/2 CrashLoopBackOff 75 (4m10s ago) 6h2m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mariadb ClusterIP 10.128.32.149 3306/TCP,9104/TCP 6h2m

NAME READY AGE
statefulset.apps/mariadb 0/1 6h2m

revant_one · December 11, 2021, 12:51am

show pod logs/description

check if this setup works

github.com

frappe/helm/blob/1e3f8ba4dd5b5a8e2fa0aa0c8876361db2eaec07/tests/script.sh#L23-L25


      
          helm repo add bitnami https://charts.bitnami.com/bitnami
          helm repo update
          helm install mariadb -n mariadb bitnami/mariadb -f tests/mariadb-local-values.yaml

use these values helm/mariadb-local-values.yaml at 1e3f8ba4dd5b5a8e2fa0aa0c8876361db2eaec07 · frappe/helm · GitHub

I’ll change the title to “ERPNext Kubernetes installation failed” to “MariaDB Kubernetes installation failed”

Nader_Alabed · December 11, 2021, 4:20am

thanks for the link and instruction, I followed the steps but its stuck with container creating, no log generated still, but the pods description as below:

kubectl describe pods mariadb-0 -n mariadb
Name: mariadb-0
Namespace: mariadb
Priority: 0
Node: lke46013-73844-61b341c20012/192.168.178.79
Start Time: Sat, 11 Dec 2021 06:44:48 +0300
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=mariadb
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mariadb
controller-revision-hash=mariadb-864f9648c
helm.sh/chart=mariadb-10.1.0
statefulset.kubernetes.io/pod-name=mariadb-0
Annotations: checksum/configuration: 781a0a6f53555a0b763d33c5beb29f18910843d50c19be23d8912d068b98236d
Status: Pending
IP:
IPs:
Controlled By: StatefulSet/mariadb
Containers:
mariadb:
Container ID:
Image: docker.io/bitnami/mariadb:10.5.13-debian-10-r18
Image ID:
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: exec [/bin/bash -ec password_aux=“${MARIADB_ROOT_PASSWORD:-}”
if [[ -f “${MARIADB_ROOT_PASSWORD_FILE:-}” ]]; then
password_aux=$(cat “$MARIADB_ROOT_PASSWORD_FILE”)
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=120s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -ec password_aux=“${MARIADB_ROOT_PASSWORD:-}”
if [[ -f “${MARIADB_ROOT_PASSWORD_FILE:-}” ]]; then
password_aux=$(cat “$MARIADB_ROOT_PASSWORD_FILE”)
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
BITNAMI_DEBUG: false
MARIADB_ROOT_PASSWORD: <set to the key ‘mariadb-root-password’ in secret ‘mariadb’> Optional: false
MARIADB_USER: nader
MARIADB_PASSWORD: <set to the key ‘mariadb-password’ in secret ‘mariadb’> Optional: false
MARIADB_DATABASE: alfalehtech
Mounts:
/bitnami/mariadb from data (rw)
/opt/bitnami/mariadb/conf/my.cnf from config (rw,path=“my.cnf”)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-mariadb-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mariadb
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 21m default-scheduler Successfully assigned mariadb/mariadb-0 to lke46013-73844-61b341c20012
Warning FailedMount 5m39s (x5 over 19m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data config]: timed out waiting for the condition
Warning FailedMount 66s (x4 over 17m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[config data]: timed out waiting for the condition
Warning FailedAttachVolume 47s (x18 over 21m) attachdetach-controller AttachVolume.Attach failed for volume “pvc-5d39474d69b64f80” : rpc error: code = NotFound desc = Volume with id 225565 not found

revant_one · December 11, 2021, 8:05am

The volumes are not being attached. You need to use StorageClass available on your cluster to create pvc for mariadb data. Check if appropriate PVC are created/available and ready to use.

Nader_Alabed · December 11, 2021, 8:51pm

I created PVC as advised and reinstall with the PVC in the same namespace, it connect to the volume but a new error showed up: mkdir: cannot create directory ‘/bitnami/mariadb/data’: Permission denied.

kubectl describe pods mariadb-0 -n mariadb
Name: mariadb-0
Namespace: mariadb
Priority: 0
Node: lke46134-73989-61b49835905a/192.168.187.69
Start Time: Sat, 11 Dec 2021 23:19:38 +0300
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=mariadb
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mariadb
controller-revision-hash=mariadb-864f9648c
helm.sh/chart=mariadb-10.1.0
statefulset.kubernetes.io/pod-name=mariadb-0
Annotations: checksum/configuration: 781a0a6f53555a0b763d33c5beb29f18910843d50c19be23d8912d068b98236d
cni.projectcalico.org/podIP: 10.2.1.6/32
cni.projectcalico.org/podIPs: 10.2.1.6/32
Status: Running
IP: 10.2.1.6
IPs:
IP: 10.2.1.6
Controlled By: StatefulSet/mariadb
Containers:
mariadb:
Container ID: docker://dab1ef59d8dbb0098ae4e2d9acb09708bc4c9f511d7eec53520949300d9e54ce
Image: docker.io/bitnami/mariadb:10.5.13-debian-10-r18
Image ID: docker-pullable://bitnami/mariadb@sha256:b0efc13867aa45d0ecaa31f21b74e6a2fac44f720547e891ddc000ac02687000
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 11 Dec 2021 23:20:40 +0300
Finished: Sat, 11 Dec 2021 23:20:40 +0300
Ready: False
Restart Count: 3
Liveness: exec [/bin/bash -ec password_aux=“${MARIADB_ROOT_PASSWORD:-}”
if [[ -f “${MARIADB_ROOT_PASSWORD_FILE:-}” ]]; then
password_aux=$(cat “$MARIADB_ROOT_PASSWORD_FILE”)
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=120s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -ec password_aux=“${MARIADB_ROOT_PASSWORD:-}”
if [[ -f “${MARIADB_ROOT_PASSWORD_FILE:-}” ]]; then
password_aux=$(cat “$MARIADB_ROOT_PASSWORD_FILE”)
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
BITNAMI_DEBUG: false
MARIADB_ROOT_PASSWORD: <set to the key ‘mariadb-root-password’ in secret ‘mariadb’> Optional: false
MARIADB_USER: nader
MARIADB_PASSWORD: <set to the key ‘mariadb-password’ in secret ‘mariadb’> Optional: false
MARIADB_DATABASE: alfalehtech
Mounts:
/bitnami/mariadb from data (rw)
/opt/bitnami/mariadb/conf/my.cnf from config (rw,path=“my.cnf”)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-mariadb-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mariadb
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 85s default-scheduler Successfully assigned mariadb/mariadb-0 to lke46134-73989-61b49835905a
Normal SuccessfulAttachVolume 81s attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-d5a8f59130c04592”
Warning FailedMount 74s (x4 over 78s) kubelet MountVolume.MountDevice failed for volume “pvc-d5a8f59130c04592” : rpc error: code = Internal desc = Unable to find device path out of attempted paths: [/dev/disk/by-id/linode-pvcd5a8f59130c04592 /dev/disk/by-id/scsi-0Linode_Volume_pvcd5a8f59130c04592]
Normal Pulled 24s (x4 over 69s) kubelet Container image “docker.io/bitnami/mariadb:10.5.13-debian-10-r18” already present on machine
Normal Created 24s (x4 over 69s) kubelet Created container mariadb
Normal Started 24s (x4 over 69s) kubelet Started container mariadb
Warning BackOff 16s (x9 over 67s) kubelet Back-off restarting failed container

kubectl logs -n mariadb mariadb-0
mariadb 20:25:29.79
mariadb 20:25:29.79 Welcome to the Bitnami mariadb container
mariadb 20:25:29.79 Subscribe to project updates by watching GitHub - bitnami/bitnami-docker-mariadb: Bitnami Docker Image for MariaDB
mariadb 20:25:29.79 Submit issues and feature requests at Issues · bitnami/bitnami-docker-mariadb · GitHub
mariadb 20:25:29.79
mariadb 20:25:29.80 INFO ==> ** Starting MariaDB setup **
mariadb 20:25:29.82 INFO ==> Validating settings in MYSQL_/MARIADB_ env vars
mariadb 20:25:29.83 INFO ==> Initializing mariadb database
mkdir: cannot create directory ‘/bitnami/mariadb/data’: Permission denied

I appreciate your guidance and patience

revant_one · December 12, 2021, 1:05am

There is a note here (https://github.com/bitnami/bitnami-docker-mariadb#persisting-your-database) that says,

NOTE: As this is a non-root container, the mounted files and directories must have the proper permissions for the UID 1001

Make sure the permissions are correct.

refer FAQ | ERPNext Helm Chart to know how to connect pvc to pod and execute commands.

Nader_Alabed · December 12, 2021, 6:28am

Thank for your follow up, its mean a lot.
I followed the steps, but once I make the directories and delete the volume editor, the directories reset to the original with the mariadb installation.

Nader_Alabed · December 13, 2021, 11:11pm

i’m posting again to insure that with the re-installation mariadb reset the storage to original state, I tried even to change the ownership of the directories to root, nevertheless once i delete the volume editor and apply mariadb the directories are deleted.

is there a way to have both mariadb and volume editor in the same pod together, or is there a way to create the directory environment during the installation, or if there is another way to come over this.

thanks in advance

revant_one · December 14, 2021, 12:29am

What are the storage classes available? If default storage class works then volume permissions should not be a problem.

check primary.sidecars and secondary.sidecars in volumes to mount additional containers with mariadb pods.

github.com

bitnami/charts/blob/35c3398732a738967339e3d68942e14397807d9f/bitnami/mariadb/values.yaml#L456-L456


      
          sidecars: []

Nader_Alabed · December 14, 2021, 2:27pm

hi Revant,
the default storage is csi as below, still couldn’t work around this issue.

Name: pvc-5a37514acee848ee
Labels:
Annotations: pv.kubernetes.io/provisioned-by: linodebs.csi.linode.com
Finalizers: [kubernetes.io/pv-protection external-attacher/linodebs-csi-linode-com]
StorageClass: linode-block-storage-retain
Status: Bound
Claim: mariadb/data-mariadb-0
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 20Gi
Node Affinity:
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: linodebs.csi.linode.com
FSType:
VolumeHandle: 227375-pvc5a37514acee848ee
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1639412423746-8081-linodebs.csi.linode.com
Events:

revant_one · December 14, 2021, 5:04pm

check this

https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#permission-errors-when-enabling-persistence

Nader_Alabed · December 14, 2021, 7:03pm

Thanks very much, works like a charm