[docker production installation] internal server error

I have smoothly installed the ERPNext on a DO droplet using docker.
I have followed carefully the installation procedure and and went fine.
The certificates, the docker containers runs fine.

Whenever i wait later to go back to the site I get : INTERNAL SERVER ERROR.

action taken:

  • I have looked at the cotnainers logs → I found out the the erpnext-python_1 container cannot connect the the frappe_mariadb container.

MARIADB LOGS OUTPUT : Access denied for users <generated_users_of_the_docker_service>.

ACTION TAKEN:

  • logs in the mariadb container
  • open the root user
  • execute the following : Update mysql.user set password='<the_generated_password> where user='<the_generated_user>'; flush privileges; exit

THE ERROR STILL HERE AND AT THE MOMENT ITS THE ONLY ERROR I AM ENCOUNTERING.

MAYBE SOMEONE HAS A SOLUTION

3 Likes

what are the logs for erpnext-python container and erpnex-nginx container?

[ERPNEXT-PYTHON OUTPUT]:

    Attaching to frappe_docker_erpnext-python_1
erpnext-python_1          | Killed
erpnext-python_1          | Killed
erpnext-python_1          | Killed
erpnext-python_1          | Attempt 1 to connect to mariadb:3306
erpnext-python_1          | Attempt 1 to connect to redis-queue:6379
erpnext-python_1          | Attempt 1 to connect to redis-cache:6379
erpnext-python_1          | Attempt 1 to connect to redis-socketio:6379
erpnext-python_1          | Connections OK
erpnext-python_1          | [2020-12-14 12:23:16 +0000] [21] [INFO] Starting gunicorn 19.10.0
erpnext-python_1          | [2020-12-14 12:23:16 +0000] [21] [INFO] Listening at: http://0.0.0.0:8000 (21)
erpnext-python_1          | [2020-12-14 12:23:16 +0000] [21] [INFO] Using worker: gthread
erpnext-python_1          | [2020-12-14 12:23:16 +0000] [24] [INFO] Booting worker with pid: 24
erpnext-python_1          | [2020-12-14 12:23:16 +0000] [25] [INFO] Booting worker with pid: 25

[ERPNEXT-NGINX OUTPUT]:

erpnext-nginx_1           | Waiting for frappe-python to be available on erpnext-python port 8000
erpnext-nginx_1           | Waiting for frappe-python to be available on erpnext-python port 8000
erpnext-nginx_1           | Waiting for frappe-python to be available on erpnext-python port 8000
erpnext-nginx_1           | Waiting for frappe-python to be available on erpnext-python port 8000
erpnext-nginx_1           | Frappe-python available on erpnext-python port 8000
erpnext-nginx_1           | Waiting for frappe-socketio to be available on frappe-socketio port 9000
erpnext-nginx_1           | Frappe-socketio available on frappe-socketio port 9000

both seem to be okay.

can you set host to % for db user in mariadb container?

MariaDB> grant all privileges on `db_name`.* TO 'db_name'@'%' identified by 'db_password';

replace db_name and db_password with ones from site_config.json

refer: https://docs.bitnami.com/aws/infrastructure/mariadb/configuration/create-database/

I will have a trial on it.
I will restart the server and get all the docker servers restart as well.

May I know, why the user is not getting all the required permission and setup.
to me the bug is located to the part that is intended to automatically provision, the mariadb settings.

I think the first boot of the mariadb, the configuration is not being set properly.

What are your thought ?

To simplify things, the docker-compose.yml (located in root of frappe_docker) adds a site-creator container. This container runs whenever docker-compose up is executed. It runs the command to create site given in the env file. if the site is found container will stop and not re-create the site.

The mariadb configuration is taken from file https://github.com/frappe/frappe_docker/blob/develop/installation%2Ffrappe-mariadb.cnf which is mounted in mariadb container.

There is no additional mariadb configuration container, because it is assumed configuration file can be mounted from host into docker.

you can check the logs of site-creator docker container, if there were errors during site creation.

Also use tagged semver like v12.14.0 instead of v12 or edge in production setup.
If you don’t use the specific tag, every time the container starts with new image it will run migrations.

edge migrations may fail some times. It is like a nightly build.
every 24 hrs, latest stable build is created and migrated into latest development build. today’s successful run, Travis CI - Test and Deploy with Confidence. Anyone can keep an eye on upcoming migrations here.

The mariadb configuration is provided through host.
in the configuration file i don’t see anywhere setting the user=<generated_user> and password=<password_generated>. with that not available, it make sense that mariadb send back:
Access denied for <generated_user>@<host_ip>

needed mariadb root password is set here

If it runs in the travis ci cron job it should run anywhere, Travis CI - Test and Deploy with Confidence. I don’t know what is causing the mariadb access denied problem.

yeah then that’s make sense though.

but currently, i am re-installting the docker-compose in verbose mode.

this is what i got from the output:
` | 2020-12-14 16:56:28+00:00 [Note] [Entrypoint]: Stopping temporary server
mariadb_1 | 2020-12-14 16:56:29 0 [Note] mysqld (initiated by: root[root] @ localhost [])│
mariadb_1 | 2020-12-14 16:56:29 0 [Note] Event Scheduler: Purging the queue. 0 events │
mariadb_1 | 2020-12-14 16:56:29 0 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1 | 2020-12-14 16:56:29 0 [Note] InnoDB: Starting shutdown… │ command ‘cat’ from deb coreutils (8.30-3ubuntu2)
mariadb_1 | 2020-12-14 16:56:29 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql│ command ‘ratt’ from deb ratt (0.0~git20180127.c44413c-2)
/ib_buffer_pool │
mariadb_1 | 2020-12-14 16:56:29 0 [Note] InnoDB: Buffer pool(s) dump completed at 201214 │Try: apt install
16:56:29 │
mariadb_1 | 2020-12-14 16:56:30 0 [Note] InnoDB: Shutdown completed; log sequence number │
1625461; transaction id 21 │on
mariadb_1 | 2020-12-14 16:56:30 0 [Note] InnoDB: Removed temporary tablespace data file: │{
“ibtmp1”
mariadb_1 | 2020-12-14 16:56:30 0 [Note] mysqld: Shutdown complete

mariadb_1 | 2020-12-14 16:56:31+00:00 [Note] [Entrypoint]: Temporary server stopped │}root@erp:~# docker exec -it frappe_docker_mariadb_1 /bin/sh
mariadb_1 | │# mysql -u root
mariadb_1 | 2020-12-14 16:56:31+00:00 [Note] [Entrypoint]: MySQL init process done. Ready│ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: NO)
for start up. │# mysql -u root -p
mariadb_1 | │Enter password:
erpnext-worker-long_1 | 16:56:31 RQ worker ‘rq:worker:a81ccb375acd4a38a7e721ff3e1dc523.c27c515738f0.1│Welcome to the MariaDB monitor. Commands end with ; or \g.
4.long’ started, version 0.12.0 │Your MariaDB connection id is 22
erpnext-worker-long_1 | 16:56:31 *** Listening on long… │Server version: 10.3.27-MariaDB-1:10.3.27+maria~focal mariadb.org binary distribution
erpnext-worker-long_1 | 16:56:31 Cleaning registries for queue: long │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] mysqld (mysqld 10.3.27-MariaDB-1:10.3.27+maria~f│Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
ocal) starting as process 1 … │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Using Linux native AIO │Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic buil│
tins │MariaDB [(none)]>
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Uses event mutexes │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Number of pools: 1 │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Using SSE2 crc32 instructions │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Initializing buffer pool, total size = 2│
56M, instances = 1, chunk size = 128M │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: Completed initialization of buffer pool │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: If the mysqld execution user is authoriz│
ed, page cleaner thread priority can be changed. See the man page of setpriority(). │
mariadb_1 | 2020-12-14 16:56:31 0 [Note] InnoDB: 128 out of 128 rollback segments are act│
ive.
`

I have my mariadb container attached to stdin and after setting the earlier configuration you asked me to do so i got this.
mariadb_1 | 2020-12-14 17:18:45 22 [Warning] 'proxies_priv' entry '@% root@ad69486d7af8' │ ignored in --skip-name-resolve mode.

which mean it did not set the configuration properly

well the settings of the maria db on the initialization does not raise an error but instead send a warning with that there is not break of the process hence it wil continue to run and travis ci won’t see it as an error

These tests run in ci job. They create mariadb and postgres sites, ping them, migrate and then check homepage again.

Job log shows if migration was successful or failed.

May be try running the tests locally?
Test output should be like Travis CI - Test and Deploy with Confidence

The job logs are not showing the [alert] from MariaDB ‘on the initialization of the db user. The initialization of the user is not done properly but mariadb does not throw and error but a warning that silently pass, until you want to access the db with that user it denies your access’

To reproduce this please run the docker-compose up and watch the logs to see what i mean.

I tried docker-compose up locally, It worked. I shared my screencapture gif here, it’s about 5 min in duration.

Wait for the site-creator container to complete.

check logs with docker logs frappe_docker_site-creator_1 -f

It will show “Internal Server Error” if site-creator container is still running.

I made a ~8 mb gif of my screen. Posted it on frappe_docker issue.

Edit: docker-test.sh is useful unless proven useless. I trust it for now.

I have not tried it locally only on production.
i will give it a trial on my local computer and see.

after failing at installing the ERP docker production on my DO droplet.
I have decided to take an action and try it on PWD instance and its still failing.

This definitely confirm that the problem is not related to my installation but the current docker production deployment fails.

maybe you have a feedback. Thank you

http://ip172-18-0-75-bvd2221qckh000fkljtg-80.direct.labs.play-with-docker.com/

Wait for 10 min

It’s up now. Try username as Administrator and password as admin, I’ll keep it till they stop the session