Automated Backup with Docker setup

Install / Update Setup
1

I am having issues with backing up erpnext on S3. I have set up the backup using the UI Integration and received an email notification that the backup is successful but there are no objects in my s3 bucket.

Prior to this, I have received the following error messages while backup:

Oops, your automated backup to Amazon S3 failed.

Error message: Traceback (most recent call last): File “/home/frappe/frappe-bench/apps/frappe/frappe/integrations/doctype/s3_backup_settings/s3_backup_settings.py”, line 83, in take_backups_s3 validate_file_size() File “/home/frappe/frappe-bench/apps/frappe/frappe/integrations/offsite_backup_utils.py”, line 105, in validate_file_size file_size = get_file_size(latest_file, unit=“GB”) File “/home/frappe/frappe-bench/apps/frappe/frappe/integrations/offsite_backup_utils.py”, line 75, in get_file_size file_size = os.path.getsize(file_path) File “/opt/bitnami/python/lib/python3.7/genericpath.py”, line 50, in getsize return os.stat(filename).st_size TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

I have setup ERPNext using docker following the Single Bench instructions here: GitHub - frappe/frappe_docker: Docker images for production and development setups of the Frappe framework and ERPNext

ERPNext: v12.16.2
Frappe: v12.12.0

Any help will be highly appreciated. Thanks.

2

I don’t think if it has anything to do with Docker,

3

As a devops workaround you can setup docker based backup and push as a cronjob

Note: It’s not as easy as docker-compose up -d you may need additional configuration and knowledge about containers.

4

Thanks @revant_one, I will give docker based backup a shot.

If I manually want to take backup of my docker setup willl taking an sqldump from mariadb container sufficient? What will I miss in that case?

I am very new to ERPNext and trying to get a hang of it.

5

For backup you need.

  • SQL dump
  • copy of private files, /home/frappe/frappe-bench/sites/site.name.com/private/files
  • copy of public files, /home/frappe/frappe-bench/sites/site.name.com/public/files
  • copy of site_config.json, /home/frappe/frappe-bench/sites/site.name.com/site_config.json

I don’t think non docker S3 backup pushes site_config.json, you’ll need to manually back it up.

Why backup site_config.json? Encryption key is invalid, Please check site_config.json on restore

6

@revant_one I did take a backup of all of the files you have suggested. Now, to test if I have done it correctly I was trying to restore the backup on a new instance using bench restore command.
But I am receiving an internal server error with the following error logs:

    self.connect()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 599, in connect
    self._request_authentication()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 861, in _request_authentication
    auth_packet = self._read_packet()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, "Access denied for user '_ac0d16c08c91ddf8'@'172.18.0.4' (using password: YES)")

Not sure what I did wrong. Any pointers?

7

In case of fresh restore you will need to restore using bench new-site command.

refer Tutorial: Duplicate ERPNext site - #9 by revant_one

refer this docker based restore commands (I don’t think you are using docker based commands)

8

I was not using docker based commands as it needs backup on s3. Will put it manually on s3 and try again.

Will try if it works with new-site command.

Thank you so much.

9

use --no-mariadb-socket option with new-site command

bench new-site sitename.localhost --no-mariadb-socket --source_sql db.sql
10

@revant_one I tried restoring the backup on my existing system and It worked perfectly.

Thanks

11

@revant_one I would prefer the s3 upload to work.

Being a developer I can contribute to fix this issue. I just don’t know where to start. If you can point me to some resource or modules to look at it can give me a head start.

12

The frappe framework feature of S3 backup has open issues

Attempt to fix?

  • you’ll need to follow the trail from s3_backup_settings.py
  • use print() and console.log() Debugging
  • If you prefer debugger VSCode, PyCharm

You can use docker based alternative, It’ll also backup and push site_config.json.
more: GitHub - frappe/frappe_docker: Docker images for production and development setups of the Frappe framework and ERPNext

docker based alternative

docker task for s3 backups are tested everyday as part of cron job.

2 Likes
13

It seems to have started working after taking a manual backup for the first time.

Please see my comment here: Offsite backup fails · Issue #14823 · frappe/frappe · GitHub

14

you need cron job to run in your cluster.

refer GitHub - frappe/frappe_docker: Docker images for production and development setups of the Frappe framework and ERPNext

15

Sir this link is empty

image
image1759×851 46.3 KB

can you give the updated page which include the cron job addition to frappe HR running on docker

16