S3 backup settings and query

After being switched to a much degraded internet connection, our S3 backups are failing.

Looking into the app I can’t see where the settings are for things like what command is being run to perform the backups, and how any schedule is set.

I am guessing they are buried in the database ?

Either way can anyone tell me where to look so I can run the command by hand to see what’s failing. Or at the very least point me to some logging ?

Search for “Error Log” in your search bar to get some idea

Sadly all I see is this. And even when I click each error there is no date/time noted.

This is not a direct solution for the issue. It is an alternate method which I think is more efficient for backing up multiple sites or multiple benches.

I run bench --site all backup and restic backup sites commands from given bench to take database backups and have incremental snapshots of files including db dumps moved to s3 as a restic repository.

Official docs: Examples — restic 0.15.1 documentation

It reduced size of backups when I moved to restic snapshots. Before restic I archived the files and pushed them to s3 as well. It was time consuming and inefficient, it created copies of files in archives. With restic I keep last 30 snapshots, I take them every 12 hrs, it means I have snapshot of last 15 days and backup size is less than what it was when I was doing it without restic.

example cron / script you will have to run.

# Switch working directory to frappe-bench 
cd /path/to/frappe-bench
# backup db only
bench --site all backup
# list previous restic snapshots or init restic
restic snapshots || restic init
# backup files including db dumps
restic backup sites
# cleanup snapshots
restic forget --group-by=paths --keep-last=30 --prune

More: frappe_docker/backup-and-push-cronjob.md at main · frappe/frappe_docker · GitHub

Thanks for that. It’s way beyond my paygrade though :grinning: I just have to fill in while S3 is failing.

To help anyone else. I eventually discovered that before being parcelled up for S3, ERPNext seems to store the backups in: frappe/frappe-bench/sites//private/backups

So it’s quite easy to pull the files from there for offsite backup as a stopgap.