Frappe Bench Backup and Restore Guide

Frappe Framework
,
1

Overview

The bench backup command in the Frappe Framework is a critical tool for creating backups of a site’s database and associated files (both public and private). It ensures data integrity and recoverability in case of system failures, corruption, or migration needs.

What It Does

Database Backup

  • Creates a dump of the site’s database in .sql.gz format.
  • Includes all tables, records, and database schema necessary to restore the site later.

Public Files Backup

  • Compresses and backs up the public/files directory.
  • Contains user-uploaded files that are publicly accessible through the application (e.g., images, documents).

Private Files Backup

  • Compresses and backs up the private/files directory.
  • Includes sensitive files meant for internal use, such as attachments or confidential documents.

Storage Location

Backups are stored in the sites/{site-name}/private/backups/ directory by default.

How to Use It

To run the backup command, navigate to your Frappe Bench directory and execute:

bench --site [site-name] backup

Example:

bench --site ngo.local backup

Options

  • --with-files: Includes public and private files in the backup.
  • --compress: Compresses the database dump (enabled by default).
  • --verbose: Provides detailed output during the backup process.

Where to Find the Backup Files

After running the command, the backup files can be found in:

sites/[site-name]/private/backups/

Typically, the directory contains:

  • database-timestamp.sql.gz: The database dump.
  • files-timestamp.tar: The public files backup.
  • private-files-timestamp.tar: The private files backup.

When to Use It

  • Before Upgrades: To ensure you can restore your site if the upgrade fails.
  • Before Migration: To transfer your site to another server or instance.
  • Periodic Backups: As part of regular maintenance to safeguard against data loss.
  • Pre-Restoration: To preserve the current state before restoring an older backup.

Restore Dump Database and Public & Private Files

Step 1: Obtain the Backup Files

Ensure that you have the following backup files:

  • Database dump file (e.g., database-timestamp.sql.gz)
  • Public files archive (e.g., files-timestamp.tar.gz)
  • Private files archive (e.g., private-files-timestamp.tar.gz)

Ensure that you enable scheduler, maintenance, server_script, and developer mode in your site:

bench --site {site_name} set-config enable_scheduler true
bench set-config -g developer_mode 1
bench --site [site-name] set-maintenance-mode off
bench set-config -g server_script_enabled true
bench --site [site-name] set-config server_script_enabled true

Verify After Enabling

Check if the scheduler is now enabled:

bench --site {site_name} doctor

You should see:

{site_name}: Scheduler is enabled

Step 2: Prepare the Environment

Ensure the bench environment is running:

bench start

Make sure that the backup files are in the correct format:

  • Database dump: .sql.gz
  • Public and private files: .tar.gz

Step 3: Restore the Database and Files

3.1 Restore the Database

bench --site {site_name} restore {path_to_database_file}

Example:

bench --site mysite.local restore /path/to/database-timestamp.sql.gz

3.2 Restore Public & Private Files

bench --site {site_name} restore {path_to_database_file}       --with-public-files {path_to_public_files_archive}       --with-private-files {path_to_private_files_archive}

Example:

bench --site mysite.local restore /path/to/database-timestamp.sql.gz       --with-public-files /path/to/files-timestamp.tar.gz       --with-private-files /path/to/private-files-timestamp.tar.gz

3.3 If restore fails, use --force:

bench --site {site_name} restore {path_to_database_file} --force

The --force option will force the restoration and may overwrite existing data. Use it with caution.

Step 4: Verify the Restoration

  • Visit your site in a web browser.
  • Check that all data and files have been restored properly.
3 Likes
2

That’s a great project you started.

I remember some data loss I incurred in the beginning due to lack of knowledge.
That was due to my lack of understanding how apps are integrated into a dev setup and not having set up my git repo correctly for my tiny starter app to be safe from a hard git --reset.

Backups and Restoring is something you need to be able to trust (and how does one build trust in a system principally built by others?).

Another instance of “Ensure that you have the following backup files” was made hard when I wanted to take backups of a docker instance.
That in itself is not simple if you lack knowledge of docker.

It was more difficult at the time because the scheduler in docker didn’t work as it should, which interfered with getting my requested backups.

This was not made easier by the system not having a proper email setup, which in itself is another task you need to get right if you want to get the “download link mailed to you once the backup is ready”.

With frappe_docker and frappe on kubernetes, even getting a scheduled backup needs different methods.
And how do you restore into a docker or a kubernetes instance?

All these aspects are certainly useful extensions to this guide for anyone using a production instance (officially recommended to be using a docker setup).

Even a simple but kept current list to all the relevant documentation would be a very useful thing.

I’d like to add that these things are great impediments to an implementation, because before having my colleagues using the system productively, I want to be sure no data loss will occur (their work done!) or at least can be recovered from.

So getting this (Dev)Ops thing right is of paramount importance, besides all the other configuration and app building aspects. It’s easy to tinker and get started, but once the first bliss and expectations built, you better make sure to get these things right.

And the Ops aren’t really taught by the other usual teaching paths.
I won’t blame anyone for that, though. It would be like insulting one’s parents.

1 Like
3

Why these are needed? that is no true. I think bench scheduler enable should also work but is not related to backups.

3 Likes
4

while developing, i always got problems if these options are not set,
so it became like a priority step for me

1 Like
5

Thank you for sharing your experience.
That’s true—unfortunately, this framework is a hidden gem, but it lacks clarity.
You often have to go through trial and error yourself to really understand how to work with it.

I believe this knowledge gap can be reduced if we gather everyone’s experiences into a proper tutorial or guidebook, just like other technologies.
It’s not easy, but I believe one day it will be done.

1 Like
6

Can you please add these tutorials on the official documentation?

1 Like
7

Hi,

backup/restore documentation is already available on official site.

ChatGPT is generating the answer from official documentation only.

Thanks,

Divyesh

1 Like
8

it’s already in the documentation,
i just redesign it the way i understand it

9

the documentation has everything butt not well explained or not cover everything
check this repo please:

thanks

10

That’s also a phrase that needs some thought.
How do you ensure or even check the “restored properly” part?
There are literally hundreds of tables in the system.

How to you check something like this? That “everything” is where it should be, without corruption, even hidden state corruptions, and working.
Taking 3 minutes clicking around here and there probably won’t do the trick.
And even then, where do you click and check for what?

What would be a better, more systematic way to check for the integrity of the restore?

Additionally even telling you that so many records are lost (or there)?
That here and there a document state is corrupted?
A field’s value missing?

For instance, the first and the last timestamp of the tables might help. The number of values in a table. But you also need a diff from a new system, and from the old (backupped) working system. Are these in the backup? Or elsewhere?
And the user’s needs are different from an admin’s needs to ensure “I can now safely work again” without hidden surprises.
Maybe blockchain could help. Also a nice buzzword to have if used for something real, useful. “Backup and Restore with Blockchain Peace of Mind Ensuring” :slight_smile:
A mature product for real-life Backup-Restore even after problems (not just housekeeping migrations).

In other words, what would be needed to be done to be alive and kicking again after some event having broken the system and created the need for the restore (other than migration)?

1 Like
11

i didn’t work in that scale :slightly_smiling_face:,
btw this documentation i used with my team to test our app with dump data,
for production level, this type of scale was the devops team responsibility, and no one share with us this kind of experience or even write topic about it.

i am sorry i can’t help in this type of cases because i don’t have experience with it.
but i will be so thankful if you share any useful explaination related to this topic :pray:

your question hit the big point that we really need
thanks

1 Like
12

NP. :slight_smile:

You chose a general catch-all title which promises so much for future people entering search terms in the forum software.

So I said to myself that this might well become a central hub for this part of Ops, at least for some time, that I might as well share my thoughts about a high quality standard for such a guide.

That’s my part of contributing to the general discussion and your guide.
Of course I’m not there yet. Who is, anyway?

The loss of data can have very different monetary values attached to it according to the use case, from almost 0 to many thousands of whatever currency, according to the data entering frequency and the products’ values moved around in warehouses and deliveries.
If auditing is involved, the values will be even higher. Especially for tax auditing.
And if data retention is mandatory.

1 Like
13

I’d like to add that maybe the auditing part might be the path to a good recovery verification.

If frappe framework becomes really good at ensuring integrity, signaling data loss and integrity violations and makes recovering from data loss events easy (even if involving manual reentering from paper, limiting any effort of it to the strict minimum and providing as much clarity about the losses as possible), then the restore verification seems more like a useful by-product to the auditing functionalities to me.

1 Like