Blog
You Don't Have Backups Until You've Restored One
- databases
- devops
- laravel
- typescript
- reliability
Ask a team if they have database backups and the answer is almost always yes. There's a cron job. It runs every night. There's a bucket full of files with dates in the names.
Ask when they last restored one and it gets quiet.
A backup you've never restored isn't a backup. It's a hope with a timestamp.
What "success" actually means
When a backup job reports success, here's what you know: a process ran and a file got written somewhere.
Here's what you don't know:
- Whether the file is complete or got cut off halfway.
- Whether it's consistent, or caught some tables mid-write.
- Whether it includes everything you need (triggers, routines, the second database somebody added last year).
- Whether anyone still has the key to decrypt it.
- How long it takes to load back in.
None of those show up until you try to restore. The worst possible time to find out is during the outage.
Get a consistent dump first
If you're on MySQL or MariaDB with InnoDB tables, a plain mysqldump can capture tables at slightly different moments while the app keeps writing. Ask for a consistent snapshot instead:
mysqldump --single-transaction --quick --routines --triggers \
--databases app_production | gzip > app_production.sql.gz--single-transaction dumps everything from one consistent point in time without locking InnoDB tables. --quick streams rows instead of buffering whole tables in memory. --routines and --triggers are there because people forget them, and then the restored app behaves "weird" and nobody knows why.
If you have old MyISAM tables hanging around, --single-transaction doesn't make those consistent. Find out before it matters.
In Laravel, schedule it and make it loud
If you're using spatie/laravel-backup, the scheduling is a couple of lines in routes/console.php:
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Schedule;
Schedule::command('backup:clean')->daily()->at('01:00');
Schedule::command('backup:run')->daily()->at('01:30')
->onFailure(fn () => Log::critical('backup.failed'));
Schedule::command('backup:monitor')->daily()->at('09:00');backup:monitor checks things like "is the newest backup too old" and "is the storage used getting out of hand," and the package can notify you when those fail. Turn the notifications on and send them somewhere a human actually looks.
The failure you need to worry about isn't the loud one. It's the job that quietly stopped running three weeks ago because the scheduler died on one box. If nobody gets told when a backup doesn't happen, you'll find out the hard way.
Don't keep the backup next to the thing it's backing up
A backup on the same server as the database protects you from a bad query. It does nothing for a dead disk, a deleted server, or a compromised box.
At minimum:
- Ship backups off the machine, to storage in a different account or region if you can.
- Encrypt them, and store the key somewhere other than right next to the files.
- Make sure the credentials that write backups can't also delete every old one.
And remember the database isn't the whole app. Uploaded files, the .env values you'd need to boot the thing, and any third-party config that only lives in a dashboard all count too.
Actually restore one
Here's the part everyone skips. Pick a backup, load it into a scratch database, and check it.
mysql -e "DROP DATABASE IF EXISTS restore_check; CREATE DATABASE restore_check"
gunzip < app_production.sql.gz | sed 's/`app_production`/`restore_check`/g' | mysql restore_checkThat sed is there because --databases writes CREATE DATABASE and USE statements into the dump. Without it you'd restore right over the real thing, which would be a very memorable way to test your backups. (Running this on a separate box is even better.)
Then don't just check that it loaded. Check that the data makes sense. A small TypeScript script with mysql2 is plenty:
import mysql from 'mysql2/promise';
const db = await mysql.createConnection({ database: 'restore_check' });
const [[orders]] = await db.query<any[]>(
'SELECT COUNT(*) AS total, MAX(created_at) AS newest FROM orders'
);
const ageHours = (Date.now() - new Date(orders.newest).getTime()) / 36e5;
if (orders.total === 0 || ageHours > 26) {
throw new Error(`Restore looks wrong: ${orders.total} orders, newest ${ageHours.toFixed(1)}h old`);
}
console.log(`Restore OK: ${orders.total} orders, newest ${ageHours.toFixed(1)}h old`);
await db.end();Pick a couple of tables that should always have recent rows. If the newest order in last night's backup is from two weeks ago, you just learned something important, and you learned it on a Tuesday afternoon instead of during an incident.
Time it
While you're restoring, write down how long it took. Download, decompress, import, done.
That number is the real answer to "how long would we be down?" If the database has grown to the point where a restore takes four hours, that's fine to know. It's not fine to discover.
The other number worth knowing is how much data you'd lose. Nightly backups mean you can lose up to a day of writes. If that's not acceptable for the business, you need something more than a nightly dump, like point-in-time recovery from your database host or binary logs. Have that conversation before the bad day, not during it.
WordPress isn't special
Same rules. wp db export gives you the database, but the site is also wp-content/uploads, the theme, and whatever plugins somebody configured by hand. A hosting panel that says "daily backups" is nice. Restore one to a staging site anyway and click around.
Make the restore boring
The goal is a restore so routine that nobody's nervous about it. Put the restore check on a schedule. Write the steps down somewhere that isn't the server that just died. Have someone other than the person who set it up run through it once.
Then when somebody asks "do we have backups?" you can say yes and mean it.