How to Write a Disaster Recovery Plan When You Have 3 Engineers

Your database just disappeared. Your cloud provider had a regional outage. Your engineer accidentally ran DROP TABLE in production. You need a disaster recovery plan, and "we will figure it out when it happens" is not one.

The good news: a practical DR plan for a 3-person engineering team does not need to be a 50-page document. It needs to answer five questions and include tested procedures for each.

Question 1: What are you protecting?

List every critical system and data store. For a typical SaaS startup, this is: the application database (PostgreSQL, MySQL), file storage (S3 buckets), application configuration (environment variables, secrets), source code (GitHub), and the infrastructure definition (Terraform state).

For each item, define two numbers:

• RPO (Recovery Point Objective): How much data can you afford to lose? If your RPO is 1 hour, you need backups at least every hour.
• RTO (Recovery Time Objective): How long can you be down? If your RTO is 4 hours, you need to be able to restore everything within 4 hours.

For most startups: RPO = 1 hour, RTO = 4 hours. These are reasonable targets that do not require expensive infrastructure.

Question 2: What can go wrong?

The most common disaster scenarios for startups:

• Database corruption or accidental deletion (most common, usually human error)
• Cloud provider regional outage (rare but impactful)
• Ransomware or security breach (increasingly common)
• Application bug that corrupts data (common during migrations)
• DNS or certificate issues that take down the entire application

Question 3: How are you backing up?

Set up automated backups for every critical data store:

Database: Use RDS automated backups (free, continuous) with point-in-time recovery enabled. Also set up a daily pg_dump to S3 in a DIFFERENT AWS account (cross-account backup). This protects against account compromise. Test a restore monthly.

File storage: Enable S3 versioning and cross-region replication for any bucket containing customer data. Cost is minimal (you pay for the replicated storage).

Source code: GitHub provides this inherently (every clone is a backup). But also ensure your Terraform state is backed up. Use an S3 backend with versioning for Terraform state.

Secrets: Use a secrets manager (AWS Secrets Manager, HashiCorp Vault) rather than environment variables. Secrets managers provide versioning, audit logging, and rotation capabilities.

Question 4: How do you recover?

Write step-by-step runbooks for each scenario. Keep them simple enough that any engineer on the team can execute them at 3 AM while half-asleep.

Database recovery runbook (example):

1. Identify the point in time you need to recover to
2. Open the RDS console, select the database, choose "Restore to point in time"
3. Wait for the new instance to become available (typically 15-30 minutes)
4. Verify the data by running a set of predefined queries
5. Update the application configuration to point to the new database
6. Restart the application
7. Verify the application is functioning correctly
8. Terminate the old database instance (after confirming recovery)

Total recovery time: 30-60 minutes. Well within a 4-hour RTO.

Question 5: How do you test it?

A DR plan that has never been tested is a hope, not a plan. Schedule quarterly DR tests. Pick one scenario each quarter and run through the recovery procedure. Track how long it takes and document any issues.

Start simple: restore your database from backup to a separate instance. Verify the data is correct. Time the entire process. If you can do this successfully, you have covered the most common disaster scenario. If you want the restore path built and tested rather than assumed, that is part of our DevOps work.

Ransomware changes the math on backups

Everything above assumes the disaster is accidental. Ransomware is not, and an attacker who reaches your cloud account will look for backups before encrypting anything, because destroying your recovery path is what makes the ransom work. Automated snapshots sitting in the same account, under the same credentials, protect you from an engineer running the wrong command and from nothing else.

The property you need is immutability. In practice this means backups written to object storage with object lock enabled in compliance mode for a defined retention window, held in a separate account whose credentials do not exist anywhere in your production environment. The production role should be able to write a backup and should not be able to delete or overwrite one. If the same IAM policy allows both, an attacker with production access has both.

Test the boundary rather than assuming it. Take your production deployment role, attempt to delete a backup object, and confirm the denial. It takes five minutes and it is the single most useful assertion in a small team's DR plan. The related check is whether the cross-account backup job is failing silently: a nightly dump that has been erroring for six weeks because a credential rotated is a very common finding, and nobody notices because nobody alerts on the absence of a success.

The restore nobody tested

Restoring a database is not the same as restoring a working application, and the gap is where RTO estimates fall apart. A logical dump restores tables and data. It may not restore extensions, roles and grants, auto-increment counters at the right value, custom types, or the search path your application assumes. Restoring into a fresh instance in another region also means a fresh parameter group, a fresh security group, and a fresh subnet configuration, and any of those being wrong looks identical to the application: connections fail.

Then there are the things around the database. Your DNS records have a TTL, and if yours is set to 86400 seconds because someone copied a default, a cutover you can perform in fifteen minutes takes a day to propagate. Your TLS certificates may be tied to an issuance method that assumes the old infrastructure. Your search index, if you run Elasticsearch or a vector store, is derived data that has to be rebuilt, and rebuilding it on a large corpus can take longer than everything else combined. Nobody includes index rebuild time in their RTO until the first real test.

Payment and messaging integrations deserve their own line. If your processor has been sending webhooks during the outage, some of them failed and are being retried, some have exhausted retries and are gone, and your order state is now partially divergent from theirs. Know in advance how to request a replay and how to reconcile. The same applies to any queue that dropped messages and any outbound email that bounced during the gap.

Who decides, and who talks

The technical runbook is the easier half. The half that goes wrong is human. Somebody has to decide that this is a disaster and that the recovery procedure starts, and if that decision has no owner it gets made by committee at 3am while the outage continues. Name one person and one backup, give them the authority to declare it without waiting for anyone, and write down the threshold: this many minutes of total unavailability, or confirmed data loss, or confirmed unauthorized access.

Communication runs in parallel and needs its own owner, because the person executing the restore cannot also be answering customer emails. Decide now what you will say at 30 minutes, at two hours, and at resolution, and where you will say it. A status page hosted on the infrastructure that just went down is not a status page. Neither is a Slack workspace if your identity provider is part of the failure, which is a reason to keep one out-of-band channel and a printed or offline copy of the contact list.

Check your contracts too. Enterprise agreements frequently carry notification obligations with a clock attached, sometimes 24 or 48 hours, and privacy laws add their own. Under PIPEDA you must report breaches of security safeguards creating a real risk of significant harm to the Privacy Commissioner and to affected individuals as soon as feasible, and keep records of breaches regardless of whether they meet that threshold. Knowing which customers have contractual notification clauses before the incident saves hours you will not have.

The evidence an auditor wants

DR shows up in SOC 2 under availability, and in ISO 27001 through the backup and continuity controls. The requests are consistent and easy to satisfy if you have been running the quarterly tests already. Expect to be asked for your documented plan with a recent review date and a named owner, your defined RPO and RTO with the business reasoning behind those numbers, evidence that backups completed successfully during the observation period, and records of at least one restoration test with the date, who ran it, how long it took, and what went wrong.

The last item is where teams over-polish and undermine themselves. A test record saying everything went perfectly reads as a test that was not really run. A record noting that the restore took 90 minutes against a 4 hour target, that the application configuration had to be updated by hand, and that a ticket was opened to automate it, is far more credible and is exactly what an assessor is hoping to see. Findings in your own test are evidence the control operates.

What lower RTO actually costs

Every hour you shave off recovery time costs more than the previous one, and the curve gets steep quickly. Point-in-time restore within a single region, which is what most of this article describes, is nearly free because it uses features you already pay for. A warm standby in a second region means paying for duplicate compute and storage continuously, plus the engineering to keep configuration in step, and it typically doubles your infrastructure bill for the covered services. Active-active across regions means solving data consistency, which is an architectural commitment rather than a line item.

Before buying any of that, ask what an hour of downtime actually costs you and whether any customer contract carries an availability commitment with a credit attached. Most seed and Series A companies find that a four hour RTO is comfortably inside what their customers tolerate and their contracts require, and that the money is better spent on making the four hour path reliable than on chasing thirty minutes. Multi-region is a conversation to have when a specific customer requires it in writing and is paying accordingly.

When you should not buy this from us

If you run a single managed database with automated backups already enabled, and your entire infrastructure is defined in Terraform, you can build and test this yourself in two days. Enable point-in-time recovery, add a cross-account copy with object lock, write the runbook in your existing docs, restore once to prove it, and put a recurring calendar entry on the quarterly test. Paying a consultancy to produce a plan document you could have written is a waste, and a plan document by itself changes nothing about your ability to recover.

The cases where outside help earns its cost are narrower. You have a genuinely complex estate where the restoration order across services is unclear. You have a contractual availability commitment you are not confident you can meet. Or you have never once tested a restore and would rather someone who has done it before run the first test with your team watching. If you want the runbooks and the tested restore path rather than a document, that is part of what a retainer covers, and the same arrangement means the contracts and access already exist if the day comes when the outage is an intrusion. If you are chasing an audit and DR is the open item, our compliance work starts with a gap analysis so you know what is actually missing before you spend anything.