BLOG /
Disaster Recovery Testing: Types, Checklist, and Why a Pass Goes Stale

Disaster Recovery Testing: Types, Checklist, and Why a Pass Goes Stale

Lev Petrov
Lev Petrov
R&D Team Leader
Disaster Recovery Testing: Types, Checklist, and Why a Pass Goes Stale

TL;DR

  • A disaster recovery test proves one thing: that the plan worked against the environment you had on test day. In the Veeam Data Trust and Resilience Report 2026, 90% of organizations felt confident they could recover, yet only 28% of ransomware victims got all of their data back.
  • There are five common types of DR testing: plan review, tabletop, walkthrough or simulation, parallel test, and full interruption. Each proves something different, so most teams need a mix.
  • A useful DR testing checklist runs in three stages. Before the test, scope the critical services and confirm the dependency map. During it, record actual recovery times. After it, assign owners to every gap.
  • DORA asks financial entities to test systems behind critical functions at least yearly, and to retest continuity plans after substantive changes. NIST SP 800-34 sets the federal baseline.
  • Tests run on a calendar, and the environment changes every day. The Gambit resilience platform keeps the dependency map, backup immutability and drift view current between tests, so the next test starts from today's environment.

The DR test passed in March. By April, the team had moved two services to a new region, rotated the identity provider's signing keys and added a payments vendor. Yet the test report still said “pass.”

And that gap is common. Cutover puts the mean success rate for meeting a defined RTO on mission-critical applications at around 64%. Identity, one of the first systems attackers go after, gets tested even less. In Quest's 2026 survey, only 24% of organizations test identity recovery every six months, and 24% never test it.

Below, we cover the five test types, a before, during and after checklist, and how often to run each one.

What is disaster recovery testing?

Disaster recovery testing is the practice of running a recovery plan under controlled conditions to check that critical services come back within their recovery time and recovery point objectives. It also checks that the people, runbooks and dependencies behind those services work when it counts. A DR test answers a narrow question: can this service come back, in this environment, within the time and data loss the business agreed to? A useful test covers the RTO and RPO for each service, the runbooks that describe the steps, the people who run them, and the dependencies the service needs to work again.

Dependencies are where a DR test differs from a backup restore test. A restore test proves the data comes back, while a DR test proves the application runs on it, with identity, DNS, network rules, secrets and the services it calls next. Our post Congrats on the Backup! Now, Where's the App? walks through what breaks between those two.

So start with the minimum viable business, the smallest set of systems, data and processes the company needs to keep earning revenue. Those services set the scope of your first tests and the strictest targets.

The 5 types of disaster recovery testing

Each test type trades effort and risk for confidence. Lighter tests catch gaps in the plan. Heavier tests catch gaps in the environment.

Test type What it proves What it misses Effort Risk
Plan review The plan is current and complete Whether anything actually recovers Low None
Tabletop exercise People know their roles and decisions Technical gaps and real timings Low None
Walkthrough or simulation The steps work in an isolated environment Production load and live dependencies Medium Low
Parallel test Recovery systems work with real data How production behaves during a cutover High Low to medium
Full interruption or failover The business can run from the recovery site Little, for the services in scope Very high High

Plan review. The owners read the plan against the current environment and fix what's out of date. Reviews are cheap and fast, and they catch stale contacts, retired systems and missing services.

Tabletop exercise. The team talks through a scenario, such as a ransomware attack on the identity provider, and walks each decision. Tabletops train people and expose unclear ownership. But they can't show whether a system actually comes back.

Walkthrough or simulation. The team performs the recovery steps in a sandbox or isolated environment, without touching production. Dependency gaps usually start to surface here.

Parallel test. The team brings up recovery systems alongside production and checks that they work with real data. Production keeps running, so risk stays low, but the cost of standing up a second environment is real.

Full interruption or failover. The team shuts down or fails over production and runs from the recovery site. Failover gives the strongest proof and carries the most risk, so most teams save it for Tier 1 services, on a planned date.

Most programs also run component tests in between: a single database restore, or a single-service failover. They are quick, low risk and easy to repeat.

No team can predict the next disruption, and the list keeps growing. AI agents now make changes in production on their own, and in one widely reported case an AI coding agent wiped a company's production database. Attackers use them too, as our research on Aurora ransomware shows. So test what breaks: identity gone, a region down, backups deleted or encrypted, a critical dependency changed without warning. Each scenario stands in for many causes. Include at least one where the recovery tools are under attack too, as they were in the Ababil of Minab campaign.

Backup and DR vendors have built tooling for these tests. Veeam's SureBackup boots backups in an isolated environment to check they're recoverable. Zerto offers test failovers that run without disrupting production. Use them for the data and infrastructure layers, and plan the application dependencies around them.

Disaster recovery testing checklist

A test is only as good as its preparation. Use this checklist for each critical service in scope.

Before the test

  • Scope. Agree with the business which services keep revenue flowing, starting with the minimum viable business. Then map what each one runs on. In most enterprises that map lives in people's heads and spreadsheets, and one service spans several teams. So name one person accountable for each service in the test, even when many teams run it.
  • Targets. Set the RTO and RPO for each service, with the business reason. Our post on the real cost of downtime shows how to tie them to revenue.
  • Dependency map. Confirm what each service runs on today: identity, DNS, network rules, secrets and the services it calls. Include security tools, which most plans leave out, as our post on the security-tool blind spot explains.
  • Backups. Check coverage, retention and immutability for each service, and pick a recovery point you've checked. Our post on immutable backups and the 3-2-1 rule covers the basics.
  • Infrastructure as code. Compare declared state with what is running. Changes made by hand will not come back from code.
  • Roles and comms. Name who runs each step, who decides, and how the team talks if email or chat is down.
  • Pass and fail criteria. Write them down before the test starts, for each service.

During the test

  • Timestamps. Log the start and end of each step.
  • Actual RTO and RPO. Measure them, and compare them with the targets.
  • Deviations. Record every step that differed from the runbook, and why.
  • Failback. Test the return to production. It is part of the recovery.

After the test

  • Gap report. List each gap with an owner and a due date.
  • Runbooks. Update them with what the team actually did.
  • Retest. Rerun the steps that failed.
  • Evidence pack. Keep timestamps, results and sign-offs for auditors and regulators.

How often should you test disaster recovery?

Most enterprises review the DR plan quarterly, run tabletop exercises twice a year, test Tier 1 components quarterly, and run a parallel or full test at least yearly. Add a test after major changes. The right cadence depends on how critical each service is and how fast its environment changes. Use this table as a starting point.

Test Tier 1 (minimum viable business) Tier 2 Tier 3
Plan review Quarterly Quarterly Twice a year
Tabletop exercise Twice a year Yearly Yearly
Component tests Quarterly Twice a year Yearly
Parallel or full test At least yearly Yearly Every two years

Some events should trigger a test outside the calendar: a cloud or data center migration, a major architecture change, a new critical vendor, or an emergency change like the patch that becomes a planned outage.

Regulators are moving in the same direction. DORA's Article 24 requires financial entities to test the systems that support critical or important functions at least yearly. Article 11 adds continuity plan tests after substantive changes. For US federal systems, NIST SP 800-34 sets the contingency planning and testing baseline.

Why does a passed DR test go stale?

A test result describes one environment on one day, and cloud estates change faster than that. Engineers make changes by hand in the console, infrastructure code drifts from what is running, AI agents make changes on their own, and new dependencies arrive with each release.

Each change can break a recovery path the last test proved. When a service moves regions, the runbook can still point at the old one. A new API call adds a dependency nobody mapped. Dependency maps built by hand go stale for the same reason, as our post explains why your cloud map is already wrong.

So a quarterly or yearly test can't carry the whole load. A calendar sets when the team rehearses. But the environment sets when the plan stops matching.

Between tests: continuous readiness

Most of us run DR tests on a calendar because that's what the tools allowed. Each test is also a heavy lift for the business and the team: weeks of preparation, engineers pulled off roadmap work, and sometimes a change freeze or planned downtime. So testing more often isn't a realistic way to close the gap. Readiness starts slipping within hours of a test, and that's the gap the Gambit resilience platform fills, without adding another test to the calendar. It connects through read-only APIs, with no agents to install, and keeps three inputs to the next test current:

  • The dependency map. It maps cloud resources and backups to the business applications they support, and updates the map as the environment changes.
  • Backup posture. It checks whether backups are immutable, and flags short retention windows and single-region dependencies.
  • Drift. It catches infrastructure-as-code drift and new dependencies, whether a person, a pipeline or an AI agent made the change, so you know when to retest.

Your team still runs the tests. With a current view, they can keep critical services running or recover them fast, and fix the gaps that matter most first.

What comes next

Auditors and regulators now ask for evidence of recovery, along with the plan itself. DORA supervisors expect test results tied to critical functions. And AI-driven change will widen the gap between test days. Teams that pair a test calendar with a current view of their environment will have the stronger answer.

‍

Find out which critical services would keep running, and how fast the rest would come back. Book your Resilience Assessment.

FAQ

You ask? We answer

What is a disaster recovery test plan?

A DR test plan sets out what a test will cover and how it will be judged. It names the services in scope, the scenario, the test type, the RTO and RPO targets, who runs each step, and the pass and fail criteria. Write it before the test, and keep it with the results as evidence.

What scenarios should a DR test cover?

No list covers every disruption, so test what breaks: identity, a cloud region, backups, and a critical dependency changed without warning, whether by a person or an AI agent. Include at least one scenario where attackers target backups or recovery tools, since that's where many plans break. Rotate scenarios across tests so each recovery path gets exercised.

What is a DR tabletop exercise?

A tabletop is a discussion-based test. Participants walk through a scenario, such as a ransomware attack or a region outage, and talk through each decision and step. It builds shared understanding and exposes unclear ownership, but it does not show whether systems actually recover.

Who should be involved in a DR test?

Bring in the service owners, the infrastructure and platform engineers who run the steps, security, and a business owner who can confirm the service works for customers. Add a decision-maker with authority to declare a disaster, and someone to keep time and record results.

What should a DR test report include?

The scope and scenario, the target and actual RTO and RPO for each service, timestamps for each step, every deviation from the runbook, and a list of gaps with owners and due dates. Include sign-offs, so the report can serve as evidence for auditors and regulators.

Related Blogs

blog
July 9, 2026

You Can't Recover What You Can't Describe

May Kogan
Co-Founder & CTO
Read More
blog
July 27, 2026

Why your cloud map is already wrong

Gil Goldberger
Senior Software Engineer
Read More
blog
July 15, 2026

Congrats on the Backup! Now, where's the App?

Naama Etzion
Security Researcher
Read More

Resilience, verified. In your inbox

The latest from Gambit: research, insights, and live sessions