Every business needs to be prepared for the worst. IT infrastructure faces the risk of downtime at any moment – whether from a natural disaster, cyberattack, or simple human error. They need to know how to recover critical systems and data quickly and effectively. 

But many firms discover gaps in their disaster recovery plans only when it’s too late. This is why disaster recovery testing is so important. 

How do you do it? And more importantly, how do you do it well? 

Recently, our Head of Technical Operations, Ryan, created a video on this very topic. Whilst you’re welcome to watch for a more personal walkthrough, we’ve distilled those insights into this article for those who prefer reading. 

The fundamentals of disaster recovery testing: the six Ws 

As a Microsoft Azure MSP, at Synextra we find ourselves in a unique position when it comes to disaster recovery testing. While some organisations might conduct DR tests once every few years, our team is involved in these crucial tests regularly across various client environments. Offering Disaster Recovery as a Service (DRaaS) means we’ve seen what works, what doesn’t, and how to avoid the most common pitfalls. 

 So before diving into any DR test, we always start by asking six fundamental questions. These form the backbone of any successful disaster recovery testing strategy: 

Who? 

We typically see two distinct groups involved in disaster recovery testing. First, there’s the technical team conducting the test. Then there’s the stakeholder group – often driven by compliance requirements such as ISO certification or cyber essentials. Ideally, you’re testing because you want to ensure your DR solution works, with compliance being a beneficial side effect rather than the primary driver. 

What? 

This encompasses both your desired outcomes and your testing scope. We encourage our clients to be specific here. Do you need to test every server, or would testing one of each application type suffice? Your scope might include specific applications, databases, or just critical systems. The key is defining clear, measurable objectives. 

When? 

Timing isn’t just about picking a date. We’ve learned through experience that preparation time is crucial. A rushed DR test can affect production systems, so we always ensure proper lead time for planning and preparation. The ‘when’ also needs to consider business operations – you’ll want to minimise potential disruption to critical business functions. 

Why? 

This might seem obvious, but we often ask clients to articulate their specific reasons for disaster recovery testing. Beyond the apparent need to ensure business continuity, there might be compliance requirements, stakeholder assurance, or specific business risks to mitigate. Understanding your ‘why’ helps shape the entire testing approach. 

Where? 

In the Azure context, this usually means deciding between testing in production or in a separate test network. Each approach has its merits and challenges. We’ll help you weigh factors like network isolation, resource requirements, and potential production impact to make the right choice for your organisation. 

How? 

This breaks down into two crucial decisions: how will you define success, and how will you execute the test? We generally see two main approaches to execution:

• Live failover: This involves taking production servers offline and failing over to your DR environment. It’s the most realistic test but carries more risk and usually requires out-of-hours work.
• Test failover: This creates a separate test environment without affecting production. While less disruptive, it might not catch all potential issues due to necessary compromises in the test setup.

 Now, we’ll cover the three lifecycle phases of a DR test: pre-test, executing the test, and post-test. 

Pre-testing phase: the three pillars of disaster recovery

 So now that you’ve figured out the details, it’s time to make preparations. 

 Compute considerations 

When planning disaster recovery testing in Azure, compute resources need careful thought. In our experience, you don’t always need to replicate your entire production environment. Consider whether testing one of each application type would suffice – this can significantly reduce complexity while still validating your DR capabilities. 

 For compute planning, we focus on:  

• Server inventory and dependencies
• High Availability (HA) requirements in the DR environment
• Application-specific requirements
• Startup sequences and dependencies

 If you’re doing a test failover rather than a live failover, you might not need HA configurations – a single application server might be sufficient for testing purposes. 

Storage planning 

Storage often proves more complex than initially anticipated. When working with clients, we need to consider: 

• Storage synchronisation mechanisms (e.g., DFS, Azure zone replication)
• Access methods in the DR environment
• Storage location mapping
• File share dependencies

 A practical approach we often take is testing representative samples rather than every storage location. For instance, if you have 20 different file shares that are all on the same file server or Azure file storage, testing one or two might be sufficient. However, if specific shares are linked to critical applications, these need to be included in your test scope. 

Network architecture 

Network configuration is where we see the most potential for issues – it’s often the difference between a successful test and a problematic one. One wrong subnet configuration could accidentally trigger a live DR scenario instead of a test! 

 Key network considerations include:

• Whether you have layer 2 extension capabilities between sites
• IP address parity between environments
• Subnet mapping and configuration
• Static IP requirements and management

For IP parity, we ask questions like:

•  If a server is on 10.100.100.10 in location A, will it be on 10.200.200.10 in location B?
• Is there a consistent mapping pattern?
• Are any servers using DHCP that might cause issues?

 For security, we’ll think about: 

 Firewall policy replication between environments

• Security group mappings
• RBAC configurations
• Internet access requirements

For test failovers, security can sometimes be more relaxed since the environment is isolated. However, for live failover testing, you need exact security parity – every firewall rule needs to be mapped correctly to the new IP ranges. 

And then you’ll want to consider internet access. You can’t miss these if you’re testing internet-facing applications: 

• External DNS management
• SSL certificate handling
• Load balancer configurations
• Public IP mapping

Capacity planning 

 A crucial element we often see overlooked is capacity planning in the DR environment. We help clients check:

• Quota limits in the secondary region
• Whether quota increases have been replicated from primary to secondary regions
• Available CPU and memory resources
• Storage IOPS requirements
• Bandwidth requirements
• Connection limitations
• Network throughput capabilities

 We’ve seen cases where organisations have upgraded their primary environment but forgotten to mirror these changes in their DR configuration. For instance, you might have increased your quota to 500 CPUs in your primary region but still have the default 50 CPU quota in your DR region. 

Executing the DR test 

Domain and access management 

The first and most crucial step in any DR test is getting your domain services operational. Over years of conducting DR tests, we’ve learned that rushing this fundamental stage often leads to cascading issues throughout the rest of the test. Domain controllers must come online first – this isn’t just a best practice, it’s non-negotiable. 

Once your domain controllers are up, you’ll need to carefully work through Active Directory restoration and testing. This process requires patience and attention to detail. We typically spend considerable time verifying authentication services and access permissions across all systems. In our experience, investing time here saves hours of troubleshooting later. 

User access planning 

Before diving into application testing, you need a solid plan for how users will interact with the DR environment. This goes beyond simply ensuring systems are online – it’s about creating a testing environment that mirrors real-world usage as closely as possible. 

Key considerations include:

• Jump box configurations and access methods
• RDP access permissions and security
• Testing user group memberships
• RBAC assignments in the DR environment

DR Testing methodology 

A successful DR test is more like a carefully choreographed dance than a sprint to the finish line. We’ve developed our testing methodology through countless DR tests across various client environments, and the key is systematic progression. Start with your predefined startup sequence and stick to it religiously. Don’t be tempted to skip ahead or test multiple components simultaneously, even if everything appears to be working smoothly.  

Each application component should be tested individually before you begin testing integrations. This methodical approach might seem time-consuming, but it dramatically simplifies troubleshooting when issues arise – and they almost always do. Throughout the process, maintain clear communication channels between all team members. We’ve found that regular status updates and clear escalation paths are essential for smooth test execution. 

Real-time documentation 

Documentation during a DR test isn’t just about ticking boxes. It’s a way to create a detailed record that will prove invaluable both for immediate troubleshooting and future planning. We recommend maintaining a living document throughout the test that captures not just what you’re doing, but why you’re doing it and what you observe. 

Your documentation should include:  

• Detailed timestamps for all actions and observations
• Configuration changes and their rationale
• Issues encountered and their symptoms
• Implemented workarounds and their effectiveness
• Successful test completions and verification methods

The most valuable documentation often comes from unexpected situations. When something doesn’t go according to plan – and this happens more often than not – document your troubleshooting process and resolution in detail. These insights often become the foundation for improving your DR strategy. 

Post-testing DR activities 

Reflect and learn 

The real value of a DR test emerges in the aftermath. While it’s tempting to quickly close out the project once systems are back to normal, we’ve found that thorough post-test analysis is what transforms a good DR strategy into an excellent one. Schedule a detailed review session with all stakeholders while the test is still fresh in everyone’s minds. 

During these sessions, we encourage open and honest discussion about both successes and failures. What surprised you during the test? Which systems performed as expected, and which threw unexpected curveballs? Even seemingly minor observations can lead to significant improvements in your DR strategy. 

Fix and implement 

Post-test remediation isn’t just about fixing what went wrong during the test – it’s about strengthening your entire DR capability. We always remind our clients that issues discovered during testing are gifts; they’re opportunities to fix problems before a real disaster strikes. 

Start with your critical findings: 

• Address any immediate security concerns
• Fix configuration mismatches between production and DR
• Resolve identified networking issues
• Update incomplete or incorrect documentation

 The key is to implement these fixes in both your DR and production environments. We often see organisations fix issues in their DR environment while forgetting to mirror these changes in production, leading to configuration drift that will cause problems in future tests or, worse, during a real disaster. 

Upgrade where necessary 

Sometimes, a DR test reveals that your current infrastructure isn’t quite up to the task. This isn’t a failure – it’s valuable intelligence. Through our experience with numerous clients, we’ve found that resource requirements often evolve faster than DR plans account for. 

Take a hard look at your performance metrics from the test. Did systems perform as expected? Was failover as smooth as it should be? This analysis might point to necessary upgrades such as increased bandwidth, additional computing resources, or enhanced storage capabilities. Document these requirements and build a business case for any significant investments needed. 

Document and plan ahead 

Documentation shouldn’t be an afterthought – it’s a really important part of your future success. Think of your documentation as writing a letter to your future self or team members who might need to execute the DR plan under stress. What would they need to know? What would have made your recent test easier if you had known it beforehand? 

Focus your documentation on:

• Updated step-by-step procedures based on test findings
• Successful troubleshooting approaches and workarounds
• Network and system configuration changes
• Dependencies and their impact on recovery order
• Performance benchmarks and metrics

Finally, use this test as a foundation for planning your next one. In our experience, the most resilient organisations treat disaster recovery testing as an ongoing cycle rather than a one-off event. Schedule your next test while lessons from this one are still fresh and use your documentation to build an even more robust testing plan. 

Remember, each test makes your DR strategy stronger, but only if you take the time to learn from it and implement those learnings effectively. This methodical approach to post-test activities is what separates organisations that merely have a DR plan from those that have genuine disaster readiness. 

Azure-specific DR considerations and gotchas 

Through years of conducting DR tests in Microsoft Azure environments, we’ve seen numerous technical challenges that can catch even experienced teams off guard. Here’s what you need to watch out for with disaster recovery in Azure.

Office applications and cloud services 

Office applications can have unique challenges in DR scenarios, particularly around authentication and connectivity. In test environments, these apps often need to “phone home” to Microsoft’s services, which can be complicated by network isolation. Consider how your Office apps will authenticate in the DR environment and plan for any required internet access. This is especially crucial if you’re running a test failover in an isolated network. 

Active Directory complexities 

Active Directory restoration in DR scenarios demands special attention. AD servers don’t take kindly to being restored without proper preparation. Your DR environment needs a properly configured AD server, and you’ll need to follow specific steps during restoration to maintain directory services integrity. Pay particular attention to: 

• Domain controller startup sequences
• FSMO role holders and their restoration
• Replication configurations
• Authentication service availability

Group policy considerations 

Group Policy often causes subtle but significant issues during disaster recovery testing. Simply having your GPOs present isn’t enough – they need to be properly mapped to your DR environment. We frequently see issues where servers end up in the wrong OUs or inherit unexpected policies in the DR environment. Take time to verify that:

• Testing servers are in the correct OUs
• GPOs are being applied as expected
• Security filtering is working correctly
• WMI filters are functional in the DR environment

RBAC and security management 

Role-Based Access Control (RBAC) becomes particularly interesting in DR scenarios. You need to ensure that your testing users have appropriate permissions without compromising security. We often find that permissions that work perfectly in production need adjustment in DR, particularly when dealing with:

• Service principal access
• Resource group permissions
• Management group hierarchies
• Custom role definitions

Network access and RDP 

Remote Desktop access often proves trickier than expected in DR environments. We’ve seen numerous cases where RDP access works for some users but not others, usually due to complex interactions between group memberships, security policies, and network configurations. Plan your jump box strategy carefully, ensuring that testing users have appropriate access paths to the systems they need to test. 

FSLogix 

For organisations using virtual desktop infrastructure, FSLogix presents specific challenges. Profile containers and Office containers may not behave as expected in your DR environment. 

Pay special attention to:  

• Profile container accessibility
• Exclusion group configurations
• Storage location mapping
• Permission inheritance

DNS and certificate management 

Some of the most persistent issues we encounter relate to DNS and certificates. Stale DNS entries are particularly problematic – we’ve seen cases where servers that have existed for years, through multiple upgrades and changes, retain outdated DNS entries that only surface during disaster recovery testing. 

Certificate management also requires careful planning. Consider:

• SSL certificate availability in the DR environment
• Certificate storage locations
• Certificate server failover
• Certificate chain validation

Time synchronisation 

While it might seem minor, time synchronisation issues can wreak havoc in a DR environment. Many servers won’t tolerate time differences greater than five minutes, and when you can’t reach your usual NTP servers, time drift becomes a real concern. Plan for time synchronisation in your DR environment, especially for longer test periods. 

Windows Firewall configuration 

Firewall rules often need adjustment in DR environments, particularly when IP addresses change between primary and DR locations. For instance, if a server moves from 10.100.100.10 to 10.200.200.10, all associated firewall rules need to reflect this change. We recommend:  

• Creating a complete mapping of IP changes
• Updating firewall rules proactively
• Testing connectivity between services
• Verifying security group memberships

Internet access 

Internet access requirements vary significantly between live and test failover scenarios. In test failovers, you might need to carefully manage or restrict internet access to prevent conflicts with production systems. However, you still need to ensure that critical services can reach necessary external resources. This balance requires careful planning and configuration. 

Ready to strengthen your DR strategy?  

Disaster recovery testing is a seriously important investment for your business’ resilience.

Throughout this guide, we’ve shared insights gained from conducting numerous DR tests across various Azure environments. While the process might seem daunting, you don’t have to navigate it alone. 

At Synextra, we’re Azure experts as well as partners in your business continuity journey. Our team of cloud specialists combine deep technical expertise with a friendly, collaborative approach. Whether you’re planning your first DR test or looking to enhance your existing DR strategy, we’re here to help make sure your business is prepared for whatever challenges lie ahead. 

Want to discuss your disaster recovery testing needs? Get in touch today – we’d love to hear from you.