Testing Plans

One thing we're going to want to do is test out our disaster recovery plan. Without testing it out, you have no idea if it's flawed or not, and there are several different ways that we could go about testing it out.

Why Test the Plan

Within our DR plan we are going to have processes that we're going to carry out to carry out our disaster recovery plan, but without testing this out we really don't know how effective it is.

So let's develop a little scenario here. Let's say we have a bank of web servers, we have a couple of database servers that are in high availability, and what we want to do is we want to test out our disaster recovery site and make sure that things can fail over correctly and that there's no issues.

Live Failover

The most basic form of this is just do a live failover. That is, all of our customers are on our live site and we bring it down and move them over to our backup site. So this is a true test to make sure that we can comply with everything that we need to comply with. But the problem with this methodology is our customers could be down, and if anything goes wrong and we go outside our recovery time objective, then this is really problematic for our customers. So this might not be the best solution many times. Although it's great from a making sure that everything fails over correctly standpoint, it could really have a negative impact and really be problematic to us too. It could be very painful.

Walkthrough or Tabletop Exercise

So what are some different ways that we could do this? Well, one, we could just sit down and run through our process, and this is like a walkthrough or a tabletop exercise. We call it a tabletop exercise because we're just bringing out the instructions with the team and then we're going through it step by step and just seeing what it looks like and seeing how it is, and making little corrections and correcting things as we go.

Simulated Exercise

We also have some sort of simulated exercise that we can do. Maybe we have a whole virtual environment and we can test it out in our virtual environment. Or I've had a multiple stage, like a stage environment before production, so we roll things out to stage and then production. Well, I would test out failing over stage, and it wouldn't be perfect, it wouldn't be exactly like production was, but it would give us that experience and make sure that our checklist and everything that we would go through, our processes, were accurate and were going to be effective.

Parallel Processing

There's also parallel processing. Maybe we have that live system out there that our customers are getting into, but we have our backup copy. Maybe we sever this connection between these two, bring this copy up, test it out without actually failing over our customers to it. So we've got some parallel processing going on.

Communicate the Test

Now there have been times that I've been failing over the non-production equipment, our staging equipment, and realized that there was a dependency with our live copy and we needed to fix that, but that did cause some outage and so there was some concerns over that. So even if I'm doing one of these other environments, no matter what environment I'm doing, with the exception of the tabletop here, but whatever environment I'm doing, I want to make sure I communicate these tests out to make sure whoever is affected with these tests, or could possibly be affected with these tests, understands what's happening.

As we go through these processes and test out the plan, we're going to learn a lot of things, so we're going to make adjustments to these processes and plans to really make sure that we're continually improving it and making it better.