If you woke up tomorrow and ran a marathon, how would you fare? It’s highly doubtful that you would successfully run the 26.2 miles without
months of training, drills, and exercises.
The same is true for disaster recovery (DR): The chance that you could successfully recover IT operations without having exercised your DR
plans on a regular basis is slim at best. The chance that you could successfully recover and meet your recovery objectives is zero. Yet Forrester
finds that exercising DR plans is one area in which many organizations continue to fall short.
Although most enterprises claim they conduct a full exercise of their DR plans at least once per year, anecdotal evidence suggests that the
majority of these exercises are not comprehensive and thorough; enterprises often just exercise a portion of the plan or a subset of
applications. Indeed, many of the organizations Forrester has spoken with know that they need to improve their DR exercise program, but face
barriers such as a lack of executive support, limited employee resources, and a fear of interrupting business processes. If this sounds all too
familiar, consider the following 10 best practices for updating and improving your current DR exercise program:
1. Define Specific Exercise Objectives Upfront
Exercising for the sake of exercising is a waste of time. Make sure that there are clear and concrete objectives and goals set up front that
will help determine the ultimate success of an exercise. One objective may be as simple as, “Verify our stated recovery time and recovery point
objectives.” You could orient other objectives around training, such as, “Familiarize the database administrators with the plans for recovering
2. Include Business Stakeholders
Business owners play a vital role in your DR exercises, and you need to involve them from the start of the exercise until you have recovered
all services. Business stakeholders should verify the successful recovery of services. This has the dual benefit of ensuring that you have properly
recovered business processes with all of their critical components as well as ensuring that business stakeholders know what to expect in terms
of recovery capabilities and performance at the recovery site during an actual declaration.
3. Rotate Staff Responsibilities
It’s important that the person who wrote the DR plan is not the same person who executes the test, as it is unlikely that that individual
would be available in a real disaster. Some companies Forrester interviewed went so far as to have employees with little specific knowledge of a
system executing those tests, such as a system administrator running the database DR test. An important secondary benefit of a DR exercise is
training; by assigning staff to take on new roles during exercises, you are essentially cross-training staff in different areas.
4. Develop Specific Risk Scenarios For Your Exercises
Many enterprises conduct their DR exercises without specific scenarios; they tell the response team to assume the data center is “a smoking
hole.” It is important, however, to define specific risk scenarios even for DR testing for two main reasons: 1) It provides a more realistic situation
for the response team to react to, and 2) different scenarios require different actions from the IT staff. For example, the DR plan for a short
outage at the primary data center that only requires resuming operations would be different from a long-term outage that requires failover (and
eventually failback), which in turn would be different from scenarios where only portions of the IT infrastructure were down.
5. Run Joint Exercises With Business Continuity (BC) Teams
In our research, Forrester found that many BC and DR teams run all of their exercises separately and often fail even to communicate when
they run exercises. However, you should aim to exercise the full enterprise BC and DR concurrently at least once per year. This is especially
important if the data center is in the same location as corporate headquarters.
6. Vary Exercise Types From Technical Tests to Walk-Throughs
A common misconception in IT is that walk-throughs and tabletop exercises are not necessary for DR exercises. While it’s true that these types of exercises won’t test the technical capabilities of a failover, they are still critical for training, awareness, and preparedness. Interviewees told us that the majority of the time, exercises that didn’t go as planned actually struggled most with communication and employees’ understanding of their roles during the exercise. Non-technical exercises such as walk-throughs and tabletops will help make these processes go more smoothly.
7. Make Sure to Test All IT Infrastructure Concurrently at Least Once Per Year
Waiting longer than a year risks too much change in IT environments and personnel — you need to bring new staff members throughout the
organization up to speed on DR plans. The most advanced firms run full DR tests as often as four times per year. In between full tests, most
firms conduct component tests that vary in frequency depending on the criticality of the systems and rate of change in the environment.
8. Identify Members for the Core DR Response Team
The stress of working under time and resource restraints for long hours, often during nights and weekends, is something people cope with
in different manners. When picking a core response team to lead IT recovery, it’s important to pick people who can work under extreme amounts
of pressure (and sleep deprivation). During an exercise or test, identify those individuals who can remain calm and collected.
9. Learn From Your Mistakes
The point of running DR exercises is to find potential barriers to recovery while in a controlled environment. If you aren’t encountering
problems during your exercises and tests, it’s more than likely you aren’t looking hard enough, aren’t testing thoroughly enough, or you have
designed scenarios for recovery that are too simple. When you complete exercises and tests and you have identified problem areas, use what
you have learned to update plans and create best practice documents.
10. Report Results to Stakeholders
If your organization has recently made significant investments in improving preparedness, most likely executives and other business
stakeholders want to know what the return is on their investment — how prepared are you? Reporting exercise and test results regularly and in
a timely fashion gives executives and business leaders visibility into your DR program. Remember that the results are not pass/fail but should
detail aspects of recovery that went well and areas for improvement.
Rachel Dines is an analyst at Forrester Research,
where she serves Infrastructure & Operations professionals.