Thu, June 26, 2008 — CIO — Designing a disaster recovery plan has traditionally forced companies to strike a delicate balance. To create a plan that restores operations quickly, an enterprise needs to invest significant capital. On the other hand, costs can be cut dramatically if an enterprise is willing to withstand longer periods of operations downtime. During the planning stages and while the computer network runs properly, the forces to reduce costs are felt the strongest and often prevail. But when disaster strikes and the network goes down, everyone starts screaming to get the network up and running again, as fast as possible. Finding a way to walk this tightrope is a major challenge, but with the advent of virtualization, deploying disaster recovery plans that restore operations quickly—and at a reasonable cost—is quite possible.

At Transplace, we developed a new disaster recovery plan based on virtualization when we moved our infrastructure to a new production data center in 2007. We also took that time to refresh our hardware and review our overall architecture. Previously, we ran daily backups and physically moved the data to an off-site location. With this process, we risked being down for a half day if we experienced a problem in the middle of the day. This type of plan also limited us in that we only backed-up once a day, which meant we risked losing a day's worth of work. This plan also required us to have dedicated servers that sat idle except when we executed a recovery.

After we moved into our new data center in Dallas at the end of 2007, we began to plan our new disaster recovery data center in Arkansas, into which we moved in February 2008. At the storage level, we deployed network-attached storage and SnapMirror software from Network Appliance to create virtual storage for our database and application servers. SnapMirror allows us to send copies of all changes to our backup facility on a near real-time basis without impacting the performance of the applications. Anytime a record changes in production, it sends a copy to our disaster recovery facility. This shared-storage approach also allows us to manage storage centrally. We buy storage only when we need it.

At the database level, we deployed IBM P570s with AIX as the operating system, leveraging its logical partitioning technology. This combination allows us to partition each server to look like multiple servers, and we can run multiple database servers by sharing the capacity of the individual servers. In the disaster recovery facility, the database server runs four to six copies of Oracle that we use for testing and development most of the time, but if the need arises, we can shut down the virtual servers and run the disaster recovery instance of Oracle on that same server. This also allows us to make the most efficient use of our Oracle licensing costs, which are charged by each physical CPU core.