India’s Massive Blackout Tests Outsourcing Providers

In what experts are calling one of the worst blackouts in history, more than half a billion people were without power when the grids in north and west India buckled twice in as many days. Multinational outsourcing providers with operations in the region appear to have passed the most significant test to date of their well-worn disaster recovery and business continuity processes.

“Indian providers have had to deal with these contingencies—international telecom cables being sliced, large union strikes that impact workers—so they are generally more prepared to deal with [them],” says Steve Hall, partner with outsourcing consultancy ISG.

“The problem of outages [in India] is by no means new,” adds Stan Lepeak, director of research for advisory services at KPMG. “[But] the recent event was extreme.”

News: Outsourcers Apparently Don’t Miss a Beat in Face of Massive India Power Outage

While the southern city of Bangalore remains the unofficial capital of India’s IT outsourcing industry, most major providers have operations in the northern metropolis of Delhi, and nearby Noida is home to the Indian headquarters of HCL Technologies and CSC as well as a major IBM Global Services data center.

Tier 1 and tier 2 providers have made significant investments in backup plans capable of providing at least five to seven days of continuous operations in the event of a power failure, says ISG’s Hall. Most campuses are equipped with diesel generators, water and other off-grid power sources.

“Their major operations centers are set up to operate like independent nation-states,” says Jim Slaby, research director of sourcing security and risk strategies for outsourcing consultancy HfS Research. “They also have contingency plans to relocate workers to backup locations in response to extended outages at any one facility, and the ability to reinforce the physical security of their facilities in the event of natural disasters [or] civil unrest.”

Infrastructure, Livelihood Affected by Blackouts

Indeed, in situations like the recent blackout, human resources issues often prove to be the biggest challenges. “The issue this time around is impact on transportation where people rely on mass transit,” says Atul Vashistha, chairman off offshoring consultancy Neo Group. As with a natural disaster, many employees remain at home.

“If a situation arises that becomes a widespread problem for the populace as a whole, the workers who staff facilities—with full power backup and supplies—will be more likely attending to family and personal matters than showing up for the next shift,” says Tom Young, partner with ISG, who has seen similar problems after hurricanes in Florida, for example. “Minimally, they will be distracted; worst case, they [will] not show up at all.”

How-To: 10 Disaster Preparedness Questions to Ask Your Cloud Services Providers

New Delhi-based NIIT Technologies offered the 10 percent of its employees affected by non-functioning metro buses company-provided transport to and from work during the outage, says Deepak Khosla, NIIT’s global head of sales and marketing.

The fact that Indian IT service providers have diversified their operations geographically helps them to manage the risk of outages in a particular location, says Lepeak. “Localized issues can be mitigated with out-of-region or out-of-area support,” says ISG’s Young. “[But] widespread problems make that more impractical.”

The outages have highlighted the importance of further investment in India’s power grid and India’s dependence on imported coal and oil, says Hall of ISG. But the infrastructure risk due to power failure will remain an issue for the foreseeable future, according to Alan Hanson, senior vice president of Neo Group.

“There are some cities in India where power is in short supply on a forward basis—not as a result of these events—[and] that may factor into operations planning,” says Hanson. Frequent power cuts cripple daily business operations in Hyderabad, for example, and commercial power charges continue to increase facility costs, according to Neo Group’s quarterly Global Supply Risk Monitor Report.

China faces similar power shortages. The country will have a power shortfall of 30 to 40 million kilowatts this year, according to the China Electricity Council (CEC), which will be reflected in regional, temporal and seasonal failures. Meanwhile, the CEC predicts a sharp drop in thermal power investment due to the low margins of thermal power projects and China’s efforts to lower carbon emissions. The drastic drop in power supply is the primary driver for increase infrastructure risk for those with IT operations in China, according to the Neo Group report.

Back in India, power had been restored as of Wednesday, Aug. 1, with seemingly little impact on IT outsourcing customers. “Over the short term, the impact may be greater on Bollywood shooting schedules [than IT service delivery],” says Neo Group’s Hanson. “But the good news is that it causes buyers to ask the right questions.”

3 Ways to Protect Yourself From Power Failures

Savvy outsourcing customers should take steps to ensure they are protected from power failures, whether outsourcing to Chennai, Charlotte or Changzhou.

1. Vet your provider’s plan. Never rely solely on what the vendor says, says Lepeak of KPMG. Do your own due diligence using skilled professionals. Look for failover plans that include a separate power grid, says Hanson of New Group, adding, “always question the feasibility of plans that require resources to, for example, travel and fly to an alternate location and whether that location&mdashif reachable—would be operable.” Make site visits&mdashnot just to the main facility, but also any backup operations.

2. Put it in writing. “When push comes to shove, providers will take care of their biggest, highest-value customers and the ones with the most carefully negotiated contracts first,” says Slaby of HfS Research. “If you’re not one of those, you may feel the sting of disruptions in India’s critical infrastructure from halfway around the world.”

How-To: Negotiate a Contract with a Cloud or SaaS Provider

Push for specific recovery-point and recovery-time objectives, penalties for business continuity failures, an indemnity figure at least 10 times the total contract value and the ability to exit the contract without penalty. “If your provider’s operations prove to be brittle in the wake of some large-scale critical infrastructure outage, you may be more interested in finding another provider than getting more service credits from your current one,” Slaby says.

3. Test and retest. Make sure there is not only redundancy in the provider’s systems, says Hall, but also insist on quarterly tests of internal power generation capabilities.

Stephanie Overby is a regular contributor to CIO.com’s IT Outsourcing section. Follow everything from CIO.com on Twitter @CIOonline, on Facebook, and on Google +.