Why CIOs need to worry about Chennai

I come from the beautiful city of Chennai in the south Indian state of Tamil Nadu.

Many of you would have heard of the natural disaster that has shut down the city of Chennai this past week – besides being my hometown, it is also a leading hub of offshore IT services where global firms in America and elsewhere have extended teams providing development and infrastructure support. Given my unique perspective on the situation as a long-time technology professional with deep connections to the Indian IT sector, I thought I would share this post with you.

For the past few weeks, Chennai has been battered by relentless rains and flooding—the worst in over a hundred years. The city has been under water for many days. Power supply has been shut off due to safety concerns. Phone lines are down. Schools and offices are closed. The venerable local newspaper, The Hindu, failed to publish the daily edition for the first time in its history since it was launched in 1878. The airport is closed, and 3,500 people are stuck trying to get out. The common mode of transport in Chennai today is by boat. Yes, boat.

My immediate family members are either marooned in their homes with no power and telecommunication or have moved in as guests in high-rise condos belonging to friends and neighbors.

It continues to rain. There’s no respite expected for several more days. The government has declared Chennai a disaster zone, and the army has been deployed to help thousands of stranded citizens.

What does all this have to do with you if you’re a chief information officer (CIO) in an American company?

Plenty, if you have chosen to outsource IT operations to a firm that is managing your data centre operations and applications from Chennai, one of the largest hubs of offshore IT services from India.

Over the past couple of decades, large American firms have been outsourcing IT support and maintenance to tech firms that run operations out of India. The business case was always good—low costs, capable talent, and reliable infrastructure. I have personally signed and managed these contracts for clients, and have led large global teams from Chennai and other locations to deliver on the business case and cost savings.

But I’ve never experienced a disaster of this nature. And I don’t know a single CIO who has, either.

Newspaper reports indicate offshore providers have kicked in with contingency plans and have been flying out associates from Chennai to other locations to try and maintain service levels. But with airports and railway stations shut now, and no way to get in or out of the city, CIOs can only keep their fingers crossed and hope for the best. Indian firms, rightly, are focusing on the safety of their employees. The impact of these floods could go far beyond Chennai, and potentially touch our daily lives in the US and elsewhere.

Besides the immediate human costs of this disaster, the impact could go far beyond Chennai, and potentially touch our daily lives in the US and elsewhere. For example, if a major clinical application for a hospital system in a data centre supported remotely by software engineers in Chennai goes down, patient lives in the US could be at risk. We wouldn’t know till something like this actually happens, and we hope it doesn’t. However, rains are expected to continue for another two weeks in Chennai, so we have to keep our fingers crossed.

Let’s take a quick look at how large outsourcing contracts address these types of situations.

Every large outsourcing contract has a provision for disaster recovery (DR) and business continuity plans (BCP).

Typically, this is an exhibit at the very bottom of a 250-page MSA that contract negotiators get to when they are completely worn out from all the wrestling with more important commercial and service-level discussions. The DR/BCP contract language typically addresses the most critical applications, with agreement on both sides to identify alternate DR/BCP sites, document DR/BCP procedures, and update them from time to time. There is also usually an agreement to do DR simulation testing, like a routine fire drill, however whether these procedures are actually followed is a different matter.

CIOs have to ask themselves how their staff, sitting in the US, could possibly appreciate the extent of the problem in Chennai, especially if they have never been to the site.

Most organisations maintain a very thin retained staff with the skills required to manage IT operations (after all, that is part of the business case), and prefer to cut back on travel budgets than to commit to an annual site visit to their offshore provider’s facilities. The result is that IT staff are clueless about how to respond to a situation in faraway India, and have to rely on their service providers to do their best. Since they have never been there, they fail to appreciate the ground-level situation, and the living and working conditions of their extended IT support staff on a normal day, leave alone on a rainy day.

Back in December 2004, Chennai was hit by a tsunami that hit many other countries in southeast Asia, including Indonesia, Thailand, and Sri Lanka. I happened to be in Chennai that day, and what I remember most was the disbelief among people who never imagined that the Bay of Bengal could rise one morning and sweep away several thousand people into the sea in a matter of minutes.

The tsunami of 2004 gave very little warning if any. Since then, early warning systems have been set up all across the Pacific Rim to prevent another tsunami from taking us by surprise. The torrential downpour that’s brought Chennai to a standstill today was a known disaster in the making, yet there was very little the city could have done to stop it.

When the waters finally subside—and I pray for all my friends and family in Chennai that it happens soon—CIOs in the western world will have to step back and assess their own early warning systems, assuming they have made it through this in one piece.

This post appeared first on Linked In