Bad Decisions

The Purloined Letters: PlusNet

We’ve all lost an important e-mail message or two. A casual click of the delete button. A misconfigured office server. It happens. But 700GB of e-mail? Gone for eternity? It happened. In 2006, British ISP PlusNet lost 700GB of customer e-mails forever. An engineer accidentally deleted the e-mails and then tried an old admin trick to retrieve them. The trick backfired, and instead made the mail irretrievable, despite PlusNet’s efforts.

“I would like to take this opportunity to once again extend our sincerest apologies for the inconvenience caused by the loss of any customer emails and in particular the unread emails. This has been taken very seriously within PlusNet. An extensive analysis of how this happened has been carried out and working practices implemented to prevent it happening again.” — Alistair Wyse, technical director, in a PlusNet press release explaining what happened

Whiplash: JPMorgan

They say you can’t go home again. JPMorgan Chase proved it sure is hard, anyway. Consider its fate before you jump on a multibillion-dollar outsourcing deal. In 2002, JPMorgan’s $5 billion contract with IBM for IT support was the largest on record, and received widespread publicity as the wave of the future for cutting costs and increasing innovation. But after JPMorgan was acquired by Bank One, it soon became clear that the deal was no bargain. In an embarrassing turn, much of what had been outsourced was brought back in-house in 2005. The cost was huge, in dollars as well as in fractured employee trust and ruined morale, lost talent and years of management time.

“Bringing outsourced work back in-house can cause such disruption to an organization that most people don’t do it. It’s a very difficult and painful change for an organization to go through.” — Ralph Schonenbach, CEO of the Trestle Group, an outsourcing consultancy

Nightmare Before Christmas: Comair

Since at least 1997, Comair knew it needed to replace a creaky old flight-crew management system. The options for new systems were rather raw and unproven. “Let’s wait til something better comes along,” they said. Here’s where the foreshadowing creepy music begins. They kept waiting, postponing, nursing the legacy system along, and finally signed a deal to replace the whole thing in 2005. Too late. It gave up the ghost with a dramatic failure on Christmas Eve 2004. It brought down the entire airline, canceling or delaying 3,900 flights and stranding nearly 200,000 passengers. The network crash cost Comair and its parent company, Delta Air Lines, $20 million, damaged the airline’s reputation and prompted an investigation by the Department of Transportation

“Anything that can damage a parent company’s brand or reputation has to be managed in some way. Risk assessment of worst-case scenarios at Comair should have happened at Delta.” — Eric Bardes, former Comair executive of the airline’s IT department

All Systems Down: CareGroup Health

CareGroup had developed a well-deserved reputation for being a leader in using information technology to drive improvements in health care. But in November 2002, CareGroup’s flagship hospitalBeth Israel Deaconesswas driven back to the dark ages of paper-based patient records by a debilitating series of network crashes. Over five days, the IT department for Beth Israel would frantically try to track down the cause of the problem.

“I made a mistake. And the way I can fix that is to tell everybody what happened so they can avoid this.” — CareGroup CIO John Halamka

Black Monday: New York Stock Exchange

Better living through technology. Yes, that’s what we’re after. But in the “Black Monday” stock market crash on Oct. 19, 1987, technology made life much worse. The Dow Jones average fell by 508 points, or 22.6 percent. Normal market conditions had set off the downturn, but a Securities and Exchange Commission report later noted that computer-directed strategies used by institutional investors trading large volumes of stock accounted for up to 68 percent of New York Stock Exchange trades at times during the Black Monday debacle. The day ended with an hourlong panic sell-off.

“I think we can say with considerable certitude that the crash was not caused by fundamental news about the economy, either in the U.S. or in other countries.” — Mark Rubinstein, then the Paul Stephens professor of applied investment analysis at the University of California at Berkeley, in an article that appeared in “Risks in Accumulation Products,” Society of Actuaries, 2000

“What did we miss? Nominate your own Worst IT Disaster by sending it to Chris Lindquist at CIO.com.”