Failure and What You Can Learn from It

Don Goldstein learned a classic lesson when he joined an insurance company in the mid-1990s as vice president of IT. He inherited a data warehouse project intended to mesh information across divisions to provide a complete look at each customer.

More on CIO.com

What I Wish I Knew Before I Became CIO

Puckett Finishes 100 Days at Dupont

Why CIOs Need to Focus Their Attention on Growth

To Avoid Failure, First Define Success

“This would be the thing that finally gave everyone what they had been looking for,” Goldstein recalls. “Namely, how do we get all of our information to mean something?”

But too much data needed too much cleansing, he says, and the project plan allowed many months between deliverables. Along the way, business groups argued about who should update which information.

“It just got way, way out of hand,” he says. “It cost too much, took too much time. It didn’t have a hope to work.”

A data warehouse was finished, but for just one business unit. Users still had to ask IT to generate their reports. The project died soon after being delivered to the business unit, when key project members left and funding dried up.

The lesson he learned, of course, is that you can’t boil an ocean. Ever after, says Goldstein, who is now CIO of CB Richard Ellis, a $5 billion commercial real estate services company, his projects have had narrow and specific milestones managed by a group of IT and business people, each accountable for different aspects of the work.

“It seems simplistic, looking back,” he says. But the failure so affected him that he avoids the term “data warehouse.” “So ominous,” he says. You can almost see him shiver.

How you respond to failure—or any event that doesn’t go as expected—shapes how you handle the next one. Early in her career, Twila Day, CIO of the $37.5 billion Sysco food service company, had to find the skills and inner strength to handle a negative boss. She’s since applied the lesson to deal with negative people generally. “You learn more when something goes wrong than when everything goes right,” she says.

Surviving failure not only teaches lessons but builds resiliency—a trait critical to handling the uncertainty we face today. In this economy, which yet shows no signs of great and lasting recovery, you want tested leaders cool enough to handle difficult, unpredictable days. Someone who has never failed is half as valuable as someone who has endured a “humbling experience” and learned from it, says John Puckett, DuPont’s CTO of corporate IT and CIO of central research and development. Puckett’s 42-year career includes the CIO post at Toysmart.com (a casualty of the dotcom bust) as well as vice president and general manager at Polaroid (whose business model was overtaken by the rise of digital cameras). “You better hope your leaders have failures under their belts,” he warns.

But failing now, however educational, could put your job and even your company at risk. With recent cuts in budgets and staff, choosing to do one project may mean choosing not to do another; mess it up and you let your company down twice.

Meanwhile, in some circumstances, failure may be beyond your control, yet you may have to clean it up anyway, notes Robert Fort, CIO of Virgin Megastores North America. The company is closing all of its U.S. music stores because renting the real estate they occupy can generate more money for its owner than the retail outlets can, Fort says. Learning how to shut down a collapsed company is a new experience for him, he says. “I have the desire to do this with integrity.”

He knows that how he handles failure will determine his future as a CIO.

Read Your People Carefully

Faulty management of a PeopleSoft Financials project in 1997 is a mistake Steve Hanna can’t forget. At the time, he was CIO of Host Marriott Services, a company that supplied concessions to airports and other venues.

To lead the implementation, Hanna and the company controller chose two up-and-comers: one from IT and one from the controller’s staff. The team hired a good integrator to help, they held weekly status meetings and everything appeared to be going right, Hanna recalls.

But underneath, much was wrong. The day a critical piece was due, the duo told Hanna it wouldn’t come in. “I asked, ‘How far away are we?’ They said, ‘We don’t know.'”

IT failures, like plane crashes, typically happen after a sequence of small problems culminate into a large, visible breakdown, says Michael Krigsman, president of Asuret, a consultancy that specializes in helping companies avoid project failures. “Usually there was a triggering event and multiple other steps along the way that also failed,” he says.

Even after Hanna and the two managers spent the next weekend dissecting their project documents and reports, as well as which code and modules had been developed, they couldn’t figure out what had happened. “We ended up redoing the entire project.”

All along, it turns out, the project managers hadn’t disclosed problems. They’d hint at trouble, saying, for example, that they were late testing a piece of code but would make up the time on the weekend. But Hanna, who is now CIO at Kennametal, missed the warnings. He says he should have sniffed out what they weren’t saying by asking probing questions and watching body language.

“When you have executives trying to make a name for themselves, you have to fly low to the ground with them,” he says. “It was probably the biggest mistake I’ve made in my career.”

Sometimes you think you’ve put the right people in place only to find out you misjudged them, says Christopher Barron, SVP and CIO of CPS Energy, a $2.2 billion energy and utilities company.

Last year, CPS Energy wanted to build a mobile application portfolio to give executives and key staff members access to corporate financial and operational data via smartphones and laptops. Barron assigned a tech whiz to the project who alienated the company’s senior executives with his arrogance.

“This person was unwilling to admit there were kinks that needed to be tuned and had burned relationships with people,” he says. The project failed, Barron says, but not the effort. The company learned which technologies worked and has now put the application in production.

Barron also learned that he needed a different kind of project leader. He reassigned the techie to an R&D position, from which he hands off implementation to someone else. To interact with the company’s CEO, CFO and chief risk officer on the new system, Barron tapped top desktop and network support staff specially trained to work with business executives. These staff members have undergone advanced conflict resolution training and learned how to address executives respectfully. They’ve also been taught to use body language to appear approachable, and they even wear a uniform: slacks and a company logo shirt. “Jeans and an untucked shirt don’t give you gravitas with executive customers,” Barron says.

Being upfront about his missteps helped Barron convince his C-level peers to go ahead with a new iteration of the project, he says. Owning mistakes enhances a CIO’s reputation, he says.

In Hanna’s case, he admits he picked inexperienced project leaders and didn’t watch them closely enough to “hear the unspoken.” Now he asks for actual—versus expected—results, supported by details and evidence. He has also sharpened his senses. “I wander around and talk to people in general, not just about the project, and see if they laugh or avoid eye contact,” he says. A good CIO can “pick up more in a five-minute conversation at the coffee station than at an all-day project debriefing.”

When that PeopleSoft project went bad, Hanna had to tell his boss­—the CEO—that it would be months late and cost “a lot more money” than budgeted. The project managers, he says, soon quit to consult. Hanna, meanwhile, lost a bonus riding on the project’s success. The CEO noted the events in a performance review. “His comment was, ‘I hired you as a thought leader and you let me down,'” Hanna recalls. “It cut really deep.”

Know When to Quit

If taking failure personally burnishes the lessons, taking it too personally destroys the chance to learn. In a dysfunctional environment, such as in a department led by a bad boss, it can be hard not to succumb, says Day, Sysco’s CIO. She was laid off as a young programmer. Without the finances to be picky, she resolved to take the first offer she got. Her new boss apparently thought little of her.

“He didn’t believe females should be in IT,” she says. “He would say, ‘You can’t do this job, you’re not smart enough.'” At first, she disliked going to work: “A negative, almost depressing situation,” she recalls. Then she decided to disprove his barbs, absorb as much knowledge as she could and get out. Among the lessons she learned, she says, was not to waste energy trying to change other people. It’s best to learn as much from a bad situation as you can and move on.

Quitting on a project, however, can be more difficult because afterward, you’re stuck amid the financial, political and personal consequences.

As a result, the wherewithal to stop a project often doesn’t exist, says consultant Krigsman. He was recently hired by a client to evaluate a troubled project. In a meeting with the IT and business staff, the sponsor—an executive vice president—asked when the team would achieve a given milestone. In these situations, denial and fear often prevent changing direction until after the project has become a public spectacle.

“He looked around and there was silence,” he recalls. “I knew there was no way on God’s green earth this milestone could be achieved in any reasonable time frame.” The sponsor called on Krigsman, who told him the bad news.

“When the team is scared to tell management the truth and feels that their jobs may be in jeopardy, the conditions have been created for denial to blossom, and failure is almost assured,” he says.

John Halamka, CIO of Beth Israel Deaconess Medical Center, tries to prevent a punishing atmosphere that could spur staff to hide mistakes. During a major network outage in 2002, doctors and lab technicians couldn’t access patient charts or test results and 100 temporary workers had to hand-carry records around campus. Yet no one pointed fingers at specific people.

Rather, Halamka and his staff examined which processes and technology allowed the problem to develop. They learned they needed to institute thorough downtime procedures and invest $2 million to upgrade networking gear and, later, $5 million to build redundant data centers.

Looking at trouble as a process failure and not the fault of one person “gives us the freedom to diagnose and treat ourselves,” he says. “We become very transparent about what our organization’s strengths and weaknesses are.”

CB Richard Ellis’s Goldstein says IT managers must stop wayward projects before they cause damage, but no one likes to give up.

“Sometimes it’s easier to try to keep momentum going, try to get more funding and resources,” he says. “You assemble a team, a budget, what you believe are achievable objectives. You make commitments. The last thing you want to do is not live up to them.” (See “To Avoid Failure, Define Success”)

One way he now guards against the pressure to keep going in the face of sure failure is by building a project team from a variety of business sponsors, end users and IT staff. Included are a couple of managers with the rank to kill a project. A subject matter expert at the vice president level or higher, he says, has the pull to refocus or even stop work that goes off the rails or no longer makes sense when business goals change.

“It takes both courage and some political capital to stand up and say, ‘We’ve made a mistake.'”

Understand the Risks

Scientist, inventor and statesman Benjamin Franklin might be history’s most successful mistake maker. He was clonked on the head by the whirling blades of a fan attached to a rocking chair he built and exhausted himself wearing his own wooden swim paddles. Franklin’s self-cooling rocker never took off and the paddles were too heavy for swimming. But his flying a kite in a thunderstorm led to harnessing electricity, and his help orchestrating the American Revolution turned out okay.

In all, he amassed an impressive record in his 84 years and advised, “Do not fear mistakes. You will know failure. Continue to reach out.”

Easy for Ben to say; no one ever laid him off.

Still, says Barron, if you’re not making fruitful mistakes, you’re not taking enough risk. Risk, after all, brings progress. Barron’s project to create mobile, roaming, overseas access to the company’s financial data is one example. “We threw away six figures but we learned exactly what we needed and we could buy exactly what we needed when it was available,” he says. The company built its now-working system on NetMotion’s mobile VPN software and Cisco’s Soft Token authentication product.

“I have a couple kids and don’t want to be out on the street. But I’m not doing my job if I don’t experiment,” he says.

Yet evaluating risk with a backdrop of multitrillion-dollar bailouts and the biggest unemployment numbers in two decades takes special focus, says Beth Israel’s Halamka.

For example, infrastructure upgrades often must be done regardless of the economy, to keep IT running smoothly, he says. In discussions with his peers and boards of directors, he multiplies the odds that something bad will occur by the consequences of the bad thing. Had he done that in 2002, he says, he might have determined that a six-year-old network infrastructure could buckle under growing network demands. Then he’d have known that medical records would be inaccessible, at the potential cost to patient lives. “No patients were harmed. But it was a wake-up call to show how dependent we all [are on] technology,” he says.

With a mandate to cut expenses, some CIOs are investigating new technologies that promise cost savings. Cloud computing—consuming technology over the Web as a service from a vendor rather than running it in-house—may be untried for many applications, but the risk is worth the reward of cheaper IT, says Jerry Hodge, senior director for information services of small kitchen appliance distributor Hamilton Beach Brands.

Hodge is replacing Lotus Notes e-mail with Google’s Gmail for 500 employees. He expects to save close to 60 percent of his current messaging costs over five years, he says. “We tend to be conservative and ask a lot of questions. We can’t afford a mistake,” he says. The projected savings, plus the fact that he can reassign some of his staff from managing e-mail to more creative and important jobs such as business intelligence, convinced him Gmail was a risk worth taking.

It’s Not Always Your Fault

No matter how risk averse you are or how much experience you have dealing with adversity, you may still encounter failure beyond your control. The collapse of a company is the kind of failure that hurts—and teaches—on many levels.

Art Hall was vice president of customer care operations with NetBank, an Internet-only bank that went under because of the housing and liquidity crisis. At its launch in 1996, the startup raced ahead of stalwarts such as Bank of America and Wachovia to offer customers online access to checking, savings and other accounts. “People worked 60 to 70 hours a week, living the brand, believing in the company and cascading that passion to everyone else,” says Hall, who is now a management consultant at Alvarez and Marsal.

But established banks caught on to the Web and NetBank stagnated. “No one anticipated, or admitted, that the average consumer wouldn’t take everything out of a known branch and put it in a virtual bank,” he says.

In May 2007, EverBank announced plans to acquire the NetBank, but the deal fizzled. ING bought the accounts of NetBank’s 104,000 customers in September 2007, and NetBank filed for Chapter 11 bankruptcy protection a month later. It never emerged. Before NetBank, Hall had been laid off from First Data and he drew upon what he learned from that experience. He relieved pressure by talking with a mentor outside of NetBank, praying and resting more on weekends and advised coworkers to find their own ways to cope. Such outlets are important, he says, “so there’s a forum to get out some of the emotions in a safe environment, and you’re not walking around as a ticking time bomb.” The lesson came home to Hall with the suicide of a colleague. “He took his life on his parents’ front lawn,” he says quietly. It happened as the final 10-week push began to close NetBank. “That was the darkest day.”

Those who have experienced failure say that in the middle of it, events move fitfully and whatever learning there is may not be immediately clear.

Robert Fort is riding that roller coaster while he shuts down the remaining six Virgin Megastores in the United States. The music retailer was sold to a partnership of two New York-based real estate firms, including Related Companies, in late 2007. “We’ve known for over a year we were fighting for our survival,” Fort says.

He and other officers kept employees informed of “the realities,” he says, referring to the speculation that the music chain faced long odds. In March, when the partnership decided to close the U.S. stores, staff morale actually spiked temporarily, he says. “There’s a strange emotional boost you get when things are made clear.”

After a few days, though, spirits sank as people thought about medical plans and made severance calculations. Now, he says, judging from the Facebook and Twitter posts of employees, the mood has changed again. “We’ve noticed the story shifting away from bummer to nostalgia. There’s a big demographic that grew up on CDs, and we’re an iconic part of that.”

CEO Simon Wright put Fort in charge of the internal, national wind-down. Fort expects to be out of a job by the fall. Next year, more lucrative tenants will begin moving into the former Megastores, including the massive space in New York’s Times Square. Planning and executing the shut down while managing his emotions and those of his staff (now down to seven) is new territory.

As for his future, Fort has considered everything from taking the summer off to consulting to seeking another full-time CIO position. But it’s hard to make a decision when there is so much work to be done.

He’s a bit frustrated, too, because, he says, Virgin Mega-stores recorded record financials for the past two years. They added new technology in the stores, such as a real-time data warehouse and business intelligence system, sampling kiosks, and a converged voice and data network to differentiate from the competition.

But the sinking economy outpaced that good work. “I know that IT alone won’t make a company,” he says. “But we have a lot of pride.”

Do you Tweet? Follow me on Twitter @knash99. Follow everything from CIO Magazine on Twitter @CIOMagazine.