by Mike Anderson

What It’s Like To…Walk Into An IT Disaster

Dec 15, 20044 mins
IT Leadership

Everywhere I go, IT is the butt of jokes.

“You want something from IT, it’s either $1 million or 12 months.”

Or, “I’d call the help desk but it’d be four days before they’d get here.”

I’m not surprised to hear the cracks. I’m a turnaround CIO. CEOs hire me to clean up their IT messes. But what is surprising is that the biggest IT disasters have been ones where the CEO didn’t even realize there was a problem. Or didn’t realize how easily it could be fixed.

The worst situation I ever saw was at a company that had grown dramatically during the dotcom era. Half the IT staff were contractors who had worked there upwards of a decade. The company was paying them twice what they would have paid in-house employees. The mainframes went down three or four times a day, the warehouse software was buggy, financial applications didn’t work and the ERP system was a monster to maintain. And, of course, there was a giant backlog of IT projects.

There was a lot to do, and a new CIO has a very short honeymoon. You have to deliver some kind of success right away.

In this case, I had to provide some stability, stat. I brought in the mainframe vendor and said, Your system goes down every day. Fix it. They dug in and discovered that a lot of the hardware—the boards, the memory, the firmware—wasn’t up to date. Turned out the former IT director didn’t have any maintenance contracts. In addition, the air-conditioner on the roof would fail periodically. In the summer, the system would crash; the systems manager would discover it was the A/C and call maintenance; then the manager would crank up the cold air. Of course, doing that is like taking a wineglass out of a hot dishwasher and filling it with ice water. Crack!

So we updated the hardware, put maintenance contracts in place and fixed the A/C. It was commonsense stuff, but it was important to get IT out of at least one doghouse in the first 30 days.

Then it was time to exterminate the warehouse bugs. I asked the IT managers what would happen when the system failed. They said, We get Jose. When the warehouse guys get an error message, Jose restarts the program and fixes the data. I asked, How long does that take? They said, All day.

I asked Jose if he knew what the problem was. He handed me five pages of notes and said if he had two weeks, he could fix it so it would never go down again. I had Jose teach some contractors how to restart the application and fix the data and gave him the time he needed.

Financials were next. Only the IT people knew how to close the books. The accountants would do their work, show up at the end of the month, and ask, Would you guys close the books for us? And IT did. Of course, IT had no idea if the figures would reconcile.

So we taught the accountants how to close. They didn’t want to learn, but I appealed to the CFO. I said, We don’t want you doing our system backups; you don’t want us doing your books.

After that, the contractors had to go. I called the CEO of the consulting company. I said, I need your guys out within the year. He said, I’ve been waiting for you to realize how much money you’ve been wasting on us. So as projects finished, we rolled the consultants off them, gave them 30 days to do knowledge transfers, and had them all out by end of year, reducing staffing costs by a third.

Then there was the ERP problem—customized to the point of uselessness. We put out an RFP and got started on implementing a new system.

Finally, I created a project management office with business stakeholders to begin dealing with the backlog.

Three years after I got there, things were stabilized. It wasn’t rocket science. It rarely is.

There were still cracks about the IT department. But the real problems had been solved. I just smiled and moved on.

—As told to Stephanie Overby