Outsourcing: What You Can Do If Your Security Vendor Fails

1 2 Page 2
Page 2 of 2

"We’re not interested in breaking in new security vendors. I want to see Wall Street firms and large banks on their customer list. My ideal would be a large, funded company with diversified resources," he says. His last requirement is the tricky part?an outsourcing partner has to be "one that’s also highly competent." While the expertise still resides in the boutiques, the CIO anticipates that large general service companies will start bailing out the smaller companies. That way, they acquire the smarts, they have steady bottom lines, and they make security a component of larger managed services packages. And indeed, AT&T was ready to buy Pilot but walked away at the last moment, several ex-Pilot sources and customers say. Symantec has already bought a boutique company, Axent.

If this expertise-through-acquisition scenario plays out as such, CIOs will have the best of both worlds?stable business and expertise. But that presents other challenges. For example, Symantec has products to sell. Partnering with Symantec likely means partnering with Symantec’s products too. And service levels may drop as the smaller boutiques are subsumed by larger companies.

But for the health-care CIO, less service and expertise is fine. Outages are not.

Three weeks after the incident, with his contingency up and running, he says, "We dodged a bullet."

Taking It In-House

NEIL HENNESSY, VICE PRESIDENT OF IT engineering for PeopleSoft, learned about Pilot’s collapse in a most unusual way. At the end of a weekly meeting with his Pilot rep, the man announced, "I have to go back to the office and get fired at 4:30." And in the week leading up to Pilot’s bankruptcy filing, Hennessy says he was fending off a wake of vultures.

"One guy calls from Southern California, and he’s telling me how he can offer me everything Pilot did for less money," Hennessy says. "So I ask him, ’How many employees do you have?’ and he tells me 40. So I said to him, ’Pilot had 400. Why would I trust you?’"

Truthfully, Hennessy had started to lose faith in Pilot beginning a year before its collapse. He was particularly worried about the company’s scalability. "They just couldn’t step up. Not that they didn’t try. Their model was very secure, but we started looking at other options back then," he says.

Hennessy’s favored alternative was to phase out his managed security contract and take the task back in-house. This, after all, was the year of uber-viruses and broad, destructive hacks. Hennessy decided that security was just too critical to outsource.

His transition plan meshed with his contingency plan. Hennessy already had a backup carrier with a "dark" data line, one that’s not turned on but could be activated in an emergency. And he started building an internal security staff of five with five more to come.

So when his Pilot rep told Hennessy he was going to get fired that afternoon, Hennessy was able to set the plan in motion, and the transition to in-house 24/7 security was done in five hours. He credits the quick shift to his engineering team, whom he ranks somewhere between "real strong" and "the best in the world."

The cost of doing it all in-house was and will continue to be massive, of course. Hennessy won’t deign to put a number on it, but he readily accepts the fact that he’s paying a premium for in-house security. "It’s definitely far more expensive doing it in-house," he says. "On the other hand, there’s far less risk. I’m paying to sleep well."

Why is it more expensive? To begin with, recruiting talent is hard. There’s little out there, and there are plenty of posers. Some experts put the ratio at about one real expert for every 10 claiming expertise. Certifications are partly to blame. A rŽsumŽ with a dozen security certifications might look impressive, but it’s misleading. Some certifications are simply for specific products and teach nothing about best practices or security policy. A firewall "expert" might know how to configure the box but have no knowledge of what policies should be enforced or even where the firewall ought to be placed in the context of a specific network.

Paying talent sufficiently is even harder than finding it. Stephen Northcutt, founder of the Global Incident Analysis Center and security consultant, says security contractors demand up to $500 per hour. Salaries are 5 percent to 10 percent higher than what standard IT staff earn.

Keeping talent is the hardest task of all. Northcutt says many true security experts are hopping jobs six times a year, upping their salaries $5,000 at each post. Len Cibelli, a former sales executive at Pilot, expects to get a 20 percent raise from his next employer.

Even so, Hennessy is convinced of the rightness of his decision. "We know doing it in-house is more expensive, but we’ve just decided it’s better than outsourcing," he says. While talent is thin, Hennessy says a few strong candidates have come his way due to the economy.

The Hartford Financial Services Group, which was not a Pilot customer, has taken many of the same steps as Hennessy. Hartford Assistant Vice President of IT Jack Stoddard outsources little security, only ceding tasks such as auditing and penetration assessments to outside vendors. He retains 30 full-time security staff members, tries to recruit the best he can find, pays premiums for them and trains his staff continually. He is adamant about the limitations of the outsourcing model.

"I don’t see us ever outsourcing," Stoddard says. His CIO, David Annis, believes acquiring and grooming security expertise in-house is critical, even if it costs more. He calls outsourcing "throwing in the towel." But he understands why so many companies do it anyway. Security is so complex and demands such constant reassessments, he says, that doing it in-house requires a "fair amount of redundant due diligence."

Postscript

ON MAY 9, EXACTLY TWO WEEKS after Pilot disbanded, it was liquidated. There was a Hail Mary as several managed security vendors tried to take over the business, but that collapsed. Emergency operations and support were halted. AT&T finally cut the circuits, and Brown at VisionTek received a page. VisionTek was still waiting for the local carrier to supply a data line, so Brown boxed up his equipment and drove it to its temporary home in a downtown Chicago facility.

On the same day, Pilot’s homepage finally changed its cheerful, "Yes, we’re open" message. "Pilot Networks has filed for bankruptcy" was all it said. There were some snippy redesigns of the Pilot website, obviously tacked up by bitter ex-employees. The title bar of Pilot.net read: "Pilot Network Services is now Imaginary Network Services Inc."

Someone left behind a sarcastic note, which only hinted at what had gone on. Anyone who happened by and clicked on "What’s new" would see the note: "Here is the latest about Pilot: We’re done! Pilot is no more. This company is an ex-secure ISP. If it weren’t for being nailed to the perch, it would fall over. Alameda, CA, May 9, 2001."

You can almost hear creaky saloon doors rattling in the wind and tumbleweed staggering through the dust.

Related:

Copyright © 2001 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
Survey says! Share your insights in our 19th annual State of the CIO study