Outsourcing: What You Can Do If Your Security Vendor Fails

ON APRIL 25, PILOT NETWORK SERVICES went out of business, abandoning 200 customers that relied on them for something rather important: security. There had long been signs of Pilot’s distress. Customers had recently reported spotty service from the managed security company. Pilot’s stock, once at $50, had plummeted to 21 cents per share, and Nasdaq delisted it. Yet this was not some high-flying dotcom that appeared one day, took some easy venture capital, then vanished. Pilot was an established, 8-year-old vendor with 400 employees and, by most accounts, superior security technology and practices. Its customers included PeopleSoft, VisionTek, The Washington Post Co. and several large health-care institutions and banks.

Despite all that, the end came quickly. Pilot employees received four e-mails in rapid succession. The first said the phones would be disconnected. The second added that pagers and mobiles would be taken away. The third said the CFO had resigned. And for anyone who couldn’t see the elephant?not just in the room but squirting river water in their faces?the last e-mail said, “At 4:30 p.m., you’re fired.”

Pilot did keep a skeleton crew to manage customers’ security through the data lines. Responding to desperate pleas from Pilot customers, AT&T suspended the order and kept Pilot’s operations center connected, even though it wasn’t getting paid.

With no one watching their networks and an outage threatening at any moment, Pilot customers felt naked. They were suddenly wide open to hackers and viruses. Because some companies routed office-to-office traffic through Pilot, they were at risk of losing secure virtual private network (VPN) connections and remote access. Pilot had hosted entire Web networks for other companies, making them even more vulnerable to a complete meltdown.

One such company, Providian Financial, was so distressed that it sent several IT staffers to man Pilot’s operations center. That probably frightened Pilot’s other banking customers, none of whom were expecting a competitive financial institution to have access to their network security.

While it’s perhaps the most dire example of failure in the slowing economy, Pilot’s breakdown is not an aberration. Other managed security companies are hurting too. The Salinas Group had already folded. Exodus endured an atrocious first quarter in which CEO Ellen Hancock said everything was on track “except revenue.” Recently, Exodus and another managed security company, Counterpane, announced that they are joining forces for efficiencies of scale. MyCIO.com, once independently operated, was folded back into its parent company, McAfee. Two other managed service boutiques, Vigilante and Networks Vigilance, have merged. “Spending has tapered,” says Bruce Murphy, CEO of Vigilinx, another managed security company. “A billion dollars in equity just dried up.”

In a matter of days, the managed security services option turned into a frightening one for CIOs. Until now, outsourcing security management to a boutique company like Pilot seemed the best way to go for two reasons: One, that’s where the most cutting-edge security expertise had migrated, and two, doing security in-house was considered too expensive and difficult for most companies. But in the wake of the Pilot disaster, many CIOs are reevaluating two alternatives: outsourcing their security needs to a large, general services company such as IBM Global Services, or taking care of them in-house.

The problem is that none of the three available options is the clear winner. Each carries significant risk, and former Pilot customers are trying them all. But all of them agree on one point. Outsourcing security is more work than just writing a check every month. It’s a full-time job that requires in-house resources. Treating it as any less?and many do?is playing Russian roulette with the entire enterprise.

Small Is Better?

ONE WEEK AFTER THE IMPLOSION, Pilot filed Chapter 7 in Oakland Bankruptcy Court. Its website, The Pilot.net, made no mention of the company’s troubles. In fact, the site looked exactly the same as it had before the collapse. It had an eerie feel, like some Western ghost town.

Pilot’s outage couldn’t have come at a worse time for Ann Marie Durso, CIO of VisionTek, a memory and graphics card company in Gurnee, Ill. She had joined the company in October 2000 and was in the thick of a strategic ERP project that will help the company launch online retail sales. An outage would mean revenue losses on online sales, and each day without a secure, high-speed connection would add several days to the ERP project.

VisionTek has subscribed to Pilot for four years. Like a marriage, the partners just got comfortable talking less. Security was assumed, and just two months before Pilot went down, Durso had been baited with a renewal discount. Pilot offered to renew her contract at a cut rate if she paid for a full year up front. She did.

“We got blindsided,” she says. “We thought [that since] this was a provider that had been around since ’96 for us, there was less of an inclination for us to question them. But outsourcing isn’t an abdication. You can’t just hand it off. Ultimately, the business will hold me accountable, so I have to manage the third parties. I have to constantly ask, Are they still growing? Can they handle scale? Are they keeping their skills up?”

As soon as Durso heard about Pilot, she and her network manager, Mike Brown, went from office to office briefing VisionTek’s executives, one at a time, on what the collapse meant to the company.

“It wasn’t pleasant,” Durso recalls about the experience of having to break the news to the CEO, the CFO and the controller. Interviewed by CIO the day Pilot filed for Chapter 7, Durso was still frayed. “But we’re doing the right things. We had a full contingency in place in two days,” she says.

The contingency went something like this: First, get the executive staff’s permission to move forward on choosing alternative security providers. Second, create a worst-case plan. For VisionTek, this meant Brown put his pager on and never took it off.

Worst case, if AT&T cut the network connections to Pilot, Brown would be paged. He’d box up his servers and drive them from Gurnee to downtown Chicago, where an alternative provider had offered space and dial-up connections until VisionTek could find a full-time provider.

Next, VisionTek brought in two ex-Pilot engineers as contract consultants because they knew Durso’s security better than she did. In fact, the day after Pilot went down, VisionTek wasn’t sure of its security status because it had, over time, become Pilot’s responsibility to manage.

Together, the Pilot engineers and Durso figured out where they stood and got the network to a point where “we were at least able to limp along,” she says. With security patched together, Durso, Brown and the consultants turned their attention to evaluating other security vendors. Ironically, she wants a partner similar to Pilot in scope and methodology. Durso liked Pilot’s level of expertise. She liked its 24/7 monitoring. Finding another Pilot with stable financials is unlikely. But Durso knows larger companies often have less expertise.

Highly sought security talent flowed to the boutique companies for two reasons. First, top IT security experts?often from the military and government agencies such as the CIA?left public service in droves a few years ago to start their own companies. Subsequently, venture capitalists heard tales of Pentagon-level security, so there was plenty of money out there, until recently. Second, there was fraternal loyalty; security experts gravitate to companies run by their peers.

But the startup trend led to a glut. There were too many boutiques, and they were burning cash fast. That, in turn, led to aggressive selling, such as Pilot’s offering discounts for a year’s service for customers that paid up front. Customers took the deals, which in turn prompted the security vendors to scale up too fast. All of this is precedented; the ASP market did the same thing two years ago and has stalled ever since.

If small security-only companies can’t escape the economics of their smallness, the larger general purpose IT service companies can’t get out from under the weight of their hugeness. Brown evaluated several larger companies and came away unimpressed.

“My experience is the bigger companies don’t have the expertise or the service,” he says. “We looked at two of them, and it was a circus. They couldn’t even get coordinated internally. They hadn’t gotten our business, and they were already infighting as to who would handle our account.”

So for Durso, it becomes a balancing act. She’d like to stay with a security-only company because of the expertise and service. At the same time, she feels as if she has to slide up the scale to find a stable business. “Really we’re looking for a company like Pilot in terms of service,” Durso says. “But you find yourself opting to be more conservative.

“No one has all of the story we want,” Durso adds. “You’re always ending up with some kind of trade-off.”

As Durso now realizes, outsourcing security is not buying your way out of work but rather buying your way into expertise and then managing it. But expertise is still the thing. She’ll sacrifice only as much of it as is necessary in order to find a company that won’t go out of business and forget to tell her.

Playing It Safe

ABOUT THE SAME TIME DURSO SHOOK hands on her discount, the CIO at a major health-care organization on the West Coast called a meeting with a Pilot executive. This CIO, who asked not to be named because he believes it would paint a target on his network, had been an early sign-on for Pilot.

About 10 months ago, he watched his service lag and Pilot’s stock swoon at the same time. It gave him pause, so he set up a “frank discussion” with a high-level Pilot executive. At the meeting, the CIO challenged the executive on service levels and asked direct questions about the health of Pilot’s business and its capability to support him. The Pilot executive answered each question, and the CIO was reassured.

Even so, he wasn’t taking chances. After meeting with Pilot, he revisited his contingency plan and now feels fortunate that he was ready to go when he found out that Pilot was no more. “We worried,” he says. “We probably should have worried more. Next time, I’d be even more aggressive.”

This CIO’s contingency was relatively smooth. He started with a crude but sturdy frame-relay connection provided by Verizon. Once that was working, he set out to upgrade to a high-speed connection also provisioned by Verizon. After that was in place, he worked on adding secure access to his network in the form of a VPN. His e-mail contingency followed the same slope: first, low-bandwidth access to e-mail, then high-bandwidth access, then secure high-bandwidth access, which brought him back close to what Pilot had provided.

Concurrent to building the network up, he reinserted security services into his network while he sought a new managed security partner. He started by assigning one person to monitor the network, a pale substitute compared with what he was paying Pilot to do. But it was monitoring nonetheless.

For the first awful week, the CIO had to rely on volunteer ex-Pilot and Providian employees, who composed the management skeleton crew. But within three days, he was out from under Pilot, albeit with a temporary structure. “We’re still sorting it out,” he says. “We have some services. We won’t have others like filtering for a while. What we have now is OK.”

In choosing his next outsourcer, this CIO echoes Durso as he considers the trade-offs between the small vendor with talent versus the big vendor with a stable business. But he’s leaning the other way?toward a bigger company with more generalized services. He chose Genuity for his network connections. Choosing a managed security provider is predictably taking longer, but he wants a similarly large company, possibly Genuity.

“We’re not interested in breaking in new security vendors. I want to see Wall Street firms and large banks on their customer list. My ideal would be a large, funded company with diversified resources,” he says. His last requirement is the tricky part?an outsourcing partner has to be “one that’s also highly competent.” While the expertise still resides in the boutiques, the CIO anticipates that large general service companies will start bailing out the smaller companies. That way, they acquire the smarts, they have steady bottom lines, and they make security a component of larger managed services packages. And indeed, AT&T was ready to buy Pilot but walked away at the last moment, several ex-Pilot sources and customers say. Symantec has already bought a boutique company, Axent.

If this expertise-through-acquisition scenario plays out as such, CIOs will have the best of both worlds?stable business and expertise. But that presents other challenges. For example, Symantec has products to sell. Partnering with Symantec likely means partnering with Symantec’s products too. And service levels may drop as the smaller boutiques are subsumed by larger companies.

But for the health-care CIO, less service and expertise is fine. Outages are not.

Three weeks after the incident, with his contingency up and running, he says, “We dodged a bullet.”

Taking It In-House

NEIL HENNESSY, VICE PRESIDENT OF IT engineering for PeopleSoft, learned about Pilot’s collapse in a most unusual way. At the end of a weekly meeting with his Pilot rep, the man announced, “I have to go back to the office and get fired at 4:30.” And in the week leading up to Pilot’s bankruptcy filing, Hennessy says he was fending off a wake of vultures.

“One guy calls from Southern California, and he’s telling me how he can offer me everything Pilot did for less money,” Hennessy says. “So I ask him, ’How many employees do you have?’ and he tells me 40. So I said to him, ’Pilot had 400. Why would I trust you?’”

Truthfully, Hennessy had started to lose faith in Pilot beginning a year before its collapse. He was particularly worried about the company’s scalability. “They just couldn’t step up. Not that they didn’t try. Their model was very secure, but we started looking at other options back then,” he says.

Hennessy’s favored alternative was to phase out his managed security contract and take the task back in-house. This, after all, was the year of uber-viruses and broad, destructive hacks. Hennessy decided that security was just too critical to outsource.

His transition plan meshed with his contingency plan. Hennessy already had a backup carrier with a “dark” data line, one that’s not turned on but could be activated in an emergency. And he started building an internal security staff of five with five more to come.

So when his Pilot rep told Hennessy he was going to get fired that afternoon, Hennessy was able to set the plan in motion, and the transition to in-house 24/7 security was done in five hours. He credits the quick shift to his engineering team, whom he ranks somewhere between “real strong” and “the best in the world.”

The cost of doing it all in-house was and will continue to be massive, of course. Hennessy won’t deign to put a number on it, but he readily accepts the fact that he’s paying a premium for in-house security. “It’s definitely far more expensive doing it in-house,” he says. “On the other hand, there’s far less risk. I’m paying to sleep well.”

Why is it more expensive? To begin with, recruiting talent is hard. There’s little out there, and there are plenty of posers. Some experts put the ratio at about one real expert for every 10 claiming expertise. Certifications are partly to blame. A rŽsumŽ with a dozen security certifications might look impressive, but it’s misleading. Some certifications are simply for specific products and teach nothing about best practices or security policy. A firewall “expert” might know how to configure the box but have no knowledge of what policies should be enforced or even where the firewall ought to be placed in the context of a specific network.

Paying talent sufficiently is even harder than finding it. Stephen Northcutt, founder of the Global Incident Analysis Center and security consultant, says security contractors demand up to $500 per hour. Salaries are 5 percent to 10 percent higher than what standard IT staff earn.

Keeping talent is the hardest task of all. Northcutt says many true security experts are hopping jobs six times a year, upping their salaries $5,000 at each post. Len Cibelli, a former sales executive at Pilot, expects to get a 20 percent raise from his next employer.

Even so, Hennessy is convinced of the rightness of his decision. “We know doing it in-house is more expensive, but we’ve just decided it’s better than outsourcing,” he says. While talent is thin, Hennessy says a few strong candidates have come his way due to the economy.

The Hartford Financial Services Group, which was not a Pilot customer, has taken many of the same steps as Hennessy. Hartford Assistant Vice President of IT Jack Stoddard outsources little security, only ceding tasks such as auditing and penetration assessments to outside vendors. He retains 30 full-time security staff members, tries to recruit the best he can find, pays premiums for them and trains his staff continually. He is adamant about the limitations of the outsourcing model.

“I don’t see us ever outsourcing,” Stoddard says. His CIO, David Annis, believes acquiring and grooming security expertise in-house is critical, even if it costs more. He calls outsourcing “throwing in the towel.” But he understands why so many companies do it anyway. Security is so complex and demands such constant reassessments, he says, that doing it in-house requires a “fair amount of redundant due diligence.”

Postscript

ON MAY 9, EXACTLY TWO WEEKS after Pilot disbanded, it was liquidated. There was a Hail Mary as several managed security vendors tried to take over the business, but that collapsed. Emergency operations and support were halted. AT&T finally cut the circuits, and Brown at VisionTek received a page. VisionTek was still waiting for the local carrier to supply a data line, so Brown boxed up his equipment and drove it to its temporary home in a downtown Chicago facility.

On the same day, Pilot’s homepage finally changed its cheerful, “Yes, we’re open” message. “Pilot Networks has filed for bankruptcy” was all it said. There were some snippy redesigns of the Pilot website, obviously tacked up by bitter ex-employees. The title bar of Pilot.net read: “Pilot Network Services is now Imaginary Network Services Inc.”

Someone left behind a sarcastic note, which only hinted at what had gone on. Anyone who happened by and clicked on “What’s new” would see the note: “Here is the latest about Pilot: We’re done! Pilot is no more. This company is an ex-secure ISP. If it weren’t for being nailed to the perch, it would fall over. Alameda, CA, May 9, 2001.”

You can almost hear creaky saloon doors rattling in the wind and tumbleweed staggering through the dust.