Outsourcing: What You Can Do If Your Security Vendor Fails

ON APRIL 25, PILOT NETWORK SERVICES went out of business, abandoning 200 customers that relied on them for something rather important: security. There had long been signs of Pilot’s distress. Customers had recently reported spotty service from the managed security company. Pilot’s stock, once at $50, had plummeted to 21 cents per share, and Nasdaq delisted it. Yet this was not some high-flying dotcom that appeared one day, took some easy venture capital, then vanished. Pilot was an established, 8-year-old vendor with 400 employees and, by most accounts, superior security technology and practices. Its customers included PeopleSoft, VisionTek, The Washington Post Co. and several large health-care institutions and banks.

Despite all that, the end came quickly. Pilot employees received four e-mails in rapid succession. The first said the phones would be disconnected. The second added that pagers and mobiles would be taken away. The third said the CFO had resigned. And for anyone who couldn’t see the elephant?not just in the room but squirting river water in their faces?the last e-mail said, "At 4:30 p.m., you’re fired."

Pilot did keep a skeleton crew to manage customers’ security through the data lines. Responding to desperate pleas from Pilot customers, AT&T suspended the order and kept Pilot’s operations center connected, even though it wasn’t getting paid.

With no one watching their networks and an outage threatening at any moment, Pilot customers felt naked. They were suddenly wide open to hackers and viruses. Because some companies routed office-to-office traffic through Pilot, they were at risk of losing secure virtual private network (VPN) connections and remote access. Pilot had hosted entire Web networks for other companies, making them even more vulnerable to a complete meltdown.

One such company, Providian Financial, was so distressed that it sent several IT staffers to man Pilot’s operations center. That probably frightened Pilot’s other banking customers, none of whom were expecting a competitive financial institution to have access to their network security.

While it’s perhaps the most dire example of failure in the slowing economy, Pilot’s breakdown is not an aberration. Other managed security companies are hurting too. The Salinas Group had already folded. Exodus endured an atrocious first quarter in which CEO Ellen Hancock said everything was on track "except revenue." Recently, Exodus and another managed security company, Counterpane, announced that they are joining forces for efficiencies of scale. MyCIO.com, once independently operated, was folded back into its parent company, McAfee. Two other managed service boutiques, Vigilante and Networks Vigilance, have merged. "Spending has tapered," says Bruce Murphy, CEO of Vigilinx, another managed security company. "A billion dollars in equity just dried up."

In a matter of days, the managed security services option turned into a frightening one for CIOs. Until now, outsourcing security management to a boutique company like Pilot seemed the best way to go for two reasons: One, that’s where the most cutting-edge security expertise had migrated, and two, doing security in-house was considered too expensive and difficult for most companies. But in the wake of the Pilot disaster, many CIOs are reevaluating two alternatives: outsourcing their security needs to a large, general services company such as IBM Global Services, or taking care of them in-house.

The problem is that none of the three available options is the clear winner. Each carries significant risk, and former Pilot customers are trying them all. But all of them agree on one point. Outsourcing security is more work than just writing a check every month. It’s a full-time job that requires in-house resources. Treating it as any less?and many do?is playing Russian roulette with the entire enterprise.

Small Is Better?

ONE WEEK AFTER THE IMPLOSION, Pilot filed Chapter 7 in Oakland Bankruptcy Court. Its website, The Pilot.net, made no mention of the company’s troubles. In fact, the site looked exactly the same as it had before the collapse. It had an eerie feel, like some Western ghost town.

Pilot’s outage couldn’t have come at a worse time for Ann Marie Durso, CIO of VisionTek, a memory and graphics card company in Gurnee, Ill. She had joined the company in October 2000 and was in the thick of a strategic ERP project that will help the company launch online retail sales. An outage would mean revenue losses on online sales, and each day without a secure, high-speed connection would add several days to the ERP project.

VisionTek has subscribed to Pilot for four years. Like a marriage, the partners just got comfortable talking less. Security was assumed, and just two months before Pilot went down, Durso had been baited with a renewal discount. Pilot offered to renew her contract at a cut rate if she paid for a full year up front. She did.

"We got blindsided," she says. "We thought [that since] this was a provider that had been around since ’96 for us, there was less of an inclination for us to question them. But outsourcing isn’t an abdication. You can’t just hand it off. Ultimately, the business will hold me accountable, so I have to manage the third parties. I have to constantly ask, Are they still growing? Can they handle scale? Are they keeping their skills up?"

As soon as Durso heard about Pilot, she and her network manager, Mike Brown, went from office to office briefing VisionTek’s executives, one at a time, on what the collapse meant to the company.

"It wasn’t pleasant," Durso recalls about the experience of having to break the news to the CEO, the CFO and the controller. Interviewed by CIO the day Pilot filed for Chapter 7, Durso was still frayed. "But we’re doing the right things. We had a full contingency in place in two days," she says.

The contingency went something like this: First, get the executive staff’s permission to move forward on choosing alternative security providers. Second, create a worst-case plan. For VisionTek, this meant Brown put his pager on and never took it off.

Worst case, if AT&T cut the network connections to Pilot, Brown would be paged. He’d box up his servers and drive them from Gurnee to downtown Chicago, where an alternative provider had offered space and dial-up connections until VisionTek could find a full-time provider.

Next, VisionTek brought in two ex-Pilot engineers as contract consultants because they knew Durso’s security better than she did. In fact, the day after Pilot went down, VisionTek wasn’t sure of its security status because it had, over time, become Pilot’s responsibility to manage.

Together, the Pilot engineers and Durso figured out where they stood and got the network to a point where "we were at least able to limp along," she says. With security patched together, Durso, Brown and the consultants turned their attention to evaluating other security vendors. Ironically, she wants a partner similar to Pilot in scope and methodology. Durso liked Pilot’s level of expertise. She liked its 24/7 monitoring. Finding another Pilot with stable financials is unlikely. But Durso knows larger companies often have less expertise.

Highly sought security talent flowed to the boutique companies for two reasons. First, top IT security experts?often from the military and government agencies such as the CIA?left public service in droves a few years ago to start their own companies. Subsequently, venture capitalists heard tales of Pentagon-level security, so there was plenty of money out there, until recently. Second, there was fraternal loyalty; security experts gravitate to companies run by their peers.

But the startup trend led to a glut. There were too many boutiques, and they were burning cash fast. That, in turn, led to aggressive selling, such as Pilot’s offering discounts for a year’s service for customers that paid up front. Customers took the deals, which in turn prompted the security vendors to scale up too fast. All of this is precedented; the ASP market did the same thing two years ago and has stalled ever since.

If small security-only companies can’t escape the economics of their smallness, the larger general purpose IT service companies can’t get out from under the weight of their hugeness. Brown evaluated several larger companies and came away unimpressed.

"My experience is the bigger companies don’t have the expertise or the service," he says. "We looked at two of them, and it was a circus. They couldn’t even get coordinated internally. They hadn’t gotten our business, and they were already infighting as to who would handle our account."

So for Durso, it becomes a balancing act. She’d like to stay with a security-only company because of the expertise and service. At the same time, she feels as if she has to slide up the scale to find a stable business. "Really we’re looking for a company like Pilot in terms of service," Durso says. "But you find yourself opting to be more conservative.

"No one has all of the story we want," Durso adds. "You’re always ending up with some kind of trade-off."

As Durso now realizes, outsourcing security is not buying your way out of work but rather buying your way into expertise and then managing it. But expertise is still the thing. She’ll sacrifice only as much of it as is necessary in order to find a company that won’t go out of business and forget to tell her.

Playing It Safe

ABOUT THE SAME TIME DURSO SHOOK hands on her discount, the CIO at a major health-care organization on the West Coast called a meeting with a Pilot executive. This CIO, who asked not to be named because he believes it would paint a target on his network, had been an early sign-on for Pilot.

About 10 months ago, he watched his service lag and Pilot’s stock swoon at the same time. It gave him pause, so he set up a "frank discussion" with a high-level Pilot executive. At the meeting, the CIO challenged the executive on service levels and asked direct questions about the health of Pilot’s business and its capability to support him. The Pilot executive answered each question, and the CIO was reassured.

Even so, he wasn’t taking chances. After meeting with Pilot, he revisited his contingency plan and now feels fortunate that he was ready to go when he found out that Pilot was no more. "We worried," he says. "We probably should have worried more. Next time, I’d be even more aggressive."

This CIO’s contingency was relatively smooth. He started with a crude but sturdy frame-relay connection provided by Verizon. Once that was working, he set out to upgrade to a high-speed connection also provisioned by Verizon. After that was in place, he worked on adding secure access to his network in the form of a VPN. His e-mail contingency followed the same slope: first, low-bandwidth access to e-mail, then high-bandwidth access, then secure high-bandwidth access, which brought him back close to what Pilot had provided.

Concurrent to building the network up, he reinserted security services into his network while he sought a new managed security partner. He started by assigning one person to monitor the network, a pale substitute compared with what he was paying Pilot to do. But it was monitoring nonetheless.

For the first awful week, the CIO had to rely on volunteer ex-Pilot and Providian employees, who composed the management skeleton crew. But within three days, he was out from under Pilot, albeit with a temporary structure. "We’re still sorting it out," he says. "We have some services. We won’t have others like filtering for a while. What we have now is OK."

In choosing his next outsourcer, this CIO echoes Durso as he considers the trade-offs between the small vendor with talent versus the big vendor with a stable business. But he’s leaning the other way?toward a bigger company with more generalized services. He chose Genuity for his network connections. Choosing a managed security provider is predictably taking longer, but he wants a similarly large company, possibly Genuity.

Related:
1 2 Page 1
Page 1 of 2
Get the best of CIO ... delivered. Sign up for our FREE email newsletters!