Keep Your Voice-Over-IP (VoIP) Projects Running

Long before the Vonage IPO turned into the year’s worst use of investor money, CIOs had started feeling chilly about voice over IP (VoIP).

In fact, after 2005 studies showed that 32 percent of American firms were piloting VoIP, a Forrester Research survey in August 2006 showed that adoption rates were flat from the same period a year earlier. Forrester Research analyst Lisa Pierce declares that enterprise VoIP deployments in the United States “have stalled.”

The reasons why will sound familiar to most CIOs: organizational deadweight, technology fear, uncertainty and doubt, and competing demands on network upgrading funds. But you can get through a VoIP pilot without stalling. We talked to CIOs who’ve done it, to look at some common and unexpected trouble spots, plus get advice on how to avoid or conquer them.

Unexpected Tech Troubles

The problem: Can you manage a VoIP network with your existing staff once it’s in place?

The answer: Be prepared for a few glitches, and get hands-on wisdom during install.

VoIP is supposed to make the phone system better, not make it stop ringing. But one early side effect of the way a new VoIP system had been implemented at Serta Mattress was causing phones to stop ringing for as long as 20 minutes at some of its factories.

For the not-so-sleepy mattress maker, VoIP emerged as a potential answer to the problem of rapid growth during the 1990s. Ambitious and judicious management took a mattress factory with a Serta license and built it up to the point where it acquired Serta itself: By 2003, the company had gone from three facilities to 23. Each of them had a telephone system (a PBX, in phone parlance), and each a maintenance contract, eating up between $500 and $1,000 per month.

Donna Zett, CIO at Serta, thought VoIP would give her a way to run a single PBX system, based at company headquarters in Hoffman Estates, Ill. That would save the company a bundle on maintenance, for starters (its VoIP system now costs it about $2,000 a month in maintenance). And since an internal study showed that 60 percent of its long-distance calls were between facilities, VoIP promised Serta savings on phone bills too.

Still, Zett trod carefully. She put three vendors—Cisco, Avaya and Sphere Communications—through their paces. That included access to reference customers with network and corporate structures similar to those of Serta. She even went to visit six of those reference customers. Serta chose Sphere, and spent about $150,000 and four months rolling out the VoIP system. Zett had her own staff work on the installation project, to make sure it could handle the new technology.

That install experience proved vital after Serta had moved past the pilot stage and implemented VoIP in several of its locations. Zett had opted to use the wide area network to do routine Sphere software upgrades. But after a few of the sites had moved to VoIP, it turned out that slow server response time could temporarily suspend phone service.

“We could lose phone operations for 15 or 20 minutes,” she says. “And that’s way too long.”

Because they’d installed the system, Zett and her staff knew they had several ways to fix this problem, the easiest of which was to schedule the updates at night. So they decided to install local upgrade servers (ordinary PC blade servers) that download updates only at prearranged times, usually at night, which allows for a much quicker update to the hub in the local facility. Serta’s IT team also created the option of having local operators at each plant, instead of routing all calls through headquarters.

Zett’s been happy with VoIP, and especially happy that she had her staff do the rollout. “Doing our own rollout meant we learned a great, great deal about what we liked and didn’t like—we’re not dependent on our vendor,” she says.

Beat the Vendor Blues

The problem: What do you do when your vendor can’t deliver as promised?

The answer: Know when to pull the plug on one vendor, and know what questions to ask the replacement.

Saving mad money on telephone costs is what makes most CIOs check out VoIP. That’s what got the interest of Marty Resnick, director of technology operations at Norman’s Nursery, a $65 million nursery wholesaler in San Gabriel, Calif. The cost savings are what kept him going through a “Nightmare on IT Street” technology pilot, which he hopes has no sequel.

Norman’s Nursery has three facilities, none closer to each other than about 100 miles. Its phone bills in 2004 were running close to $6,000 a month, much of the cost due to calls among its three facilities. The company couldn’t transfer calls between facilities, and none of its employees had their own extensions. Resnick thought VoIP should fix that problem.

He had a laundry list of other things he’d like in a new phone system as well: a unified paging system, message boxes that combined voice mail and Outlook e-mail, and faxing from Outlook. He went to his telecom provider, SBC, which recommended a specific Nortel BCM (Business Communications Manager) and a third-party systems integrator to implement it.

Resnick wasn’t very happy about being assigned to a third-party integrator, but he trusted that SBC knew what it was doing. He got a quote of about $135,000 for the system and its implementation.

Things started to go wrong quickly. For starters, the third-party integrator seemed unfamiliar with the system, spending a good deal of time reading documentation. Worse, the system SBC had recommended didn’t actually have all the features Resnick wanted. Those it did have didn’t necessarily work. For example, the receptionist’s console was built around software from yet another third party, and it crashed constantly. Suddenly, Resnick found his two-month implementation limping into month seven.

As an added insult, he couldn’t even call someone at SBC to yell at them. SBC was in the process of merging with AT&T, and the people he knew had been reorganized into different jobs and in some cases had left the company entirely.

Resnick might have washed his hands of VoIP, but the pilot did do one important job right: It let the company transfer calls over the data network, which almost immediately led to savings on the phone bill. Costs fell from $5,000 to slightly more than $1,000 per month. But he was still paying SBC money for features he’d been promised but hadn’t received. So he went looking for a different VoIP system.

The key now: He had learned how to probe vendor answers. For instance, he knew to ask if fax integration meant his users could print faxes from their desktops. He knew to ask whether a feature was built-in or came from a third-party company. And he knew to get documentation proving functionality.

The pilot meant he had laid the foundation for VoIP. He’d assigned all his extensions and direct-dial-in numbers, and had installed a PRI (primary rate interface) line, which accommodates voice and data and allows both caller ID and direct-dial-in functionality on one T1 line, functionality you can’t get on a regular T1 line. He had also trained users on VoIP concepts.

That meant that in February 2005, when he brought in a system from Shoretel, a VoIP startup, things went smoothly. His new system was up and running in less than a month, with almost all the features he had wanted. Better yet, it cost him less than SBC’s approach: about $105,000. And he was able to ditch the SBC system.

Resnick declines to point fingers at SBC, saying he was too quick to accept it when SBC told him VoIP is a “piece of cake.” Second time through, he made sure to communicate what he needed and make sure he understood the limitations of the system.

Manage Competing Priorities

The problem: How do you roll out VoIP when top management has other priorities, and when you need several third-party partners to go along with you?

The answer: Realize VoIP pilots will take more time than most other pilot tests.

Mike Benson wishes he were done with the VoIP installation going on at his company’s 30 call centers. Benson, executive vice president and CIO of DirecTV, hoped to be saving millions of dollars a month in phone bills with VoIP by mid-2006. Instead, he’s still working toward a second-quarter 2007 finish date.

Benson wants VoIP because it costs DirecTV 1 to 2 cents to transfer a call from one call center to another. That doesn’t sound like much money until you realize that DirecTV spends about $60 million a year on telecom, about 75 percent of which Benson estimates comes from connecting calls between centers. VoIP could eliminate those transfers, saving DirecTV several million dollars a month on its phone bill.

The good news when Benson’s pilot began: DirecTV had been planning an upgrade to MPLS (multiprotocol label switching) networking infrastructure and was upgrading its Avaya equipment already. So VoIP’s cost would be inconsequential.

Benson piloted VoIP in one of the company-owned centers and found that it was “pretty straightforward” to implement. VoIP did require making sure switches at the center and outside of the network were IP-capable and had appropriate software in them.

The bad news: Benson controlled only four of the call centers DirecTV uses. The other 26 were outsourced to three companies, primarily Convergys. Planning talks with his third-party providers went slowly. Convergys was also moving to an MPLS network, and the two companies planned similar network architectures and used similar equipment. But that wasn’t true for his other providers, which didn’t necessarily have the same priorities for IT, either. And Benson would get no cost savings if VoIP weren’t in place at both ends of the network.

Today, Benson has learned some lessons about managing the multiple vendor issues. While it does take time to nudge a service provider in a direction it might not want to go, there are things that help, Benson says. In his case, having one large service provider and two smaller ones meant he could play them off against each other, to a point. The smaller providers wanted to get more of his business, so they acted more cooperative, which put pressure on Convergys to come around. (It helped that Convergys started to miss its service-level targets, which meant Benson could legitimately threaten to pull contracts.)

Another lesson learned: VoIP may be at the top of your to-do list, but it’s probably not at the top of your organization’s. Chances are, a CIO will be pulled in unexpected directions during a VoIP pilot. Benson, who of course has to respond to other, competing needs from within DirecTV, is no exception. If customer service wants a new call center built, Benson needs to pull people from the VoIP project for the duration of the call center project. This year he’s had to build a new call center for DirecTV and help his third-party providers open two other call centers.

Such dilemmas and diversions don’t dull Benson’s ardor for what he thinks will be hefty telecom savings. His centers stand completed, and he thinks by the end of 2007 all the third-party call centers will also be done, though since some of DirecTV’s outsourced call centers use Nortel equipment, DirecTV still needs to test IP calls between Avaya equipment and Nortel equipment.

Once this project is done, he’s got another VoIP pilot in mind, examining IP-enabled phones. He’s learned, though, to expect significantly longer planning time for VoIP projects, especially in a company where keeping the phones ringing is paramount.

Take the Cake

Despite the issues that come with VoIP, Forrester’s Pierce says companies do want to adopt it. “The percentage of companies that believe they will stay on old technology indefinitely goes down and down and down,” Pierce says. For smart rollouts, she recommends the “chocolate layer-cake” approach.

By that, she means implementing VoIP in phases. Use new sites or places where aging equipment or a growing workforce create opportunities for upgrading equipment. That gives staffs a chance to gain experience with what remains an emerging technology.

And makes it more likely that if the phones don’t ring, you’ll know why.

Michael Fitzgerald is a freelance writer based in Millis, Mass.

He can be reached at michael@mffitzgerald.com