Jack Lee marked Nov. 24, 2003, on his calendar because that was the day he would finally be able to change his cell phone carrier without losing his phone number-thanks to a Federal Communications Commission ruling. But Lee, president of Tangara Technologies, a company that develops software for forms, decided to wait a day before switching to AT&T Wireless to let the chaos of "number porting" die down a little. Little did he know that the chaos was just beginning.
Lee ordered his new phone on Nov. 25. When he went to the AT&T Wireless website to check the status of his order a day later, he was greeted with a message: "We could not find a porting request for this number in the system. Please contact Customer Care." It was the beginning of a two-month odyssey in which Lee estimates he made 15 to 20 calls to AT&T Wireless, sent nearly as many e-mails and spent 60 hours on the phone dealing with customer service representatives or waiting on hold-with the line often going dead when AT&T Wireless’s customer service lines became overloaded.
After being routed all over the company, Lee finally discovered what was going on. A major CRM system had crashed during an upgrade, and customer service representatives could not set up or access new accounts. The system breakdowns, which continued through February 2004, swamped other AT&T systems, gridlocked customer service phone banks and sent furious customers scurrying to other providers.
The breakdown couldn’t have come at a worse time for AT&T Wireless. It deprived the telco of thousands of potential new customers and cost the company an estimated $100 million in lost revenue. But that wasn’t all. The failure so damaged AT&T Wireless’s reputation that many analysts believe it hastened its sale to Cingular in February for $41 billion, or $15 per share, which was just under half the value of AT&T Wireless’s shares when it went public in April 2000. While an AT&T Wireless spokesman says the company would have been sold regardless of the fiasco because "it was the right thing to do," the crash and resulting confusion could not have helped AT&T cut a good deal. "The system problems made AT&T look like a wounded provider, and the sharks smelled blood and circled," says Roger Entner, a wireless and mobile services analyst at The Yankee Group, a research company.
AT&T Wireless’s mistakes offer valuable lessons for CIOs. For one, it’s unwise to freight major system upgrades with external complications. AT&T Wireless’s CRM upgrade was hamstrung from almost the very beginning by rumors of outsourcing deals and future layoffs. These rumors generated pervasive morale problems that hurt the productivity of project staff.
Second, it should be understood that complex projects require flexible deadlines. AT&T Wireless undertook a difficult upgrade that affected roughly 15 systems just before it was faced with an immovable deadline-the federally mandated Nov. 24 number portability date.
Finally, it always pays to have a plan B. Without one, AT&T Wireless was forced to move forward even as it became apparent that its upgrade would not be completed in time.
Playing Catch-Up with the Competition
When AT&T Wireless began its Siebel CRM system upgrade in 2003, it was a company that had slipped from unquestioned market leader to middle of the pack. Its overall market share had slid from an industry-leading 25 percent at the end of 2001 to 17 percent in 2003, third behind Verizon and Cingular.
Worse, AT&T Wireless was playing catch-up on its most important technology asset, its phone network. One of the older wireless companies, AT&T Wireless made an early bet on a technology called TDMA that could not handle data transfer over cell phones-the next big thing for business customers. Even before AT&T Wireless was spun out from its parent AT&T in 2001, it had begun a furious buildout of an expensive new network-global system for mobile communications, or GSM-that could not only handle data but had the added advantage of global compatibility with overseas providers. Only one other major carrier, Cingular, was saddled with the challenge of building out a new GSM network while still servicing the old one. The other carriers had all chosen network technologies that could handle data transfers.
GSM was a great opportunity for AT&T Wireless, but it was also a huge CRM challenge. The company had to convince its old customers to move off TDMA, which worked as well as most other carriers’ networks for voice calls, and onto GSM, which had poorer voice quality, according to Morgan Stanley. It also had to convince new customers that GSM was the wave of the future, that they would soon be shipping data over their phones instead of their laptops. But customers didn’t buy the pitch. By 2003, AT&T Wireless’s percentage of customers on GSM was hovering at 15 percent, according to analysts, while Cingular had 35 percent of its customers moved over (thanks, in part, to its acquisition of a cellular provider with an existing GSM network). And things weren’t improving. In the third quarter of 2003, less than half of AT&T Wireless’s new customers were choosing GSM, while Cingular was signing up 75 percent, according to Morgan Stanley.
Everyone at AT&T Wireless agreed that the company would keep its existing customers and add more new ones if its customer representatives could handle more calls and get customers up and running faster. Customer service representatives needed about 20 minutes, on average, to work through five or six screens that fetched information from about 15 legacy systems, say former employees. Slow access to customer information records was just one of the reasons why AT&T Wireless possessed the second-highest cost per subscriber (behind Sprint PCS) of the top six national carriers in 2003, according to financial services company UBS.
Why AT&T Needed to Upgrade
AT&T Wireless installed Siebel’s CRM system in 2001 to be the front end of its customer service process. The back end, however was a complex mishmash of systems, say former employees. Telco billing systems, for example, were stuffed full of different rate plans and arcane metering processes. Systems that tracked calls and set up new phone numbers (provisioning) communicated with hundreds of thousands of different telephone switches around the country and the world. To work for AT&T Wireless, Siebel’s version 6 had to be highly customized. Though the software came with integration tools, consultants usually resorted to writing point-to-point scripts to hook the systems together. Policing the overall integration in a scenario like this is difficult at best. Indeed, a former AT&T Wireless employee who worked on the project recalls the test system crashing and remaining down for six weeks during the summer of 2002 when AT&T Wireless began preparing Siebel version 6 to deal with number portability. And when Siebel 6 was finally up and running, it still couldn’t handle all the information that customer service representatives needed.
The next release of Siebel, version 7, was much more powerful. Siebel developed an industry-specific version of the new software that had many more features and could capture more telco-specific information. And new Web capabilities meant that AT&T Wireless could potentially reduce the number of customer service screens to one by building a Web portal that put all the different systems and information sources together in one place. "That was the point of the upgrade," says a former employee. "They wanted to get everything on one screen so they could sign up customers faster."
Marc Siegel, a spokesman for AT&T Wireless, agrees:"As we needed to handle more transactions, more customers, we needed a more robust system. So we upgraded. [The upgrade] was going to make it easier for our representatives to get access to information and give them a fuller array of information on the customer."
In the spring of 2003, the company decided to upgrade to version 7.5 for the roughly 3 million customers on the GSM and general packet radio service (the data portion of the new network). TDMA customers would continue to be serviced through AT&T Wireless’s legacy Axys CRM system until they could be moved over to GSM and the new Siebel system. The project was called Odyssey.
The Upgrade Begins, Badly
Though the upgrade promised gains in speed and simplicity, getting there was neither speedy nor simple. AT&T Wireless’s IT staff had to rip apart old links from the different legacy systems to the client/server-based Siebel 6 system and rewrite them to work over the Web. The back-end systems-integration work was so complex that it wasn’t unusual to see teams of 20 or more people assigned to write connections for a single system, says a former employee. Coordination between the teams-the responsibility of the lead integrator on the project, Deloitte and Touche-quickly got out of hand. (Deloitte and Touche did not respond to repeated requests for an interview.)
"Everything was siloed among the different groups, and we all worked independently of each other," says a project team member. Teams would work on a revision to their piece of Odyssey, for example, only to find that when they finished testing, code had changed elsewhere in the system, rendering the testing meaningless. "In other projects I’ve worked on, the project managers would freeze code while the teams did revisions to their pieces so you could test everything against the same code base," he says. "I didn’t hear of that happening on this project."
Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."
Meet the New Boss
On April 10, 2003, Mike Benson, a 15-year AT&T veteran, sent a memo to employees announcing that he was leaving after three years as CIO. AT&T Wireless’s Siegel says the 48-year-old Benson "decided to retire." Christopher Corrado, the head of the security solutions practice for Wipro, an Indian offshore outsourcing company, was named executive vice president and CIO. Before joining AT&T Wireless, Corrado had been CTO at two divisions of Merrill Lynch, where he presided over the offshore outsourcing of portions of Merrill Lynch’s IT. Employees speculated that it was just a matter of time before offshoring began at AT&T Wireless.
"Corrado was the hatchet man-everyone knew it," says a former employee. "We’d see our people going into these long meetings with people from Indian companies. We’d see whiteboards that had questions like, ’What opportunity do we have to offshore/outsource?’"
Former employees say morale wasn’t helped by Corrado’s first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee. "We all came away saying, ’Who is this arrogant jerk?’" Corrado could not be reached for comment.
The Deadline Looms
As November approached, AT&T Wireless was one of the last industry holdouts opposing the Federal Communications Commission’s new rule on number portability, along with Cingular and Alltel. The industry had succeeded in delaying the change since 1998 on legal technicalities, and AT&T Wireless was counting on another postponement, say industry sources. But on Halloween, the U.S. Court of Appeals for the District of Columbia Circuit rejected the appeal to move back the FCC’s Nov. 24 deadline. An AT&T Wireless spokesman told Bloomberg News that the company was ready for porting.
Things were going so badly with the Siebel CRM upgrade that there was talk on the project of trying to roll back to the old Siebel 6 system before the porting date. "Two weeks before launch there was rollback talk," a former employee says. But project managers hadn’t preserved enough of the old system to make it feasible. There was no plan B. Project members estimated that the team needed two or three more months of bug fixing and testing to get the system running reliably.
And the Siebel system wasn’t the only thing AT&T Wireless needed to get working before the deadline. Wireless number porting required new systems that had to be integrated with the different carriers’ CRM systems. Five of the six top providers outsourced the administration of the number changes to TSI Communications and used software built by Telcordia. But in June 2002 (before the other carriers had announced they were going with TSI), AT&T Wireless chose its longtime software and administration provider, NeuStar. Ultimately, this would create serious interoperability problems for AT&T Wireless.
AT&T Wireless defends its decision by saying that NeuStar was the most experienced provider in the market, having worked on landline portability in the early ’80s. It also says NeuStar’s software offering was more complete than TSI’s at the time AT&T made its decision.