What Went Wrong at Cisco in 2001

There’s Cisco Before and Cisco After, and the two crossed paths, awkwardly, this past April.

Cisco Before was CFO Larry Carter writing in April’s Harvard Business Review about the San Jose, Calif.-based company’s “virtual close” software. “We can literally close our books within hours,” Carter boasted in the article. “More important, the decision makers who need to achieve sales targets, manage expenses and make daily tactical operating decisions now have real-time access to detailed operating data.” Cisco’s decision makers possessed a godlike ability to peer into every nook and cranny of the business, 24/7, which Carter says allowed the company to forecast a slowdown in Japan’s economy and garner half of the switching market there. Cisco After was CEO John Chambers, admitting to The Economist that same month, “We never built models to anticipate something of this magnitude.”

That something was what is now inelegantly referred to as the recent economic downturn. It created a major earnings surprise for the manufacturer of switches and routers?the company’s first negative quarter in more than a decade. In the third fiscal quarter of 2001, sales plunged 30 percent. Chambers wrote off a mountain of inventory $2.2 billion high, and 8,500 people were laid off. On April 6, Cisco’s stock sunk to $13.63. Thirteen months earlier, it had been $82.

Chambers surveyed the wreckage and compared it to an unforeseeable natural disaster. In his mind, the economy?not his company’s software nor its management?was clearly to blame.

But other networking companies, with far less sophisticated tools started downgrading their forecasts months earlier. They saw the downturn coming. Cisco did not. Other companies cut back on inventory. Cisco did not. Other companies saw demand declining. Cisco saw it rising.

Even more troubling, there’s ample evidence that the company’s highly touted systems contributed to the fog that prevented it from seeing what was clear to everyone else. Cisco executives may have been blinded by their own good press. What’s clear is that overreliance on technology led the company down a disastrous path.

CIO Peter Solvik defends his company’s systems. He insists that without the forecasting software, that third quarter would have been even worse. He says that once executives realized there was a crisis, the day-to-day, near-real-time data helped Cisco quickly hit the brakes. It was just that “the speed of the swing caught everyone by surprise,” Solvik says.

Of course, surprise was exactly what Cisco’s software systems were supposed to eliminate.

The Growth Bias

MOST CIOS ARE FAMILIAR with the virtual close. Cisco has aggressively marketed it?and the rest of its internal software?as a huge competitive advantage. In numerous news accounts, CFO Carter was quoted as saying that these systems made the company both huge and nimble, Goliath’s brawn with David’s agile sling.

CIOs were envious, competitors fearful. No one came away from a virtual close demo without high praise, like Fortune: “Cisco uses the Web more effectively than any other big company in the world. Period.” Or Business Week: “It should mean zilch-o earnings surprises.” Cisco was also a perennial CIO-100 award honoree.

But not everyone was so impressed. Fred Hickey, editor of the High-Tech Strategist, notable for his hype-free, dogging assessments of networking companies, calls the power of the systems “bogus.” Frank Dzubeck, president of Communications Network Architects, a networking consultancy in Washington, D.C., who has worked with Cisco, calls the infrastructure “overrated and incomplete. There was a lot that wasn’t real with [Cisco’s] supply chain, with the inventory management,” Dzubeck says.

“For the last year, Cisco and John [Chambers] have said over and over again how their information systems make them so efficient and so on the ball,” says Jeffrey Young, author of Cisco Unauthorized (Prima Publishing, 2001), which details Cisco’s long climb and the beginnings of its return to earth. “You couldn’t go anywhere without hearing that if you weren’t trying to be like Cisco, you were falling behind, their systems are so brilliant. I’m sorry. It’s clear it’s not the case.”

What Cisco’s systems didn’t do was model what would happen if one critical assumption?growth?was removed from both their forecasts and their mind-sets. If Cisco had run even modestly declining demand models, Chambers and Carter might have seen the consequences of betting on more inventory. But Cisco had enjoyed more than 40 straight quarters of stout growth. In its immediate past were three quarters of extreme growth as high as 66 percent. The numbers the virtual close presented to the eye of the beholders?the Cisco executives?painted a picture of the present as lovely and pleasant as a Monet landscape. According to many observers, Cisco’s fundamental blunder was to rely on that pretty picture to assume the future would be equally pretty.

The Inventory Buildup

CISCO’S HISTORY is like Mount Saint Helens. Its explosive success blew the side off the old economy rules of slow, steady growth.

Cisco rose above a crop of small networking companies through two strategies: outsourced manufacturing and growth through acquisition. From the time it went public 11 years ago, Cisco was never not growing. Sometimes its growth was staggering. Its stock split 12 times in the ’90s. Its revenues went from millions to billions to tens of billions as fast as the Internet would let it. At its height?say, May 2000?44,000 people worked at Cisco, and thousands of them were millionaires. For a New York Stock Exchange minute, Cisco topped GE as the most highly valued company in the world and earned a half-trillion-dollar market capitalization. If growth continued at the same pace for another decade?and why wouldn’t it??Cisco would be as big as the U.S. economy. Without a hint of irony, analysts suggested Cisco might be the first company to have a trillion-dollar market capitalization.

In May 2000, Fortune put Chambers on its cover and asked if he was the best CEO in the world.

At the same exact time, a few components for Cisco’s networking equipment were rumored to be in short supply. Privately, Cisco was already twitchy because lead times on delivering its routers and switches were extending. Eventually those lead times would reach nearly six months on some products. Not having the components could push those delivery dates out even further. So Cisco decided to build up its components inventory. Doing that would accomplish two things: It would reduce the wait time for its customers, and it would give the manufacturers of Cisco’s switches and routers a reserve to draw on if components makers ran out.

Of course, everyone else wanted those components and the manufacturing capacity to build the networking devices too. So in order to get both, to make sure they would have them when they needed them (and they knew they’d need them; the virtual close told them so), Cisco entered into long-term commitments with its manufacturing partners and certain key components makers. Promise us the parts, Cisco said, and we promise to buy them. No matter what.

“Our forecasts were still dramatically high,” recalls Selby Wellman, a retired Cisco executive. A self-proclaimed outsider, Wellman retired last summer, for reasons he says are unrelated to business. Some of his last meetings at Cisco were about the components shortage of summer 2000. “We wanted to make sure our growth was strong, so we ordered up big time,” he says.

That seemed to work. Year-over-year growth was robust, 55 percent for the last quarter in 2000, and then a whopping 66 percent in the first quarter of 2001. As late as September, Cisco looked at its virtual close and saw plenty of bookings. It also had the pleasant problem of not being able to deliver products to customers fast enough. Combined, those numbers were enough to convince executives who literally had never seen a down quarter that everything was fine.

But some Cisco suppliers were not so sanguine.

“People see a shortage and intuitively they forecast higher,” says Ajay Shah, CEO of Silicon Valley-based Solectron Technology Solutions Business Unit, a company that manufactures parts for the networking industry and for Cisco. “Salespeople don’t want to be caught without supply, so they make sure they have supply by forecasting more sales than they expect,” Shah explains. “Procurement needs 100 of a part, but they know if they ask for 100, they’ll get 80. So they ask for 120 to get 100.”

Demand forecasting is an art alchemized into a science. Reports from sales reps and inventory managers, based on anything from partners’ data to conversations in an airport bar, are gathered along with actual sales data and historical trends and put into systems that use complex statistical algorithms to generate numbers. But there’s no way for all the supply chain software to know what’s in a sales rep’s heart when he predicts a certain number of sales, Shah says. It’s the same for allocation. If an inventory manager asks for 120 when he needs 100, the software cannot intuit, interpret or understand the manager’s strategy. It sees 120; it believes 120; it reports 120.

Furthermore, there’s a growth bias built in to the business of forecasting. If there’s a rule of forecasting demand, it’s to err on the side of needing more, not less. Be aggressive, because you don’t want to end up like Sony did in 2000 with its PlayStation 2 video game system: 100 people clamoring for three units. When that happens, 97 empty-handed customers might go buy a Nintendo. Hence, if demand is dropping off, it can be hidden behind overcommitment. Financial systems say sales are strong today, which managers who have never seen a bad quarter take to mean that sales will be strong tomorrow. So they forecast high demand. Everyone’s forecasting high demand, which in turn means it’s time to build up inventory.

Even as sales begin to dwindle.

Outsourcing’s Fatal Flaw

In the summer of 2000, Solectron’s Shah had customers from every corner begging for more manufacturing capacity. Even so, his forecasts were slowly diverging from his networking partners’, including Cisco. His were less optimistic, based on what he saw in the general economy. There were meetings about it, but nothing was resolved about the growing disparity between what Shah and his customers thought was happening and what Cisco said was happening.

“You try to talk it over. Sometimes it doesn’t work. Can you really sit there and confront a customer and tell him he doesn’t know what he’s doing with his business? The numbers might suggest you should. At the same time,” Shah laughs, trying to picture it, “I’d like to see someone in that conference room doing it.”

Here, the very core of Cisco’s infrastructure?its much-vaunted outsourced manufacturing model?worked against the company, according to M. Eric Johnson, an associate professor of business administration and a supply chain expert at Dartmouth College’s Tuck School of Business. Cisco’s partners were simply not as invested in delivering a loud wake-up call as an in-house supplier would have been.

“The outsourced model is a given, and it’s done some wonderful things,” Johnson says. “But Solectron has to watch their own business. It matters less to them if Cisco’s numbers look off. Plus, they don’t want to go tell a customer they’re wrong. Whereas if it were in-house, some low-level inventory manager might have spoken up and said, ’Look, this isn’t going to fly,’ because he has more of an incentive to [help] his company.”

Shah also says it would have been presumptuous to confront a company like Cisco and tell it it was wrong. When had Cisco ever been wrong? But now Shah thinks that overreliance on the forecasting technology led people to undervalue human judgement and intuition, and inhibited frank conversations among partners.

On top of that, there’s the possibility that despite what Fortune said, Cisco’s supply chain was not quite as wired as was hyped. Cisco, Solectron and others do plenty of business with companies that still fax data. Some customers simply won’t cater to the advanced infrastructure, making it harder to collect and aggregate information.

“Cisco has hundreds of these huge customers who aren’t going to do procurement the way Cisco wants them to,” Network Architects’ Dzubeck says. “Things like end-to-end EDI, standards, they have to occur. Cisco’s further along, but there’s a lot to do to make the infrastructure complete.”

The Autumn of Cisco’s Discontent

SEPTEMBER TO DECEMBER of 2000 is the most puzzling chunk of time for Cisco to explain. By September, a number of its component makers, such as Solectron, were backing off of their growth projections. Xilinx, a Silicon Valley-based company that makes specialized programmable semiconductors, told the world its growth would be 10 percent, down from previous projections of 16 percent to 20 percent. Both Kris Chellam, CFO of Xilinx, and Shah say most of their strategizing that autumn centered on whether the downturn would be a “V” shaped one?deep, short but right back up?or a longer, more serious “U” shaped recession.

But Cisco remained upbeat. Xilinx’s Wall Street warning came two months before Cisco Chief Strategy Officer Mike Volpi told The Wall Street Journal in November, “We haven’t seen any sign of a slowdown.” Volpi told The Journal that Cisco hadn’t changed its internal plans since the beginning of its fiscal year in August. “We have guided [Wall Street] accurately, and we can execute to plan.”

On Dec. 4, CEO Chambers crowed to analysts, “I have never been more optimistic about the future of our industry as a whole or of Cisco.”

Eleven days later, CIO Solvik says, the company saw the problem for the first time. It overlaid the virtual close and its forecast, and sales had crossed under its projections. Cisco decided to seriously curtail expenses. Three weeks later, revisiting the data would lead to a hiring freeze. Solvik says the near-real-time data prevented what eventually happened to Cisco from being worse, if that’s even imaginable.

“I’m saying it could have been a lot worse,” Solvik says. “We were able to react to trends day by day [using our infrastructure].”

But what was Cisco doing in September, October and November? Why wasn’t the virtual close telling Cisco that demand was evaporating? Why wasn’t the infrastructure that allowed Cisco to forecast Japan’s economic slump so accurately able to even see a more severe swing here? If Cisco acted quickly when the sales line fell under the forecasts line, why didn’t executives see those lines converging?

“We disclosed at the time what we thought,” Solvik responds. “Other than what we publicly stated, I can’t comment on the outlook [at that time].”

By year’s end, the economy was foundering. It was neither a “U” nor a “V” but a “7”. A cliff. The Fortune 100 halted capital spending. Alternative telecommunications carriers disappeared, along with many of the dotcoms that had been so feverishly buying Cisco gear. That equipment ended up on a gray market; barely-used Cisco switches could be had for 15 cents on the dollar, and Cisco lost money every time one was snapped up. Traditional telecom companies stopped spending too. In short, demand vanished. Cisco finally threw on the brakes Dec. 15. The freight train spit sparks and burned up the steel track as it tried to stop short of the cliff. It jolted down the supply chain, derailing suppliers like Xilinx and manufacturers like Solectron and their distributors. And Cisco, the networking industry’s big engine, went over the edge.

The Road to Recovery

CHELLAM AT XILINX attributes Cisco’s failure to act expeditiously to the fact that its software ignored such macroeconomic factors as debt levels, economic spending, interest rates, the bond market and so forth, while trusting data freighted with growth biases. In short, the virtual close and forecasting didn’t include the right economic indicators.

“They missed the shift,” Chellam says flatly. “They were focused on what their customers were ordering. No one looked at the macroeconomic factors overshadowing the entire communications industry and spoke up. Someone should have said, ’These orders can’t be sustained.’”

Chellam has created a task force at Xilinx to develop software that injects more macroeconomic indicators into his forecasts. He considers it the key ingredient missing across the networking supply chain. “I can’t rely on just what Cisco tells me,” he says. “We’re developing leading indicators that are more macro in nature. Information that comes through banking, Wall Street, debt levels and economic spending?they’re not intuitively related to me directly, but they have an effect.” In essence, Chellam is attempting to draw a more detailed map than the one Cisco used to navigate the future. It is still, however, a map, not the territory itself.

He’s also building a virtual close system, based largely on Cisco’s. “Forgetting their failure to look at [macroeconomic] factors, I still think Cisco’s use of the Internet is an asset,” Chellam says. “But tools like the virtual close are secondary to [human judgement] decisions, to process improvements. The virtual close can get better.”

Cisco is making the virtual close and its demand forecasting better, Solvik says. But it’s not clear how. As vague as Chambers sounded last May when promising a return to 30 percent to 50 percent growth sometime in the future, Solvik similarly doesn’t provide many specifics for how Cisco is updating its infrastructure.

Solvik does say that the next step in its virtual supply chain is to create much-needed visibility into all levels of the networking industry. What he calls the new network supply chain will also allow the company to communicate with all networking-related companies and even see data passed between two other companies in the chain. Also, he says Cisco is updating its forecasting software, enhancing each business unit’s forecasting capabilities.

One area Solvik is specific about is macroeconomics. He argues against its wide use, despite what Chellam says about the role of big-picture data. Solvik thinks the economy is too complex to get anything meaningful out of such broad numbers as gross domestic product or interest rates. “We can’t put a judgment across the entire supply chain. Look at the economy today. Consumer confidence is up, and capital indicators sunk again. It’s pretty darn complex,” Solvik says, with perhaps a new appreciation for the limitations of his software.

But Cisco still wants to attribute its recent problems to uncontrollable forces, as if the historic $2.2 billion inventory write-off and the steep decline of the company’s stock had nothing to do with the men at the top or the systems they trusted.

“If we could look backward, I know there are a lot of things we’d do,” Solvik says. “If we had a crystal ball…I just can’t predict how we might have reacted if we had insight into these things we didn’t know.”

By intimating that only magic could have saved Cisco, Solvik is seconding CFO Carter’s sarcastic comment to the Financial Times last April. “The slowdown happened at Internet speed,” Carter said. “We’re developing a new module for our system right now. It’s called a crystal ball.”

But some people don’t believe a crystal ball was what Cisco needed. The ability to look away from the computer screen and out the window to see the rain coming down just might have sufficed.