The Perils and Promise of Real-Time Data

When you first meet CIO Ron Rose, he’s more than happy to tell you about the 70,000 or so things that can go horribly wrong at Priceline.com, the consumer travel company built solely on a website that, in 2006, gets 10 million page views a day and books nearly $3 billion worth of travel transactions annually.

MORE ON CIO.com

More Data on the Way

Cisco’s Forecasting Disaster

IT Tools to Cut Through the Fog

Generally speaking, those 70,000 data points are monitored on a real-time IT system dashboard. The company has been testing new dashboards that offer up-to-the-second information and correlation analysis on numerous systems, including the state of the plumbing and network operations; CPU utilization; various application metrics (how much time is needed to transfer data within the system); Oracle database performance; BMC-monitored performance of things like I/O utilization; operating system paging (how much data is moving to and from the systems disks; and if the operating system is running out of RAM to work with) and a whole lot more.

All those metrics (and more) are crucial to Priceline.com’s business as illustrated by a recent Harris Interactive consumer study that found that 40 percent of online consumers will abandon their transaction (or turn to a competitor) if their initial attempt to interact with a site is foiled.

So Rose and his IT staff collect and analyze a torrent of real-time data to identify, prevent and fix problems before that happens. And he says being able to do so has saved the company millions in downtime and repair costs over the years. “Winning by not losing,” he calls it.

Rose’s IT group isn’t the only beneficiary of Priceline’s real-time capabilities. Priceline’s business analysts tap into a business activity monitoring (BAM) system, which can slice and dice up-to-the-minute information detailing the types of airline tickets, hotel rooms or car rentals that are selling, the completion percentage of different types of orders and (much) more.

All those data points (and more) give business users the ability to see trending demand for specific airline or hotel offerings, or whether visitors are completing transactions or bailing out at the last minute on certain products. The business users can then adjust that data to generate more sales. “[The business group’s] hourly reports, which summarize the financial data as it moves through the company, is the MTV of the technology department,” Rose says. “They love to keep their fingers on the pulse.”

But with that dependence on such fast-moving and variable data, Rose acknowledges that users also have to be aware of any noise lurking in the system—for example, when there might not be a statistically valid amount of data (say, too small a sample size for one of Priceline’s sales categories, such as bookings at one of its smaller hotels), which a business user may think is a trend when, in fact, it’s not. “It takes more than just a few minutes to make a trend,” he cautions.

Rose is confident, however, that Priceline is using all that near-real-time data to make better business decisions and provide a highly available website with fewer and fewer instances in which any of those 70,000 things that can go wrong do.

Everybody Loves Data

Moving to a real-time information-delivery environment like Priceline’s has long been an ambition for many companies. Real-time capabilities in business performance dashboards, systems monitoring applications, business intelligence software and supply chain management tools have propagated as companies struggle to keep up with the demands of 24/7 global operations. According to a September 2006 Teradata survey, 85 percent of responding executives say that decision-makers need more up-to-date information than in the past.

But as many companies have long known, more information, delivered more frequently, hasn’t always led to faster or better decision making.

The real-time boom has introduced some unintended busts: overwhelmed business users and IT managers drowning in too much information, with floods of irrelevant business activity alerts and system performance data leading them to make rash decisions or turn off real-time applications altogether. “If you’re not giving real-time data to the right people, at the right time, you’re opening up yourself to a lot of risk,” says David Williams, research VP of IT operations management at Gartner.

Of course, by itself, providing data in real-time isn’t dangerous. “Real-time information is always useful if you know how to make sense of it,” says Hau Lee, the Thoma Professor of Operations, Information and Technology at Stanford University’s Graduate School of Business. “That ‘if’ is the problem.”

Indeed, there are a steep learning curve and cultural change in a real-time environment that many organizations underestimate. “Everybody is crying for this data, but when you give it to them, they find fault with it,” says Heineken USA Director of IT Carol Schillat. “And the fault is that they don’t know how to use it.” Ill-planned real-time data implementations can be disastrous, negatively affecting customers, profits and productivity, according to Teradata’s survey.

To avoid the heartbreak of a failed real-time romance, CIOs need to understand which information their company really needs, how that information matches up with the way the business users do their jobs, and how and when it’s most beneficial to deliver that information. Once that analysis is complete, CIOs can install a process and IT system that delivers more actionable and correctly timed data flows. “If the business folks haven’t provided that level of detail, and IT didn’t ask for it, the system can provide not enough or too much information,” says Kevin Poole, consulting services leader at Capgemini. And in either case, “[business users] will start to ignore it.”

Real-Time Relativity

Like most terminology in the high-tech world, real-time means different things to different people in different industries. But what’s common to most people’s definition, says Royce Bell, CEO of Accenture’s Information Management Services, is that the data is delivered “within an actionable time frame,” whether that means within seconds or hours.

In the financial world, real-time data is, by necessity, defined as instantaneous. Traders, brokers and fund managers have to have information on global stock, equity and commodity markets delivered by the second. One can easily see why. In the financial services industry, downtime costs anywhere from $1.4 million to $6.5 million in lost revenue per hour, according to industry sources. A similar case can be made for systems in e-commerce companies such as Priceline.com, or in the airline industry (air-traffic controllers), utility industry (controllers monitoring electricity grids) and healthcare personnel (nurses monitoring patients), where even the smallest fluctuation in data is significant.

Step away from those segments, however, and the notion of what’s instantaneous begins to slow down by minutes, hours or days, and the question of just how much, and just how often, becomes more uncertain. “Most organizations believe they need live data, but [in reality] they tend to consume things in a daily cycle,” says John Hagerty, vice president and research fellow at AMR Research. “Daily is about as fast as they can do it.”

Consequently, other terms closely related to real-time have appeared, including near-real-time data (anything updated more frequently than daily) and right-time data (updated any time of day or week that the company has determined to be most beneficial).

At Delta Apparel, a $270 million manufacturer and distributor of branded and private-label activewear, CIO Keith Smith describes his “two worlds.” In one, subsecond real-time data informs decision making in Delta Apparel’s manufacturing operations—from tracking when an order of polo shirts will be completed to figuring out which distribution center in the United States is best-suited to distribute those shirts in the shortest amount of time. “This is where real-time data is critical,” he says. But in the other world—for sales information and budgeting—”real-time data totally falls apart,” Smith says. It’s just not practical or necessary.

Though Delta Apparel and Priceline were able to distinguish between the two worlds and adjust their data collection and delivery systems accordingly, many companies haven’t been able to. And that’s where real-time can get real dangerous.

Blinking at Real-Time

For the majority of 21st-century businesses, the possibilities of real-time data streams are endless and endlessly seductive: business activity dashboards on the PC, network monitoring alerts via e-mail, just-in-time manufacturing systems. The idea is to help people make better decisions.

But do they?

“Too much information freezes the human mind,” says Accenture’s Bell. “When there are too many choices, a normal human being won’t be able to make a choice.” Is business reaching the tipping point of information overload?

Malcolm Gladwell, in his best-selling 2005 book Blink, which looked at how we process information to make decisions, described doctors misdiagnosing heart attacks in the emergency room because they were attempting to gather too much information, in too many cases sending home patients who were actually having heart attacks and admitting patients who were not. According to Gladwell, the doctors were gathering and considering far more information than they really needed “because it makes them feel more confident…. The irony, though, is that that very desire for confidence is precisely what ends up undermining the accuracy of their decision. They feed the extra information into the already overcrowded equation they are building in their heads, and they get even more muddled.”

An analogous situation is happening in IT departments. A 2006 survey by Netuitive, a real-time analysis software vendor, found that 41 percent of respondents in larger organizations receive 100 or more alerts per day, of which at least half (more in most cases) are false positives. Of the 195 IT organizations surveyed, 39 percent said that they either intentionally set thresholds above optimum levels to avoid excessive alerting or turned off their alerting functionality completely in response. In both cases, the system has been rendered pretty much useless.

Delta Apparel’s Smith has seen how too many alerts can create a choke point. For example, when a Delta staffer creates an order, an e-mail goes out to a set of other employees who need to know. “In theory, that’s a good practice,” Smith says. “But if we’re entering a thousand orders a day, and I’m a recipient, there’s no way I could ever manage that information. No one can respond to a thousand e-mails a day.”

But it’s not just e-mail alerts. Typically, decision-makers in manufacturing companies think that they want daily updates for their material requirements planning (MRP) system so they can make changes to plans and update forecasts and inventories on a daily basis. And vendors selling MRP systems (surprise!) usually agree with them. But, “Guess what: Reality says I can’t deal,” Smith says. He says that companies simply cannot make those decisions on a daily basis because there’s too much information and too much flux. “It’s one of those concepts that’s candy in the sky,” he says.

Put another way, business managers and other decision-makers are sometimes surprised and overwhelmed by the velocity and volume of data when a real-time system fires up. “You get what you ask for, not what you expected,” says AMR’s Hagerty.

How to Calibrate Real-Time

In 2005, two researchers, one from Georgia Tech and the other from the University of North Carolina, set out to examine whether increasing the frequency of real-time data updates “enhanced performance.” Would they be able to more quickly respond to changes in the environment and see the consequences of their actions? wondered Nicholas Lurie and Jayashankar Swaminathan.

What they discovered was that managers who received more frequent data points were making more poor decisions. “The danger of real-time data is that it may come to you at a frequent rate, maybe every hour, and if you respond to that data, if there’s some random event and you treat the random event as systematic, it could really throw you off,” Lurie says.

Yossi Sheffi, director of MIT’s Center for Transportation and Logistics and author of The Resilient Enterprise, finds no fault with real-time data, only in the way people use it.

“The question is not, Is real-time information bad or good? There’s only good in it,” Sheffi says. “The danger [of real-time data] is if you would react too fast and not wait for the trend to reveal itself.” As an example, Sheffi suggests that Procter & Gamble should not start making inventory or planning decisions on Tide sales at Wal-Mart based on data they receive every five minutes.

“You don’t want to react to someone who came in and bought five boxes,” he says. What you should do is look for trends in product sales combined with historical data—for example, during the last few days or weeks—and correlate that with other event-type data, such as in-store promotions or weather information that may affect sales. “You have to use it smartly,” Sheffi says, noting that some companies right now are better at this than others.

At Priceline.com, Rose seems to have found that sweet spot for delivering real-time data to both IT and business users. The company was founded in 1998, and since then Rose says a culture of real-time data has flourished. “From day one we’ve always been about collecting business metrics on the fly,” he says.

When asked why the business prefers hourly reports, he answers that while he could offer them minute-by-minute data (the BAM system has that capability), they’ve discovered that anything under 15 minutes most likely wouldn’t be sufficiently significant to constitute a trend or something that demands their attention.

“Hourly data is good enough,” Rose says.

The Sock Market

International Legwear Group makes a lot of socks. As the third-largest sock-maker in the United States, ILG runs on a weekly cycle for processing orders from its customers, which range from Wal-Mart on down to Mom-and-Pop stores. And for the most part, ILG’s planners and forecasters are able to satisfy their retailers’ demands for socks and sync up those demands with ILG’s global suppliers.

But if you know anything about the sock market, you know how difficult ILG’s planners’ jobs can be. “Sock sales are very dependent on the weather,” says Alex Moore, ILG’s CIO. “If it stays hot into the fall, our sales do not start climbing.” Or, if a cold snap hits the South, sock sales will jump for as long as it lasts. “I don’t care how good your software forecasting system is, you can’t predict when it’s going to get cold,” Moore says.

The unpredictability and randomness that can throw off a company’s supply chain system is called noise—whether it’s an ice storm, earthquake, e.coli outbreak or terrorist attack. And what noise does is make computer systems, especially real-time systems, nervous. That, in turn, amplifies the perils of demand and inventory planning. “A nervous system is one in which you try to react to every little thing,” says Moore. “If you change your plan with every thing little thing that happens, that’s a bad thing.”

It therefore becomes critical that companies develop some type of strategy that can filter out noise and nervousness. Overreacting to sudden and random upticks in sales can produce a deadly chain reaction in the supply chain, with each supplier downstream from the first increasing its orders and supply requirements because it wants to have enough inventory to comply with the illusory rising demand. This is called the bullwhip effect.

In 1997, Stanford’s Lee cowrote the seminal article on it—“Information Distortion in a Supply Chain: The Bullwhip Effect” —and the now-famous example of the variability (and challenges) in demand planning for P&G’s Pampers product remains the best-known example. “You have to be able to distinguish between noise versus a real systematic shift,” cautions Lee.

The Beer Facts

The fifth business day of every month used to be a significant day for Heineken USA. On that day, employees got their first glimpse of the previous month’s sales data, which provided a snapshot of how well the U.S. arm of the Dutch brewer was doing. “This company lived and died by monthly data,” says Director of IT Schillat. That the business users had to wait a full month to view Heineken’s key performance indicators didn’t bother the staff. It’s just the way the beer industry operated, they thought.

But by 2000, Schillat knew differently. Industry heavyweights Budweiser and Miller had begun investing millions in building real-time connections to their distributors. Schillat turned to Vermont Information Processing, which had already made inroads in the marketspace with its supply chain products. In less than a year Schillat could stream distributors’ daily sales (prices, quantities and which retail stores the beer had been shipped to) to the business. “This was huge for us,” she recalls. “All of the sudden, we had daily sales.”

Though this wasn’t what one could call true real-time data, it was as close to real-time as Heineken had ever seen. But not everyone liked what they saw. “People didn’t have a lot of faith in it,” Schillat recalls, because they were still receiving their good old monthly feeds, and the two sets of numbers didn’t always jibe with each other.

Since then, Schillat and the business users have been working to design new processes to accommodate faster data streams. But there’s much more work to be done to deal with the users’ training and development, and with figuring out just what data is most appropriate and actionable and how business users should respond.

Game-Planning for Real-Time

That, of course, is a key part of the CIO’s job, no matter the system or project: facilitating a conversation about what the business truly needs, and where and when real-time, near-real-time or right-time data feeds are appropriate.

“This is not something the business can have a couple of meetings about, define and outsource to the IT department,” says Accenture’s Bell. “It’s a conversation about the source of data and how you use it, and what may be absolutely ideal will be completely different in 12 to 18 months’ time. It’s a continuous conversation.”

AMR’s Hagerty advises CIOs to ask these questions: How do people manage information flow in their part of the business? Do people really need real-time everything or just frequent refreshes? How should IT respond to support the business’s data needs?

“You need to rationalize this up front,” Hagerty says. “When someone says, ‘I need real-time data,’ IT should ask: ‘What are you going to do with it?’ Sometimes business users don’t like that, but IT needs to know.”

In the end, real-time data is only as good as the uses it’s put to and the processes that support its use.

“I don’t think there’s danger in trying to achieve [a real-time environment],” says Delta Apparel’s Smith. “But you could easily spend a lot of money trying to get to that candy in the sky, and then realize that the information overload is too great.

“Not all people understand the impact of real-time information.”