How to Profit From the Ultimate Big Data Source: The Weather

That itch in your throat and those watery eyes? Merck, which makes the allergy pill Claritin, anticipated your hay fever and–a year ago–started making plans to capitalize on it. With a subscription to specialized weather forecasts, Merck knew way back last July that this March would be unseasonably cold in most of the U.S., leaving many allergens dormant. Then, quite quickly, May would bring lots of warmth, pollen and spores.

Merck shared its weather intelligence, based on temperature and moisture data correlated to customer behavior by ZIP code, with Wal-Mart. Together they decided to boost promotions and supplies of Claritin and other allergy products at the time when you were desperately ready to buy.

“The upside is potentially millions of dollars in additional sales,” says Debbie Sonnentag, Merck Consumer Care’s director of category development for Wal-Mart.

Companies in all corners of the economy are factoring weather data into their business strategies, hoping to turn a profit on Mother Nature. Sears monitors weather nationwide from a crisis command center, figuring out how to stock enough snow blowers in a winter storm and air conditioners in a heat wave. Home insurer EMC Insurance analyzes hailstorm history to catch false claims. Westar Energy in Kansas schedules power-line repair crews with an eye on severe weather in other states, in case distant outages require their help.

DHL Express, a division of the $73 billion global delivery company, uses weather data to make minute-to-minute decisions that affect 3,000 flights per day worldwide. Weather, says Travis Cobb, VP of hubs, gateways and network control for DHL’s Americas region, “is the million-dollar question.”

Yet weathermen have a bad reputation for a reason: Getting it right is hard. The Weather Channel, for example, every day processes 20 terabytes of data about wind, rain, sleet, snow, temperature, tornadoes, air pressure, moisture, earthquakes, hurricanes, wave heights, lightning and ice, says CIO Bryson Koehler. And much more. Plus business customers can buy custom information created by analytics. Insurers might want to see rain accumulation modeled against auto insurance claims. Pharmaceutical companies can buy maps of air stagnation patterns to understand patient respiratory distress.

Consumer packaged goods companies, logistics businesses, restaurants, railroads, amusement parks, financial services firms–the list of weather watchers goes on. Some analyze how past weather influenced customer behavior, hoping to discover useful tidbits for the next marketing campaign. For others, anticipating future weather can reveal worthy risks to take and ways to avoid problems that competitors don’t foresee.

“Weather is the original big-data problem,” Koehler says.

Hot Competition

Weather provides no shortage of business opportunities because it affects everyone. Demand will probably never ebb. As a result, the world of weather data is competitive, niche-y and expensive. Google recently patented a robotic device that analyzes weather and a personalized navigation system that suggests changes to driving routes based on the weather. Some players even sell weather derivatives that companies buy to offset expenses incurred as the result of weather.

Dominating the market for weather forecasting services are The Weather Channel and AccuWeather, with market shares of 51 percent and 14 percent respectively, according to researcher IBISWorld. Both provide consumer weather forecasts and both vie fiercely for big-name business customers. The Weather Channel has Home Depot and American Airlines; AccuWeather has Lowe’s and Union Pacific.

Smaller firms, such as Weather Trends International and CustomWeather, specialize in corporate accounts. Boutique weather companies focus on narrow markets, such as agriculture or energy. IT vendors such as CoreLogic and Planalytics offer weather information with analysis tools and services to integrate data from ERP, manufacturing and other IT systems.

Most of these companies scoop up free data from the National Weather Service, whose mission is to protect life and property and enhance the national economy. But that’s just a starting point. The companies supplement that information with data collected from their own systems and sensors, as well as from niche players. Then they run it through secret algorithms, interpret it and create new products.

For example, AccuWeather’s Weather-Triggered Marketing service supplies a stream of data to help companies choose the best periods to offer a discount or increase inventory, for example, or to help them post social media updates that make sense given local weather conditions. Facebook friends in Miami don’t want to be bothered with Wisconsin’s ice storm. Custom services include analyzing item-level point-of-sale data against weather events to see patterns. AccuWeather says it can find relationships among local sales data and up to 200 weather variables. One gem: the top-selling food during hurricanes is blueberry toaster pastries.

This analysis requires a different kind of IT department. “Research and development is a huge piece of IT here,” says Steve Smith, chief digital officer of AccuWeather. Smith stacks his staff with data-scientist types who become experts on specific industries, such as railroads, retail or commodities trading.

The posturing by these companies can be entertaining. In marketing material touting its forecasts of February’s snowstorm in the Northeast, AccuWeather dissed rival Telvent DTN for describing what turned out to be a record-setting storm as “brisk.” The Weather Channel, which holds 77 patents, proclaims, “The only thing more powerful than the weather is our ability to help you profit from it.”

Still, there’s lots that can’t be foreseen about weather, and companies rightly worry about it.

Satellites 23,000 miles up and sensors all over the ground–on portable weather stations, buildings, vehicles, mobile devices–take millions of observations at regular intervals that are analyzed by public and private weather organizations. But the band of space between Earth’s surface and orbiting satellites is where the wild stuff happens. Two thousand to 10 thousand feet up, weather changes. Winds die or cold fronts break up. Precipitation evaporates or the air warms.

Conditions in the lower atmosphere make all the difference, says Randy Bass, a member of the aviation weather research team at the Federal Aviation Administration. The three to five inches of snow a broadcaster predicts for Wednesday turns into a foot dumped on an area where the temperature unexpectedly dropped 10 degrees. Then everyone says the weatherman blew it. Airline pilots and weather balloons transmit reports from that mystery band, but they aren’t enough. “There’s no good way to get data from there in the amount and with the timeliness we need,” Bass says.

All of which makes weather a favorite scapegoat for companies that miss financial expectations. In recent conference calls with Wall Street, various executives cited weather as the reason behind disappointing sales of sandals, green beans, doughnuts, books, airline tickets and auto parts.

In its latest annual report, FedEx issues a broad warning to investors: “We are particularly vulnerable to the physical risks of climate change that could affect all of humankind, such as shifts in weather patterns and world ecosystems.” Nothing like covering every base.

But with so much data available, often for free from the federal government, and with so much computing power on hand to crunch it, companies won’t be able to claim to be surprised by the weather anymore, says Al DeChellis, a supply-chain consultant and former VP at Alberto-Culver. “If you’re selling seasonal products and you’re not using weather data, you’re not doing your job.”

Seeing Through the Fog

At Alberto-Culver, which is now owned by Unilever Group, DeChellis pioneered the use of weather data 8 years ago to sell one product: Static Guard. The spray can of chemicals to get rid of annoying static cling is a seasonal product, but not in the way people assume, he says. Cold weather isn’t the culprit; relative humidity is. If humidity sinks to 50 percent for at least three weeks, static electricity builds to levels that make a skirt stick to a woman’s pantyhose.

DeChellis’ team worked with Planalytics to correlate dry air to consumer sales by geography and convinced some retailers to pay more attention to Static Guard, he says. In the Chicago market, for example, salesmen sometimes convinced retailers to promote Static Guard two or three times per year, rather than just once. “We considered it a major win,” he says.

Scott Jean, chief actuary at EMC Insurance, is something of a weather detective. He knows hail is a big deal in the Midwest. So are homeowners’ claims of hail damage: Hail accounts for about 30 percent of the company’s homeowners’ claims, and payouts have been increasing, he says.

As the recession hit the U.S. in 2009, Jean noticed an uptick in hail claims. He suspected the involvement of storm-chasing opportunists who call on homeowners after a bout of hail and, for a fee, inspect their roofs and help file damage claims. “They will find something wrong with a roof that could have been there 10 years before we insured the homeowner,” Jean says.

With hail data supplied by CoreLogic and Doppler radar material pinpointing hailstorms to specific dates and locations, EMC Insurance can catch errant claims. But the insurer drills down further to consider the size and intensity of a hailstorm as well as the age of the roof. “Pea-size hail won’t do damage to a good roof, but a storm chaser will say otherwise,” he explains. “We can argue reasonably that it didn’t occur.”

Whether the guy who inspected the roof is dishonest or just mistaken, EMC Insurance doesn’t want to pay bad claims. Homeowners insurance isn’t usually profitable for insurers, but Jean thinks it can be, with judicious and methodical analytics, of which weather is a key piece. “We should be able to be profitable without overcharging consumers,” he says.

Advances in analytics technology in the last several years, combined with plentiful weather data, allow such analytical creativity and exactitude, says Tom Davenport, senior adviser to Deloitte Analytics and professor at Harvard Business School and Babson College.

IT organizations are good at excavating data from internal systems and using technology tools to combine it with outside information. But some IT groups aren’t as good at understanding the context in which the material will be used, he cautions. With weather data, such insight is important. Not only do CIOs want to facilitate access to useful data, but they can also help the company create new products or services. Or help streamline operations.

No one can predict the weather correctly all the time, but getting a peek at what’s coming can help the business, Davenport says. “You can’t control it, but you can control for it.”

When Disaster Strikes

Sears faces troubles in retail overall, but the $39.8 billion national chain brought its size, experience, business relationships and technology to bear in its successful response to the Nemo winter storm in February.

As Nemo developed, the crisis command center at Sears headquarters jumped into action, says Raj Penkar, president of supply chain at Sears Holdings. Established in 2010, the command center runs seven computer monitors tracking various data and information. From local and national news feeds, Google Earth and other sources, Sears created maps of the affected areas, color-coded according to expected severity. Red is bad, and there was a lot of red on Sears’ Nemo map. Staff members from the risk management, facilities, corporate communications, inventory management, logistics, transportation and IT departments together made decisions about employee safety and store operations, Penkar says.

As reports about Nemo made it clear the storm would be a whopper, Sears scanned its inventory, store and warehouse systems to get the latest data on product stock levels. The crisis team figured out what extra inventory would be needed and how to move it closer to the trouble spots. Sears put extra snow blowers and generators, among other “recovery” products, in or just outside affected areas, ready to go to individual stores as Nemo passed. In some areas, once the roads were clear, Sears asked suppliers to truck inventory directly to Sears stores in New England, bypassing the usual stops at regional distribution centers. Of course, Home Depot and other competitors were doing the same thing, Penkar notes.

“When something like this happens, everybody needs trucks and vendors,” he says. “Not to be negative, but we all try to help customers and at the same time, we’re all trying to run a business.”

In the days leading up to Nemo, Sears had a supplier reroute four truckloads of generators in Atlanta up to the Northeast. A supplier in Wisconsin held six trucks of generators for Sears to pick up. The evening before Nemo hit full-on, Sears issued a press release about stores stocked with the right equipment. The release also noted that people could expect two inches of snow per hour and winds of 50 mph, and that Sears offers convenient in-store pickup of online orders.

Each day, Sears managers received several “LogHot” email alerts showing the severity of the storm. The alerts included maps of the storm’s predicted path and the number of stores and distribution centers that could be affected. Key personnel carried special cellphones reserved for storm communication. Field employees updated a private wiki with on-the-scene information. Entries from staff at Sears’ Gouldsboro, Pa., distribution center, for example, included notes on roads that had been shut down, regional delivery centers that had been closed and stores whose delivery trucks weren’t unloaded due to the storm.

Sales figures from that period are one way to assess Sears’ agility in handling the storm, Penkar says, but more importantly, “The right product [was] there, that our customer needed.”

DHL Express prides itself on its intense focus on how weather affects customer satisfaction. DHL analyzes feeds and data from, among other sources, the National Weather Service, partner airlines, airports, AccuWeather, The Weather Channel, and a hive of organizations that specialize in weather as it affects flying. Its three network control centers in Cincinnati, Germany and Hong Kong are staffed 24/7. The data never stops.

Of course, Mother Nature can be tricky. “Sometimes you can look at it too closely and make decisions off one 5-minute data set, and the next thing you know, the weather’s gone,” says Mark Becker, director of the network control group at DHL Express.

What guards against that? “Experience,” Becker says. Most of his duty managers and controllers have 20 or more years on the job.

Visibility, as you might imagine, is DHL’s watchword as it manages the 3,000 flights per day carrying packages to its customers in 220 countries. Betting correctly that airports would be a mess in the Northeast after Nemo, DHL flew its planes out of major airports before the storm, keeping them at its hub in Cincinnati. The contingency routing decisions were based on experience, terabytes of data, and IT systems to model scenarios. The process is worlds better than it was years ago when weather information was scarcer, says Cobb, the VP of hubs, gateways and network control.

In the past, “you would see the 6 o’clock news, the 11 o’clock news and make decisions,” he says. Even now, working with incomplete and sometimes wrong information can’t be helped, he says. “Each situation is unique. What we try to do is mitigate risk.”

DHL faced its most unusual weather challenge in 2010.

A large volcano in Iceland started to erupt in April of that year, after 200 years of quiet. Soon a giant cloud of ash covered surrounding countries, at times up to seven miles high. Government officials shut down most of northern Europe’s air space for eight days in April while the cloud lingered, and sporadically in May as parts of it drifted back. Worldwide, 104,000 flights were canceled during that period, up to 19,000 per day.

As the cloud broke up and parts of it drifted unpredictably, air safety officials had to react to the changing situation. Even when they opened the skies to planes, DHL had to determine for itself whether and how to fly. “That was dynamic,” Cobb says, understating the tension of those days.

A DHL plane was one of the first to fly into western European airspace when the sky cleared, a move worth a lot to its reputation and, therefore, its business, Cobb says. “The passion of this company is to be the last out, first in,” he says, adding that the company’s market share increased afterwards.

Removing the Emotional Factor

Making the best decisions in the moment with imperfect information challenges even the smartest managers, Davenport says. You try not to sully the statistics with biases or inaccuracies as you use them to conjure scenarios and play out ideas, he says. But that’s a particular danger when using weather data, says Koehler, CIO at The Weather Channel. “Telling the story of weather,” as he puts it, is often how even the most accurate forecasts get mangled. The Weather Channel has written a lexicon to translate weather numbers into language that consumers of the information can understand. For example, when the probability of precipitation is 60 percent or more, forecasters say “likely.” When it’s below 60 percent, they say “chance.”

Back at Merck, Sonnentag cautions her team not to get carried away adding subjectivity to the weather data they use. People “tend to be emotional” about the weather, she says. Before Merck used weather facts, people would attribute up or down sales to whatever local weather they happened to experience, she says, “even though that was based on a single market, sometimes a single day.” Hard data removes subjectivity.

Meanwhile, she hopes her second bet, on a prolonged allergy season this autumn, will pay off. For about the same amount of time that spring weather was delayed by this year, nice days are expected to continue into the fall in the Northeast. Look for a warm October, they say.

So Merck plans to promote Claritin more than it normally does in the autumn months. Sonnentag has faith in her weather data. Planning for the summer core of allergy season “is easy: Have everything everywhere,” she says. “But stepping into and out of the season, we’re really finessing that this year.”

The risk is that weather will shift and Merck will have overstocked Wal-Mart’s distribution centers. Merck could lose money if it has to truck a lot of unsold Claritin back home and throw out expired lots, she says. She’ll know in a few months.

Coming Soon: Personal Weather Forecasts

Real-time analytics will enable customized weather reports on smartphones, in cars, and on your refrigerator

Lots of mobile phones come pre-loaded with weather apps from AccuWeather or The Weather Channel. But the companies want to go beyond local forecasts, which anyone can get by typing in a ZIP code. They want to combine sensors and geolocation technology to provide personal weather reports.

Using real-time analytics, weather companies intend to combine data about where someone is standing with other information, such as the air pressure and temperature, collected by sensors in the phone. The result will be a forecast tailored to the mobile phone customer as he moves around during the day. He could also query the data to find out, for example, the chance of rain at the company softball game he’s about to play.

First, however, systems for data collection and the speed of analytics will have to improve to make personal weather forecasts possible, says Steve Smith, chief digital officer of AccuWeather. But his company is working on it. Smith also envisions collecting weather data from cars in motion, such as when wipers go on and when anti-lock brakes kick in.

“That’s how you get to something actionable and relevant to you,” Smith says. “We change weather from a commodity to something of value.”

AccuWeather also works with appliance-maker LG Electronics to embed a weather app in high-tech refrigerators. Get your morning OJ and the day’s weather all in one spot.

Readings from mobile devices could also make regional forecasts more accurate. Participating cellphone users would form a network of continuously updated weather data to fill gaps in existing weather-collection systems. For example, the average thunderstorm is eight to 10 miles wide, but observations from weather stations on the ground are typically 50 miles apart, says Randy Bass, a member of the aviation weather research team at the Federal Aviation Administration. A tighter network would make forecasts better, he says.

Personal weather forecasts are “very feasible,” Bass says. “Is it perfect? No. But it will be better than watching TV and guessing what conditions will be at a certain time.”

Kim Nash is managing editor of CIO Magazine. Follow her on Twitter @knash99.

Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.