How to Avoid Big Data Spending Pitfalls

Big Data is a powerful lure, promising to turn the massive and ever-increasing volumes of data inside an organization into a pool of intelligence that promises deep, actionable insight into every aspect of a business. However, that lure can lead you into an expensive trap if you don’t plan carefully.

“Big Data has big spending risks,” says Jeff Muscarella, IT spend management consultant with NPI Financial. Muscarella warns that Big Data projects can easily ring up seven-figure price tags after you finish paying for the hardware, software and services, and sometimes the glowing business cases presented by vendors lose their luster when you look closely. “A lot of times, when you pull them apart, they’re not as rosy as they seem,” he says.

That’s not to say that harnessing the power of Big Data is a mistake, Muscarella explains. But it does mean that organizations seeking to base their decisions on data need to start by gathering real data on how a Big Data project will benefit the business.

“This is not just new technology,” Muscarella says. “It’s new technology solving a business problem that we often haven’t proved. That’s important for CIOs to keep in mind. The business is going to be coming to them with all sorts of half-baked ideas for what they can do with Big Data. They have to ask: Will it really drive revenue? How and for how long? What will it take to build it? They need to make sure they have a crisp focus on the mission; that it is going to have a return on investment.”

For Big Data, Fire Bullets not Cannons

When you’re exploring a Big Data project, don’t dive in head first, Muscarella warns. Start with open source tools like Apache Hadoop and build a test case.

“You want to really pilot these things,” Muscarella says. “Pick something that’s manageable. Start on a small scale to prove your hypothesis. For instance, if we could mine this sensor data or these Web clicks or these purchasing habits, would what we do with these results improve our business.”

“Don’t get trapped into building the infrastructure yet,” he adds. “Prove it first and then go back and architect your solution. Assume that however you solve the problem, you’re probably going to throw it away and start over. That’s OK because at least you proved the business need before you spent a lot of money.”

Once you proven the business need, it’s time to look at the infrastructure required to manage Big Data. Big Data projects scale to petabytes and potentially exabytes of data, so making sure you get your storage infrastructure right is essential. Muscarella says that despite vendors’ arguments in favor of standardizing on one storage provider, it’s better to leverage storage virtualization technology to introduce competition in low-risk, non-strategic areas of your architecture.

“You don’t want to standardize on just one vendor,” he says. “You might want to do some of it in the cloud and you might want to do some in your internal data centers. You want to keep your options open. Once you get locked in, you truly are locked in.”

He points to hospitals that he’s worked with that standardized on a single vendor. The initial deal seemed good, he says, but when their upgrade cycle came up a few years later, they didn’t have any choices and the offers they received were far different than they would have been otherwise.

“The value you get varies widely depending on whether you have any options,” Muscarella says. “Use a multivendor strategy. Once you identify your upgrade cycle, make sure you go through all your agreements to make sure that in the vendor’s mind, you have the option to switch.”

Additionally, be careful with support for your storage. Make sure your support pricing is in a fair range, and be rigorous about identifying decommissioned hardware in your storage portfolio and negotiating the cost of supporting that hardware out of your support agreements.

Pay for an Analysis When Buying Data Mining, BI Software

Data mining and business intelligence software and services are most often sold in the context of a business case, which means that vendors will fall over themselves to provide you with a free business case analysis, Muscarella says. They’ll want to bring consultants to your site for several days, speak to your business owners and help you understand what they can bring to the table.

“They’ll make you feel very good about spending,” Muscarella says. “But those business cases are often full of holes, or there are lots over very optimistic assumptions.”

It’s better, he says, to pay them for the analysis or hire a third party to conduct the analysis. That gives them liability if the analysis proves mistaken and gives you a much better chance of receiving an honest, complete assessment.

Beware of the Big Data Bundle

Whether you’re buying hardware, software or services, avoid the all-you-can-eat agreements, Muscarella warns.

“We always tell people to beware of the bundle,” he says, noting that vendors often offer a deal in which customers can use any of the tools in their toolbox for one flat fee. “After a year, they’ll look at how many tools you’ve installed and are using, and they’ll just charge you ongoing maintenance for what you use. We find those kinds of arrangements often lead customers to deploy far more tools than they need. Three years later the vendor comes back and you’re using three times more than you need to be. And then you pay maintenance for three times more than you need.”

Instead, make sure to ask for time and materials quotes in addition to fixed bid quotes. He notes the time and materials bid can provide valuable insight into how a vendor looks at a project and how many hours they think it should take by resource and task. That information gives you the capability to benchmark pricing, margins and discounts among your vendors, providing leverage during negotiations.

Thor Olavsrud covers IT Security, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at tolavsrud@cio.com