How big data will overcome big challenges in 2016 to still stay on top

First, some fun facts: 90 percent of all data in the world has been created in the last two years; by 2020, every human will be generating 1.7 megabytes of data every second; every minute 570 new websites are launched. These are just a few of the many data-driven reasons why analysts and scientists are touting big data to become even bigger than what it currently is. But come 2016, will enterprises be serious about leveraging big data to advance their business goals? Well, maybe.

Since the inception of the term “big data,” its application has been patchy. In 2008, Google came up with Google Flu Trends, which could determine the flu patterns across the globe, depending on the entries in the search engine. This was the perfect example of utilizing big data to analyze something to have a greater impact. Which government or health organization would not want to know where the next epidemic will break?

However, this did not turn out the way it should’ve. In fact, Google failed miserably at this—at the peak of the 2013 flu season, it deviated from the predictions by an astounding 140 percent. Very quietly, this project was terminated and it became an example of how big data analytics can go wrong.

If we go back to the past, to the origin, we see that big data analytics is just an evolution of analytics. Analytics has come a long way—from being used in World War II to develop a series of groundbreaking mass data-processing machines to being used in the Human Genome project to map all the genes of human beings.

But now, what makes big data different from the earlier way of decision making? At its nascent stage, decision making was mostly based on a sampling technique. Because of the lack of time, data, and resources, we had to extrapolate data and predict a pattern. But today with the amount of data from all sources, sampling is no longer required—rather you would have to take into account everything that you have, which is a lot!

To those outside the enterprise, it is assumed that big data and analytics go hand in hand; however, there are subtle differences between them such as difference of scale, measured by volume, variety, and velocity, as well as the technologies used.

As such, Bhavish Sood, research director at Gartner, feels that big data has various technology components to it such as text analytics (to scan a broad range of text-based sources and formats), video and audio tools (for indexing, search and analytics), and social network analysis (to analyze connections and sentiments).

The Big Challenge

But now after all these years, is big data still crippled with challenges? How exactly did big data fare in 2015?

If you have a small amount of data with a limited data set, the chances of errors are minimized (hopefully). But with the advent of social media and the explosion of the Internet, our data has increased by humungous factors, thus causing an increase in the issues that we have to deal with. The problems that were present at the inception are still present, however the magnitude is lesser.

Looking at the way organizations function when it comes to big data analytics, Mahesh Bhatt, Vice President-IT at Kirticorp, poses a simple question, “Are organizations ready to handle big data?” He has his doubts.

“As some analysts and companies have pointed out, data is the new oil. Data from customers, partners, employees and products need not be just harnessed to drive efficiency and improve experience but data also presents an opportunity to create new products and services,” said Malay Shah, Director, Advisory Services, Technology practice at EY.

But here lies the challenge in adoption. Shah feels that some companies haven’t been successful with their small data initiatives, while others continue to work on gaining returns on their operational and analytical tools in which they have already invested.

Shah also said that some of the big data use cases may not apply to all companies. However, he said that companies will need to start looking at big data technologies to serve them as some of the use cases on predictive technologies, machine learning and leveraging unstructured data are becoming popular.

Security issues have always plagued most of the big data solutions. Securing organization as well as customer data is of topmost priority. But failing to comply with various data protection can cost dearly to organizations.

In the Indian context, to protect data, we do not have comprehensive privacy legislation. However, the Reasonable Security Practices and Procedures and Sensitive Personal Data or Information Rules 2011 formed under section 43A of the Information Technology Act 2000 define a data protection framework for processing digital data. Big data practices will impact several provisions in these rules.

Moreover, with the increase in data, another problem arises: Where will you store this data? According to predictions, by 2020, the global data will see a 50-fold increase, but at the same time hard drives are predicted to grow only by a factor of 15. In addition, compared to the previous years, storage space costs have also not got down quickly.

This is not it. There is more. The presence of false positives is also a matter of concern. “This may be because of jumping to conclusions based on not enough information or not getting the right information in the first place,” said Charles Race, EVP, Worldwide Field Operations, Informatica. “Most of the organizations face volume issues, and when you try and fix this issue, you can have an awful lot of false positives.”

Also, most of the times it is the data that is unclean. To make something meaningful out of unclean and unstructured data is a task in itself, said Bhatt.

“Figuratively, in every one ton of dirt, we get one gram of diamond—this is the challenge we face,” he said. “Multiple types of data in multiple formats also lead to a loss of data which hampers the analytics process.”

By 2020, the global data will see a 50-fold increase, but at the same time hard drives are predicted to grow only by a factor of 15.

Hyping big data

A recent IDC forecast shows that the big data market will grow at a 26.4 percent CAGR to $41.5 billion through 2018.

Furthermore, in a recent report by Gartner, globally 45 percent of organizations have already invested in big data technologies—an increase of 5 percentage points, while 31 percent of organizations plan to invest over the next 24 months, down from 33 percent in 2014. The number of organizations with no plans to invest has decreased from 24 percent in 2014 to 21 percent in 2015. Similar to every other year, the investment in big data in 2015 has been increasing but the growth has not been rapid.

“We are seeing the second wave of big data adoption, but it is slower and more deliberate than the first. This second wave is less enamoured by technology, instead being driven by business value. This is also reflected in our survey data,” said Sood.

“This year begins the shift of big data away from a topic unto itself, and toward standard practices,” said Nick Heudecker, research director at Gartner, during the release of the report. “The topics that formerly defined big data, such as massive data volumes, disparate data sources and new technologies are becoming familiar as big data solutions become mainstream.”

Gartner also had come up with a report authored by Betsy Burton, in which the hype cycle for big data was eliminated. In the research titled “The Demise of Big Data, Its Lessons and the State of Things to Come,” the firm said, “Hype Cycles consider any adoption trend that goes beyond 20 percent of the wider IT market to be past hype and entering into early market definition.”

“Hype is now being replaced by practicality, because the technology and information asset types offer new alternatives that are most often additive or complementary to long-standing, traditional practices. This type of overhyped evolution will happen again. When it does, information and analytics leaders should recognize it for what it is,” Burton said.