One-fifth of organisations store more than 1 petabyte of data

Data growth across Australian businesses is showing no signs of slowing with almost one-fifth of organisations responding to a survey indicating they are managing more than 1 petabyte of information.

This was a key finding of research on the challenges organisations face around big data, released this month by BigInsights and CIO Australia. Around 40 per cent of the 100 organisations surveyed also claimed to have more than 100 terabytes of data with a third of these companies indicating that more than half their data is unstructured.

The results indicate that there is a growing data issue within Australian organisations when data volumes are compared to previous years.

“What this highlights is that despite its size in population and economy, Australia is no longer immune to the forces that drive big data,” according to the research report.

The majority of data inside these organisations is relational data, however the amount of unstructured information was significant, the report said.

Almost one-third of respondents said between 20 per cent and 40 per cent of data in their organisations was unstructured, followed by 17.5 per cent indicating the figure was as high as between 41 and 50 per cent.

The figures suggested that the majority of data that exists in most organisations is still relational data. However, the growth in unstructured data should be a concern for organisations in terms of how they contextualise it in relation to structured data, the report said.

The research also found that storage is no longer a chief concern for CIOs thanks to falling storage costs and the emergence of platforms to manage large volumes of data. However, organisations need to consider what they intend to do with such large volumes of data and how to extract value from it.

Benefits of big data analysis

Almost two-thirds (65 per cent) of survey respondents indicated that the top business benefits of big data was improving customer insights.

The report said while improving customer insights was not a new pursuit, the difference now is that the sources of data and analytical tools have become much more sophisticated to provide deeper and much more immediate insights.

Second on the list of benefits was improved decision making at 61 per cent, which indicated much more data-driven decision making as opposed to intuition and gut feel, the research said. Competitive advantage was third highest at 55 per cent.

Shayum Rahim, director of research at BigInsights, said several lines of business – marketing, HR, customer service, and logistics and operations – stand to benefit from big data analysis tools.

He said this could mean organisations could be running informal and siloed projects, leading to a host of negative impacts such as disparate systems and grave security concerns.

“This is where CIOs must assert themselves,” he said. “The CIO needs to support adoption of big data and other technologies within the organisation but needs to ensure it is happening in an orchestrated way.”

Rahim said the CIO needed to be the conduit between all lines of business and the curator of the technological landscape within the organisation.

“Of course for some, this means a major cultural change,” he said.

Tightening budgets also means the CIO can become quite constricted in how they can meet the requirements of the business, said Rahim.

He suggested that CIOs need to be mindful of open source technologies available in the market today for data analysis, which have helped reduce the IT costs and provided more options.

The research found that there is a growing interest in open source database technologies with 52 per cent of respondents indicating they used or were planning to use the MongoDB database. This was followed by Cassandra and Amazon (DynamoDB) at 24 per cent; and Google BigTable (21 per cent).

“Start-ups and new pure play big data vendors are beginning to pose a challenge to traditional vendors in terms of the sophistication of the solutions, and more importantly, the lower cost of entry in acquiring big data and analytics technology,” said Rahim.

Follow CIO Australia on Twitter and Like us on Facebook… Twitter: @CIO_Australia, Facebook: CIO Australia, or take part in the CIO conversation on LinkedIn: CIO Australia

Follow Byron Connolly on Twitter:@ByronConnolly