Five rules on mining Big Data

See also: Splunk in CIO UK’s 20 companies to watch in 2012

Since joining Splunk just over a year ago as CIO, I’ve discovered that we are all riding a new wave of innovation that enables us to harness Big Data to bring a new kind of business intelligence to our customers: an operational intelligence.

There is a tremendous opportunity for CIOs and IT leaders to lead this charge.  Big data is a reference to volumes of loosely structured information that cannot be reliably managed via legacy relational database and BI tools.

The massive volume of click stream data pouring into companies’ web logs combined with all the machine data spewing forth from every part of the infrastructure represents a chance to correlate customer behavior from the user interface all the way back to the network to create a single unified view of our customer’s experience and the platform of delivery.

Some initial recommendations:

– Examine the primary objectives of your company Align the big data opportunity to breakthroughs the business is looking to achieve in a key market, particularly those driven via a digital online element.

What I’ve seen is that any customer-facing aspect of your website and related service, whether ecommerce, online support, or other digital channels is ripe with opportunity for understanding customer behavior and optimising online results with big data.

– Educate business peers to what’s possible by mining big data They will be amazed at what you can bring to the table to help them improve top line growth, customer retention, and more.

Marketing and support executives are a great audience to start with, as I’ve found them quick to grasp what may be possible from real time data analytics.

– Start with a specific project Ensure your teams collect data from every source, whether they are web logs, application logs, servers, network, storage.

Collect everything involved in the delivery of your online service or business applications.

Start with one application and grow the platform over time.

– Discover what is useful in your data over time Don’t waste time attempting to schematise it all, but capture it, store it cheaply, index it, and prepare for unexpected discoveries over time.

– Be wary of over reliance on open source tools and custom approaches You risk inheriting and unwinding a mass of unsupported code that does not bring the value expected.

Tools like Hadoop and Cassandra are useful for some projects depending on the nature of the data stream to be analysed and will require trained developers to write the necessary code. 

Other projects will benefit more from use of a full featured platform, resulting in more clicks and less code.

As CIOs, we are constantly looking to innovate, to bring something to the business which is valuable, and which will make a difference for the company.

In the new era of big data, we are standing at the edge of a new revolution, and are perfectly suited to lead the charge.

Doug Harr is the CIO of Splunk

Pic: j3netcc2.0