Where to put all that Big Data

The rate of change of data that is being generated has been noteworthy for sometime and from where i’m standing in a professional services environment, the proliferation of material is very clear to see.

The general gist is that growth is exponential and this trend is not the preserve of big business.

Collaboration inter business, and more so beyond the boundaries of the business, has blossomed beyond the traditional email but now encopasses voice and video traffic at the very least.

What is clear, the consumerisation agenda has stimulated an increase on data volumes particularly as mobile devices has driven multimedia traffic flowing round the enterprise, legitimate or otherwise.

Initiatives like docx from Microsoft will assist with slowing the storage requirements but this will not be an overnight sensation.

Data storage is moving forward as you would expect but more interestingly the choices on offer for in-house teams is expanding. Buying vast arrays of disk is not a default position.

StaaS is now a very credible alternative in passing on the considerable management burden and skill required to manage data stores.

The smart money here is spend on techniques to actively move data between high speed storage and other less expensive alternatives.

Removing duplicates is necessary. Those of us running an in-house IT organisation have to tolerate the annoyance that is enthusiastic users emailing very graphical powerpoint presentations to a large audience.

Balancing what gets stored where and deploying some of these other techniques requires a skill set likely to be beyond even the most modest of SME’s.

Some of the high end storage vendors have strong offerings but this level of investment should follow through to the support of it which can mean people and more than one for failover.

StaaS is a serious contender in addressing the accessibility to high end tech and the important management wrap around it. With StaaS comes all the capex avoidance benefits the cloud market bang on about.

Big data has another dark cloud for IT leaders, which is the recovery position. A cornerstone of our internal standard operating condition is the provision of a credible data recovery plan that fits into a more rounded business continuity plan.

The more data there is swilling around, the larger the recovery time and potential complexity. More organisations are adopting a disk-to-disk backup strategy with mag media used only as a last resort.

The backup windows have reduced dramatically for most businesses coupled with increased data volumes putting mag media as a methodology beyond reach for full backups.

Incremental is the next stopping off point but requires a full copy and a well-organised library to be sure you are bringing systems back to a recognisable condition.

The compliance question Many of us work in heavily regulated environments but all of us have to have one eye on the Information Commissioner’s increasing powers.

Keeping on top of the data mountain requirements mechanically is one issue, but ensuring the right people have the right access to it is another, assisted by adequacy of data retention policies.

Standards such as ISO27001 with its 130+ controls is worth a look as it enforces a holistic view data assets and their containment.

Hidden knowledge buried within the mountain of structured and unstructured data, both of which many businesses will have in quantity, is the next quest once all the other hygiene challenges have been taken care of.

Within the mountain of data there are trends, statistics and other indicators of business issues or opportunities. Is there something you don’t know about your business or customers hidden within?

Data as ever, is a significant asset CIO’s and the like have to manage well, despite the omnipresent issue of finding suitable owners for data. It’s assumed that because it’s electrical, it’s an IT responsibility.

That’s not a big surprise but the issue does have a few more complexities to it and requires more thought and diligence than just throughput and delivery.

Richard Hodkinson is IT Director of law firm DWF