CIOs have been fighting to keep the deluge of data at bay for years. But could they finally be losing the battle?\nAnalysts are predicting the worst. IDC says that there is already more data in the world than there is space to store it and that demand for storage capacity will, over the next three years, increase at a compound annual rate of nearly 50 per cent.\nThe figures are mind-boggling: according to a 2010 The Economist report, the US retail firm Wal-Mart \u201chandles more than one million customer transactions every hour, feeding databases estimated at more than 2.5 petabytes \u2014 the equivalent of 167 times the number of books in America\u2019s Library of Congress\u201d.\nCIOs themselves are concerned: three years ago Rick Chapman, CIO of Kindred Healthcare told CIO magazine that the firm\u2019s data volumes were increasing by 40 per cent a year, and that it was already dealing with 400 terabytes of data.\nOnly a few years ago these quantities would have been unimaginable. So what\u2019s driving the change?\nMuch of it is down to the growth in enterprise data, including the customer transactions of the type stored by Wal-Mart. Such data provides organisations with rich information about customer behaviour.\nUnstructured content is another culprit: emails, PDF files, Word documents, PowerPoint slides and voicemail messages have been around for a while, but now firms also have to deal with social media output, such as blogs, tweets and instant messages.\nAman Munglani, principal research analyst at Gartner, predicts that unstructured data will make up 80 per cent of the total available capacity by 2014.\n\nThe CIO Hitachi survey on the data pressures faced by CIOs confirms the growth trend: 87 per cent of respondents said that data storage capacity in their organisation had increased since last year, and that the biggest increase in data had come in enterprise applications, with email a close second.\nBackup backlogFinding storage space is a significant enough problem, but enterprises have three additional problems. The first is that CIOs must back\u00a0up and ensure enterprise application data can be searched and retrieved.\nTraditional tape backup has its limitations, however, when it comes to dealing with large quantities.\nAs Kindred Healthcare CIO Chapman explains, \u201cIt takes more time to back up and recover than we have time available in the datacentre overnight.\u201d\nSecondly, enterprise data must be kept private and secure. The damage to reputation for an organisation that fails to safeguard sensitive data can be significant, and there may also be financial penalties.\nZurich Insurance discovered the damage a poor storage strategy can have when it wasfined \u00a32,275,000 by the Financial Services Authority (FSA) for losing the personal details of 46,000 customers. #\nIt\u2019s not an issue confined to financial services organisations either: any organisation that does not safeguard data effectively can be subject to penalties under the Data Protection Act.\nKeeping data secure is particularly hard when data is so easily copied onto laptops or memory sticks, and when systems are under attack from viruses and hackers.\nThirdly, the increasing pressure from regulation means that old enterprise data, whether structured or unstructured, must be retained for a fixed amount of time, and in a format that makes it easy to retrieve.\nRequests for data could range from the FSA asking to see all of a bank\u2019s emails between the CEO and CFO in a particular month five years ago to a member of the public asking a local authority to provide all documents relating to a particular planning application.\nOn the other hand, keeping every piece of data isn\u2019t a realistic option, not only because of the space it consumes but also because the Data Protection Act requires that personal data is deleted after a certain amount of time.\nStorage stressesBetween them, these three challenges (backing up data, keeping it secure and making it easily retrievable) add up to a headache that won\u2019t go away.\nAs energy costs continue their upward growth, and as datacentre space becomes more expensive, CIOs are looking at new ways of addressing the storage problem.\nSome are moving away from tape backups, for example, which are time-consuming to create, require physical security to transport and are relatively fragile.\nInstead, they are using storage virtualisation technology, which compresses data more efficiently on disk and makes it easier to access.\nIncreasingly, CIOs are also turning to cloud solutions to manage data storage and archiving.\nIn the CIO Hitachi survey, 90\u00a0per cent of respondents were planning to move to a fully outsourced cloud service in future, while 62 per cent planned to opt for a part-hosted solution, archiving their data to a service provider. Only eight per cent of respondents said they would manage their data in-house in future.\nCloud solutions or X-as-a-Service capabilitiesallow service providers to capitalise on economies of scale to a greater degree than are available to most enterprises and free them from the problem of having to invest in more and more storage.\nPrivate clouds, which use technologies such as virtualisation to reduce costs and storage space, are another option for businesses that don\u2019t yet feel ready for a full move to the public cloud.\nThe analyst firm Quocirca believes that an increasing number of businesses will opt for a hybrid cloud: a combination of private and public cloud services that will enable organisations to keep control of sensitive data while taking advantage of the scalability provided by the public cloud.\nDeletion dilemmasAlthough cloud and virtualisation technologies can ease some of the data management headaches CIOs face they do not provide all the answers.\nBusinesses have to make difficult decisions about what data to retain and what to delete, based both on regulatory requirements and on business need.\nOur survey suggests that not all organisations are fully confident of the effectiveness of their policies and procedures in this area.\nOne in five respondents said they either didn\u2019t have, or were not sure whether they had, a documented procedure for the retention and disposal of documents and records.\nEffective data management, however, requires businesses to understand which data has little business value and can be deleted and which needs to be retained, either for business or regulatory purposes or both.\nPolicies that can categorise different kinds of data and assign different strategies for dealing with them must play a crucial part in the decision about which technologies to use and how to use them.