Also referred to as unstructured data, dark data is growing at a rate of 62% per year, according to IDG. By 2022, they say, 93% of all data will be unstructured. \u00a0\nGartner defines dark data as, \u201cthe information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes\u201d. Consisting of data from a huge variety of sources - emails, documents, instant messages, digital media posts, partly developed applications \u2013 or just information which isn\u2019t being used or analyzed, its nomenclature makes it sound foreboding. With new regulations such as the GDPR coming into force, businesses must gain a clear understanding of the data they hold. For structured data, this is straightforward. But dark data is much harder to manage, stored across a distributed IT environment with no single owner.\nA \u2018bottomless lake of data\u2019\nDark data tends to be text-based data, as well as video, audio files and images. It\u2019s generated by a diversity of different sources, gathered from mobile devices, social platforms, apps and internal systems to name but a few. \u00a0Much of the data generated by the Industrial Internet and the Internet of Things is unstructured, so this also falls under the dark data shadow.\nIn the workplace, employees are responsible for generating a lot of dark data. In fact, says Sony Shetty from Gartner, \u201cAcross the enterprise, employees are blindly building a bottomless lake of data and, in many cases, a corporate mantra of \u2018save everything, just in case\u2019 is encouraging the behavior\u201d. Think about the amount of data you, personally, generate, filter and store each working day \u2013 did you record your last conference call in case anyone missed it? Did you make it available as a podcast and save that, too? What about your customer calls \u2013 do you record them \u2018for training purposes\u2019 and store them as audio files? Do you have a chat function on your website and keep a record of the interactions, or use an instant message function on your desktop? One study found enterprises to be using almost 500 business applications, each generating data.\u00a0\nAll the data generated by this activity falls under the definition of dark data, and is stored across different devices, drives, desktops and SaaS platforms. Most of it will never see the light of day again. Employees leave - taking their passwords with them \u2013 customers move on, business priorities change, and no-one has the remit, the ability or the time, to remove the data. \u00a0The information quickly becomes out of date and inaccessible.\nThe need to understand data\nPrior to the GDPR, dark data would have been an accepted part of legacy business. In the UK, the 1998 Data Protection Act didn\u2019t provide any minimum or maximum period for data to be stored, so it would have been a case of \u2018out of sight, out of mind\u2019. Now, though, the GDPR requires businesses to gain an in-depth understanding of how data flows across their organization, along with stringent data governance. The new Data Protection Bill coming into force will implement the GDPR into UK law. From May 25th, if a \u2018data subject\u2019 \u2013 a client, employee or other stakeholder - asks what data a company holds on them, the company must know and share this. If they ask to see a record of when and how they gave their consent to be used, the company must provide this too, and only information necessary for its original purpose should be processed. \u201cInaccurate or outdated data should be deleted or amended and data controllers are required to take "every reasonable step" to comply with this principle\u201d, says Debbie Heywood from Taylor Wessing.\nThis is extremely hard to fulfil if data is held in silos across an organization. \u00a0\u201cBecause unstructured data is text heavy and irregular, making sense of what is being said and how it\u2019s being said \u2014 posi\u00adtively or negatively \u2014 is not for the faint of heart,\u201d says a report from the Medallia Institute.\nTapping into uncharted territory\nThe time has come for businesses to bring their dark data into the light. Doing so helps drive GDPR compliance, but the benefits of understanding dark data stretch far beyond compliance. Think of it as discovering uncharted territory: analyzing this unstructured data offers the opportunity to extract invaluable business insight which would otherwise lie dormant. It transforms information from data into strategic intelligence. Gartner cite, \u201cSome examples of data that is often left dark include server log files that can give clues to website visitor behavior, customer call detail records that can indicate consumer sentiment and mobile\u00a0geolocation data\u00a0that can reveal traffic patterns to aid in business planning.\u201d.\nFor example, most of us know that retailers are experts at using psychology to drive product placements. They understand our thought process and how we tend to move around a store, and place products accordingly. Studying filmed footage of consumers\u2019 mobility in stores helps retailers refine their product placement strategies even further. As Deloitte says, \u201cA retailer may be able to gain a more nuanced understanding of customer mood or intent by analyzing video images of shoppers\u2019 posture, facial expressions, or gestures\u201d.\u00a0 This intelligence, extracted by analyzing dark data, can translate directly into revenue as retailers apply it to their store layout.\nBy analyzing dark data businesses can:\n\nCreate a truly 360-degree single customer view, to drive engagement and boost interactions\nAnticipate, understand and respond to changes in market- and consumer-demand\nDevelop an in-depth understanding of consumer sentiment on their brands, gleaned from social platforms and multichannel interactions\nLockdown and secure vulnerable data points, and give personal data the protection it requires\nRefine the accuracy of risk management models\nAddress recurring pain points for customers and direct customer support to those areas most affected\nIdentify any links and connections between data sets\nGenerate a strong foundation for accurate forecasting\nGain a deeper understanding of website performance from web analytics\nIdentify new revenue streams. According to IDC, \u201cBy the end of this year, according to IDC, \u201c50% of Large Enterprises Will Be Generating Data-as-a-Service (Daas) Revenue from the Sale of Raw Data, Derived Metrics, Insights, and Recommendations\u201d.\n\nNow, analyzing unstructured dark data is simpler than ever before. Advanced, high-performance Customer Information Management tools automate and accelerate processes, connecting data sets for clarity and insight. Software scans both structured and unstructured data, using different data profiling techniques. The results of the scan are used to automatically generate a library of documentation, which describes a company\u2019s assets and creates a metadata repository. You can then start to explore the opportunities and possibilities which lie within the data \u2013 and that\u2019s when it starts to get really exciting.