Do you know someone who bought a lot of fancy exercise equipment but doesn\u2019t use it? Turns out, exercise equipment doesn\u2019t provide many benefits when it goes unused.\n\nThe same principle applies to getting value from data. Organizations may acquire a lot of data, but they aren\u2019t getting much value from it. This is a widespread issue that cuts across different sectors. It is estimated that nearly 75% of the data that enterprises collect remains unused,and thus, the value is not realized. So, what is the problem?\n\nIn the fitness example, the problem is typically not the exercise equipment; it\u2019s an issue with the user\u2019s habits. Similarly, getting value from data often is not a problem with the data itself. Rather, problems arise from limitations imposed by data infrastructure and data practices that block effective and efficient use. In other words, poor choices in data infrastructure and data habits can lead to data waste. \n\nWhat is data waste, and why does it happen?\n\nFundamentally, data waste means missing an opportunity to get value from data or paying too much to acquire, store, and use data. In large-scale systems, data waste comes in many forms. Some are surprising, most are expensive, and almost all are avoidable. \n\nTo avoid unnecessary data waste in your organization, first you must recognize it. The following describes five common ways that waste occurs:\n\n\u2022 Data is used and then thrown away\n\nA common data habit that results in missed opportunity is assuming data has no further value once it\u2019s been used for the particular purpose. Data is ingested, processed, transformed (perhaps for a specific report or to be stored in a traditional database), and then the raw or partially processed data is discarded. It isn't practical to save all your data, but it is important to realize data may be valuable for other projects. You lose that add-on value when you throw data away. \n\nThis type of data waste results in missing out on the second project advantage. For example, AI and machine learning projects offer great potential value, but they are speculative. Lowering the entry cost by re-using data and infrastructure already in place for other projects makes trying many different approaches feasible. That, in turn, makes it more likely to find the ones that pay off. Fortunately, learning-based projects typically use data collected for other purposes. \n\nIt\u2019s also important to go back to raw data to ask new questions and train new models, particularly as the world is constantly changing. Features that you didn\u2019t think were valuable at first may later be just what you need. You\u2019ve lost that opportunity if the data has been thrown away. \n\n\u2022 You have data but don\u2019t use it\n\nWhy does valuable data so often go unused? One reason is people don\u2019t know where it is or even possibly that it exists at all. Lack of annotation with the right metadata is a contributing factor. Another is poor communication between projects or business units.\n\nAn even larger issue is that people may not know how to see value in data. Recognizing what data can tell you is an acquired skill for people beyond just data scientists. New approaches are being developed to understand and use unstructured data, for instance. But to get the benefits data has to offer, you must learn to use it, just like you need to know how to use exercise equipment before it can do you any good. \n\nAnother factor that keeps people from fully using and re-using data is data infrastructure requiring specialized tools. This limitation makes it inconvenient for data to be used by different types of applications or different analytics and AI tools. Increasingly, people look for ways to unify their data layer and have flexible access in order to build a data-first environment.\n\n\u2022 You have data but not where it is needed\n\nData in the wrong place is about the same as data that does not exist. And "wrong place" can mean more than one thing. It may be that data is held by a different business unit, making it difficult to identify or challenging to get the permissions and access needed to share that data. Once again, there is a cost for not using data because it is somewhere other than you\u2019d like it to be. \n\nAnother way data is in the wrong place is in a more literal sense: geolocation. For large systems, major data motion from edge to data center or between data centers that are located in different cities or countries is challenging, especially if you do not have data infrastructure designed to do move data automatically. Coding data motion into applications is not an adequate alternative except in the simplest of cases. To avoid data waste, you must have a way to efficiently move data to where it is needed. Otherwise, hand-coding of data motion can lead to additional problems, including unwanted duplication.\n\n\u2022 Your system involves unwanted duplication\n\nHaving unnecessary duplication of large data sets is clearly a waste of the resources used to store and access data, but it involves waste in other ways as well. Duplication of data also entails duplication of effort, which is an additional cost. And the problem is not just a matter of too many copies of data. Approximately duplicated data sets may introduce uncertainty about data quality. Near duplicates immediately raise the question of which is authoritative and why there are differences, and that leads to mistrust about data quality. \n\nHand-coded data motion by many different users creates its own problems, as this is hard to do accurately at scale. Resultant data sets can introduce unintentional variation in data even where a verbatim copy is intended. \n\nAnother related problem is the creation of data silos in large systems. Unwillingness to share data often points to the lack of a uniform data layer with flexibility in data access. Siloed data not only results in avoidable costs, but it also limits the understanding and insights data scientists and analysts can draw from the data. Siloing and poor data discovery capabilities are wasteful through opportunity cost plus the cost of redundant storage and duplicated effort.\n\nA special example of data waste through unnecessary duplication occurs when an enterprise buys data that could have been obtained for free. This waste happens because people may not know what data options are available.\n\n\u2022 Disconnect between data producers and data consumers\n\nOne problem with connecting data producers and data consumers is that those who produce data or even those responsible for data ingestion often do not know how it will be used. That disconnect makes it harder for those who need data to know where to find it or to know what the data actually consists of when they do find it. Data producers are challenged with annotating data appropriately without knowing the ways it will be used. This disconnect between data producers and data consumers leads to a classic type of data waste in the sense of missed opportunity or unnecessary effort and expense required to track down data. \n\nReducing data waste\n\nHow can you address the issues listed above in order to reduce data waste? You need to develop a comprehensive data strategy that includes a unifying data infrastructure engineered to support flexible data access, data sharing, and efficient data motion. HPE Ezmeral Data Fabric is a software-defined and hardware agnostic data technology used to store, manage, and move data at scale across an enterprise -- from edge to data center, on premises, or in the cloud. As such, it serves as a unifying data layer that supports a wide range of applications and tools, thus inviting the re-use of data. In addition, data fabric handles data motion automatically at a platform level. \n\nOther solutions come in the form of better use of metadata to aid in data discovery and understanding, along with new data initiatives to better connect data producers with data consumers. One new initiative is the Agstack Foundation, an open-source digital infrastructure for agriculture. Another example is Dataspaces, a new service platform that helps data producers and data consumers integrate diverse data sets, enhance data discovery, and access and improve data governance and trust.\n\nThese solutions can help you reduce costly data waste and take better advantage of the value data offers. Making better use of your exercise equipment, however, is still up to you.\n\nTo find out more about data infrastructure that can help you reduce data waste, read this technical paper.\n\n____________________________________\n\nAbout Ellen Friedman\n\nEllen Friedman is a principal technologist at HPE focused on large-scale data analytics and machine learning. Ellen worked at MapR Technologies for seven years prior to her current role at HPE, where she was a committer for the Apache Drill and Apache Mahout open source projects. She is a co-author of multiple books published by O\u2019Reilly Media, including AI & Analytics in Production, Machine Learning Logistics, and the Practical Machine Learning series.