Hadoop-based data lakes can be game changers: better, cheaper and faster integrated enterprise information. Knowledge workers can access data directly, where project cycles are measured in days rather than months, and business users can leverage a shared data source rather than creating stand-alone sandboxes or warehouses.\nUnfortunately, more than a few data lake projects are off track. Data is going in but it\u2019s not coming out, at least not at the pace envisioned. What\u2019s the chokepoint? It tends to be some combination of lack of manageability, data quality and security concerns, performance unpredictability, and shortage of skilled data engineers. \u00a0\nWhat distinguishes data lakes that are \u201centerprise class\u201d, i.e., the ones that are built to last and attract hundreds of users and uses?\u00a0 First let\u2019s look at the features that are Table Stakes, i.e., what makes a data lake a data lake. Next we will describe the capabilities that make a first class data lake, one that is built to last.\nTable stakes\n\nHadoop \u2013 the open source software framework for distributed storage and distributed processing of very large data sets on computer clusters. The base Apache Hadoop includes contains libraries and utilities needed by other Hadoop modules, HDFS\u00a0\u2013 a distributed file --system that stores data on commodity machines,a resource-management platform for managing computing, and an implementation of the MapReduce programming model for large scale data processing.\nCommodity Compute Clusters \u2013 whether on premise or cloud Hadoop runs on low cost commodity servers that rack and stack and virtualize. Scaling is easy and inexpensive.\u00a0 The economics of open source massively parallel software combined with the low cost hardware deliver the promise of intelligent applications on truly big data.\nAll Data \/ Raw Data \u2013 The data lake design philosophy is to land and store all data in raw format from source systems. Structured enterprise data from operational systems, semi structured machine-generated and web log data, social media data, et al.\u00a0\nSchema\u2019less writes \u2013 this point in particular is a break-through. Whereas traditional data warehouses are throttled by time and complexity of data modelling, data lakes land data in source format. Instead of weeks (or worse) data can be gathered and offered up in short order. Schemas are used on read, pushing that analytic or modeling work to analysts.\nOpen source tools \u2013 (e.g., Spark, Pig, Hive, Python, Sqoop, Flume, Map Reduce, R, Kafka, Impala, Yarn, Kite, and many more) the evolving toolkit of programming, querying, and scripting languages and frameworks for ingesting and integrating data, building analytic apps, and accessing data.\n\nEnterprise class\nIf the Table Stakes listed above defines a data landing area, the following differentiate a data lake that is expansible, manageable, and industrial strength:\n\nDefined Data and Refined Data \u2013 where data lakes contain raw data, advanced lakes contain Defined and Refined data as well. Defined Data has a schema, and that schema is registered in Hadoop\u2019s Hcatalog. Since most data comes from source systems with structured schemas, it\u2019s infinitely practical to leverage those. Refined Data, a step up the value chain, is data that has been altered and augmented to add intelligence and value with joins, aggregations, cleansing, counts, transformations, de-duplication, et al.\nMeta Data Management \u2013 perhaps the biggest complaint of data lakes is that they become unmanageable due to lack of meta data. This includes technical meta data about the structure, format, and validation rules for data (e.g. schemas registered in Hcatalog), business meta data about business rules and meanings of data, and operations meta data about the jobs and counts.\u00a0\nLineage and Audit Trail \u2013 a data lake needs an audit trail of data and processes to show how data flows from its source to its destination and the various changes that occur as it moves and gets transformed. The audit trail can be achieved by collecting logs from across the platform.\nData Profiling \u2013 determining data quality and content is central to the analytic process.\u00a0 Profiling in the era of Big Data often requires parsing raw data to get to the numbers and values. \u00a0Also volumes can be so high that profiling jobs to tally descriptive statistics can run for hours, so that optimization and approximation techniques must be used.\nOperations Control and SLA Management \u2013 users in an enterprise class environment require performance predictability.\u00a0 To achieve this data lakes need industrial strength tools for operations management and control. The types of capabilities that are required are the ability to manage performance in a multi-tenancy environment, ability to allocate departmental charge-backs based on usage, and providing visibility into clusters with health checks. Historical views and metrics should let you see what happened when, and allow you to quickly see unusual system performance.\u00a0High availability across components and built in backup and disaster recovery means you can run even your most critical workloads, risk-free.\nSecurity Enforcement \u2013 data security is a must-have for an enterprise data lake. Hadoop poses unique security challenges, including its replication of data across multiple nodes. \u00a0Data lakes concentrate vast and often sensitive data in a relatively open environment. \u00a0Standard IT security controls \u2013 e.g., securing the computing environment and log-based network activity monitoring \u2013 are starting points but they are not enough to secure an organization from data-centric cyber-attacks. Additional data de-identification approaches include encryption or masking at the row, field, and cell level. Data-centric security calls for de-identifying the data, transforming sensitive data elements with de-identified equivalents can be used by applications and analytic engines.\nSoftware Tools \u2013 if you\u2019re serious about creating an enterprise class data lake, you\u2019ll probably consider software tools that help get the job done. Tool vendors are aggressively bringing to market solutions to accelerate the journey. This is imperative because hand coding to meet the requirements above is impractical and inefficient, and even if a firm had the talent for the hand coding, most will lack the top notch vendors\u2019 vision of operational excellence.\u00a0\n\nOne of the leading Hadoop companies uses the term \u201cEnterprise Data Hub\u201d rather than \u201cdata lake\u201d.\u00a0 I hope that moniker takes hold.\u00a0 The Holy Grail for enterprise information management is the vision of a \u201csingular version of the truth,\u201d which eluded legacy EDWs due to rigidity and delays and high costs, which forced business units to roll their own. I\u2019m optimistic that enterprise class data lakes -- i.e., the ones that are built to last -- are a strong step in the direction of both singularity and data democratization.