In the technology and data analytics space, I\u2019m continually reminded that the only constant is change. This industry loves to innovate. Time and again we innovate to overcome immediate and future challenges \u2013 with solutions that address the need for more data, faster analytics, and better architecture.\nThe innovation typically follows a trajectory of something ground-breaking, followed by years of incremental improvements that mature the offering and make it applicable to the masses. While these incremental changes are usually easy to incorporate, the problem is we have to implement the ground-breaking innovation first. This transition usually requires process changes, training, re-architecture, and a long, painful migration. Ultimately, this leads to the technology hype cycles, where businesses individually assess when or even if the risk and struggle to make a change is worth the rewards.\nLooking back\u2026a little perspective\nHadoop is a great example of both sides of this phenomenon. Several years ago, Hadoop was the new innovation on the block. In the early 2010s, it came in fast and furious as the enterprise data warehouse (EDW) killer. Although Hadoop\u2019s messaging and immature technology created confusion for many enterprises, some early adopters cut their teeth on it and made it work. Over the years, the technology matured to the point that (nearly) everyone had a Hadoop-based data lake running in their data centers.\nFast forward to 2020, and your business-critical analytical applications depend on Hadoop \u2013 but now it is on the other end of the technology cycle. The Hadoop ecosystem chugged along and evolved over the past decade, but several new technology innovations occurred in the meantime. The time has come to embrace these new innovations \u2013 and modernize your big data estate.\n4 major technology developments \nFour must-have technology developments impact the big data information estate for enterprises today:\u00a0\n\n\nContainerization and Kubernetes are game changers\u00a0Containers (and Kubernetes orchestration) can deliver a lot of benefits for big data environments. With containers, you can separate compute and storage. This capability lets you right-size your solution, drive greater efficiency, and optimize the utilization of your compute. Containers also allow you to embrace the constantly evolving ecosystem of open-source tools, enabling data analysts and data scientists to spin up their tools of choice in minutes, while getting access to the data they need. Plus, you get application portability, flexibility, and agility, meaning you can quickly and easily deploy data-intensive apps on premises or in any cloud.\u00a0\n\n\n\n\nData is everywhere \u2013 on prem, hybrid cloud, multi-cloud, and at the edgeOriginally, the big data estate for most enterprises was planted firmly on premises. But more apps are being deployed in the public cloud and often on multiple public clouds. And with the ever-increasing volume of data generated at the edge (together with network improvements), you need to think about your data globally -- from edge to cloud. Your next big data platform should adapt to the needs of your business and data everywhere. And it must be flexible to accommodate on premises, hybrid cloud, multi-cloud, and edge computing deployments.\u00a0\u00a0\n\n\nThe open-source ecosystem continues to evolve\u00a0Enterprises need to future-proof their big data investments. Over time, some vendors have focused on the pure open-source model. Others have provided value-add commercial software built on open-source technology. Turns out both approaches are right. You\u2019re going to want optimized tools from your solution provider when it makes sense, but your future big data estate also needs to evolve with the speed of open-source innovation. By implementing a solution with the ability to deploy any open-source framework, you are prepared for this constant evolution, while giving your data scientists access to the latest open-source toolkits.\n\n\n\n\nMake the infrastructure invisible \u2013 while ensuring performance, resiliency, security, and high availability\u00a0I remember a comment a CTO made to me several years ago. When arguing a point about how to improve performance of data lakes, he said, \u201cYou\u2019re all about infrastructure; we don\u2019t care about the infrastructure.\u201d I\u2019ve since embraced this mantra (after all, data science teams don\u2019t want to have to worry about the underlying storage, compute, and networking), but infrastructure is still important. We can hide the complexity of the infrastructure, making app deployment as easy and as seamless as possible. But if you don\u2019t architect your solution to ensure security, performance, and other enterprise-grade requirements, it won\u2019t make it in production. And ultimately, it won\u2019t deliver business value.\n\n\nIs the risk worth the reward?\nHadoop distributions are fighting to stay relevant, but data platform and deployment alternatives have emerged. Many enterprise organizations are re-evaluating their path forward, embarking on a new strategy to modernize their big data estate. So, now is the time to ask the difficult questions:\u00a0\n\n\nAm I currently getting the value I was expecting from my data lake? What extra value do I get when I upgrade?\n\n\nWhat will the integrated solution look like? What features\/apps will still be there?\n\n\nWhat is the roadmap? Will it change if my distribution is acquired?\n\n\nDo I have to upgrade? How do I do it? How long will it take? How much will it cost? When do I lose support on my current version?\n\n\nWill I be locked into my distribution\u2019s proprietary apps? How easy is it to bring in the latest open-source tools that my data science teams want?\n\n\nIs Apache Ozone ready for primetime? Should I trust it with my data?\n\n\nIs the risk worth the reward or should I consider another strategic solution (and another strategy partner) to modernize my big data estate?\n\n\nHewlett Packard Enterprise can help\nHewlett Packard Enterprise (HPE) knows first-hand that enterprise organizations \u2013 and their business-critical, data-intensive applications \u2013 are caught in this storm of uncertainty and change. We\u2019ve recently gone on our very own modernization journey to fulfill our vision of a data-driven business. Our new elastic data analytics solution leverages containers, the latest hardware, and open-source toolkits to bring speed and agility to our decision making and empower our worldwide users from edge to cloud.\nUnfortunately, there\u2019s no easy button, since each organization has its own requirements. But HPE can help customers navigate this process. HPE has a complete portfolio of solutions, expertise, and support to help modernize your big data estate.\u00a0\nTo de-risk the modernization process, we created the\u00a0HPE AMP Assessment Program to help clients answer the difficult questions about their big data information estate. With this offering, HPE will:\u00a0Analyze\u00a0your current-state platform, provide a detailed\u00a0Map\u00a0to modernize your current platform in a way that will meet the business needs of your organization, and finally,\u00a0Prescribe\u00a0a systematic plan to get you there. And as the output from the AMP Assessment, HPE can leverage its entire arsenal from HPE Ezmeral software, world-class hardware, and proven services to deliver the right solution for your specific needs.\nTo learn more \nIf you\u2019re looking for straight talk on this topic, I recently hosted Modern Big Data Solutions Roundtable with three of our Big Data CTOs.\u00a0 Please listen in as we address relevant topics such as upgrade\/migration paths, Kubernetes and containerization, open-source options and tools, and hybrid cloud\/multi-cloud deployments.\u00a0\n____________________________________\nAbout Matthew Hausmann\n\nMatt\u2019s passion is figuring out how to leverage data, analytics, and technology to deliver transformative solutions that improve business outcomes. Over the past decades, he has worked for innovative start-ups and information technology giants with roles spanning business analytics consulting, product marketing, and application engineering. Matt has been privileged to collaborate with hundreds of companies and experts on ways to constantly improve how we turn data into insights.