Reflecting on a Decade of Hadoop

10 years ago the data scientist was a novel idea. Fast forward to the present, where smart data professionals realize the value of all data

hadoop10 copy

Ten years. As I sit here and type the words, I still can’t believe it’s already been that long. Ten years since Hadoop entered the lives of data and data management professionals like myself and ushered in an unprecedented (and still ongoing) era of change, investment, exploration, doubt, and discovery.

To say the time has flown by would be the very definition of an understatement. To say the Hadoop experience has changed the lives and careers of everyone in its path would be an equally insufficient characterization. For me personally, and for many others, Hadoop Summit has been a major part of that experience – an annual inflection point enabling us to truly take stock of the technology impact.

It was eight years ago that I experienced my first Hadoop Summit. I remember the feeling like it was yesterday. Blown away. Socks knocked off. Pick your cliché. The experience was career-defining. I remember sitting in the sessions that week filled with excitement and anticipation at the possibilities. The data scientist was a novel concept then, a class of game-changers poised to spread their wings and do great things. Terms you’d never heard before like zookeeper, hive, and pig were zooming all around. The energy literally swept me away.

Beneath the excitement though, there was a gnawing feeling. Above all else, I consider myself a technology pragmatist, and I couldn’t help but feel as though, lost amongst the brilliance readily on display, was the question of what to do with all of the legacy technology. Companies had and would undoubtedly continue to sink thousands of hours and millions of dollars into this technology. What was to become of the relational database? What about BI implementations, and the data warehouse?

There was clearly a gap that would need bridging – an innovation aimed at bringing together the inventions of today and tomorrow with the investments of yesterday. It was this phenomenon that I was most eager to see play out, and most eager to play a role in. I left that day with equal parts curiosity, concern, and anticipation.

Fast forward to this past June, to the 10th anniversary of the Hadoop Summit and to my 8th visit to the annual soiree. It’s simply amazing how far the Hadoop world has come, from the inventive-bordering-on-irreverent approach of those early days to a much more mature, much more focused approach today, one in which smart data professionals realize the value of all data.

I attended sessions focused on topics such as “co-existing data sources” and “the convergence of relational and unstructured data.” There were more sessions than I can ever remember focused on real-life customer use cases. The tone and tenor of the event has changed so dramatically – much to my personal delight – in the past eight years. The best and brightest are no longer focused science projects and ungrounded experimentations. There’s still some of that in healthy doses, of course, as there should be, but the majority of emphasis is now on taking an enterprise-class approach to Hadoop and leveraging the power of all data. I left feeling as though the chasm that had me both excited and concerned eight years earlier was well on its way to being crossed. And what a great feeling that was.

Equally great was the feeling that analytics seem to be on everyone’s mind. Analytics is, has been, and always will be the end goal of most data projects, and the owners of data and the corresponding systems are looking more seriously than ever before at how to deliver analytics to a broad audience. Topics like “self-service data access,” “analytics for everyone,” and “delivering on the promise of BI” drew large crowds. People are all-in when it comes to driving value into the business through analytics.

All of which means that the pomp and circumstance, and the seemingly never-ending hype may be coming to an end, and the age of big data analytics may finally be upon us. And I personally can’t wait to see how we can help transform businesses and lives as a result. Here’s to the next ten years!                        

Joanna Schloss is a Director of Product Marketing, Datameer.

Copyright © 2016 IDG Communications, Inc.