EMC Greenplum Tackles Big Data With Hadoop Distribution

EMC Greenplum debuts its own Hadoop distribution, Pivotal HD, which marries Greenplum's massively parallel processing database technology with the Apache Hadoop framework to create a technology called HAWQ.

By
Tue, February 26, 2013

CIO — In a further sign that Big Data is beginning to hit its stride, EMC Greenplum on Monday debuted a new Apache Hadoop distribution that it hopes will soon become the de facto standard, pushing aside current leaders like Cloudera and Hortonworks.

"The bet that we're making at Greenplum is that the future of our Big Data business is Hadoop, so it's important that we have our own distribution," J

Josh Klahr
vice president of product management
EMC Greenplum.

"Virtually every customer and prospect we talk to is doing something with Hadoop," says Josh Klahr, vice president of product management at EMC Greenplum. "It ranges from CIOs that have heard of Hadoop and spun up a Big Data team to figure out what to do with it to folks that are being a little more thoughtful about it and have figured out a use case."

"We're seeing rapid adoption," he adds. "The Hadoop business is growing 60 to 70 percent a year. We see this as a real sea change."

Since its emergence as an Apache Lucene subproject in 2006, Apache Hadoop has quickly become the preferred solution for Big Data applications with massive repositories of unstructured data. Hadoop has many things to recommend it: It's flexible, scalable, built on commodity hardware and fault-tolerant.

However, there are hurdles to implementing Hadoop in the enterprise. One critical hurdle is the lack of useful interfaces and high-level tooling for Business Intelligence and datamining. That's where Greenplum sees its opportunity, Klahr says.

Greenplum Bets Hadoop Is the Future of Big Data

"The bet that we're making at Greenplum is that the future of our Big Data business is Hadoop, so it's important that we have our own distribution," Klahr explains. "There are some things we're adding into the distribution that aren't yet supported by Apache. We think that the Hadoop market is going to be so big that we wanted to have our own distribution."

Some elements of EMC Greenplum's distribution, Pivotal HD, may never find their way back into the Apache project. The core of Pivotal HD is the marriage of Greenplum's massively parallel processing (MPP) database technology with the Apache Hadoop framework, a technology called HAWQ. HAWQ is essentially a fully functional, high-performance relational database that runs in Hadoop and which speaks SQL natively.

"Our plan is to actively contribute certain elements back to Apache, but we're going to have HAWQ as a proprietary service that we won't be open sourcing," Klahr says.

Klahr says that HAWQ delivers performance improvements of 50X to 500X when compared with existing SQL-like services (like Hive) on top of Hadoop.

Continue Reading

Our Commenting Policies