Business Intelligence Definition and Solutions

For the business intelligence infrastructure that powers its matchmaking algorithms and maximizes the effectiveness of its numerous TV ads, the firm relies on four database and data warehousing products.

They include Oracle Database, the open-source MySQL database, another open-source data-crunching app, Hadoop, and data warehousing appliances from Netezza Inc.

For some IT managers, managing four such disparate products wouldn't worth the trouble. But not Joseph Essas, vice-president of engineering and operations for eHarmony.

"We always use multiple vendors for different things," Essas told the audience during his speech Wednesday at Computerworld's Business Intelligence Perspectives conference in Chicago.

Essas say she likes the "leverage from playing multiple people against each other." He fears that while settling down exclusively with one vendor might initially be a "bargain," it would eventually lead eHarmony to financially "bleed to death in years 2 to 5."

Essas' philosophy is interesting because it runs so counter to the site's goals, as a self-declared maker of long-term relationships.

eHarmony Inc. says that 236 of its 20 million members get married every day. That's more than 2% of American marriages per year, according to statistics based on online surveys conducted by a third party, Harris Corp., that were commissioned by eHarmony.

Related Story

Online dating: The technology behind the attraction

Marriage is only one of "hundreds of metrics" that eHarmony "deeply cares about," said Essas.

Tracking and crunching all of these metrics is key, as eHarmony must produce good matches for its members as soon as they fill out their profiles at sign-up, lest they lose them to rival dating sites.

"Their attention span with us is very short," Essas said, "So we need to get it right on the first try, if you will."

Assigning matches is a complex mathematical problem called "graph partitioning," said Essas.

eHarmony uses Oracle to do much of the initial matching. But for its hardcore data-processing, eHarmony relies on a 50-node Hadoop cluster. Hadoop is speedy, says Essas. "What used to take hours now with Hadoop takes just 30 minutes," he said.

That's important, because Harmony is rescoring its relationship matches whenever new members sign up, or even when existing members update their profile.

Hadoop also forces eHarmony to keep its data in key-value store form, rather than in a structured SQL format.

"It's really hard to build reusable data structures, especially at scale, in SQL," he said. Using Hadoop also makes it easier to figure out the cause of slow queries compared to using a SQL database, he said. And it forces eHarmony's developers to be more disciplined about what data it stores permanently, preventing the database from getting "too bloated."