Both sides of the big data debate

A multi-ethnic, mostly male crowd teemed in and out of windowless meeting rooms and the sprawling vendor exhibit hall. Some stood mesmerized like disciples before large LED screens. Others wandered between booths, munching on popcorn or nursing craft beers. Many lounged in crowded common areas, discussing graph databases and Hive. Another day, another tech industry event in Silicon Valley.

Or was it? There was a DJ spinning club music (I think I caught Skrillex’s pumping remix of GTA’s “Red Lips,” and Lana Del Rey’s “Summertime Sadness” looped reliably) and a few presentation titles (“Can you afford to drop ACID?,”) actually caught me off guard.

Last week’s Strata Hadoop World drew a predictable crowd with some surprise features. Like most tech industry affairs, it has its admirers and detractors. I spoke with attendees on both sides and have summarized the two prevailing perspectives.

Perspective 1: ‘Nothing New’

An old friend, a media company exec who had worked on more than his share of enterprise data warehouses, took a seat beside me during a panel discussion of big data trends.

He leaned toward me. “It’s the same old shit,” he whispered. “They’re jabbering on about all these new tools like they’re revolutionary. But they’re just tackling problems no one ever solved the first time!”

I asked him why he thought there was nothing new, when fully half of the vendors exhibiting had been in business for less than five years, and I hadn’t heard the terms “data quality” or “dimensional modeling” even once.

“Everyone’s just trying to nail down data so they know how to talk to customers better and increase average purchase amounts,” my friend said. Before I could argue with him, a panelist launched a treatise on “journey engineering.”

My friend looked at me triumphantly. “Same circus, different monkeys,” he said dismissively. He pulled some earbuds from his backpack, squeezed my shoulder, and headed for the door.  

Perspective 2: ‘Sunset or Perish’

On Tuesday I keynoted the Women in Big Data luncheon, which turned out (encouragingly) to be standing room only. Women in their 20s and 30s were there to network, hungry to meet their peers working in the big data world.

After my talk, a woman approached me and thrust her phone in my direction. “I built this app in my data science class,” she said. I examined her app (which I promised to keep secret) and was duly impressed, not just with the functionality but with the idea. Where did she come up with that? I thought. Risking embarrassment, I asked, “What did you use to build it?”

Suffice it to say I thought “Docker” was something men put on one leg at a time and “Flask” was something you hid in your desk. The young woman continued to explain how she would soon replace her Kron jobs with streaming data. She was smart and earnest and she addressed me as if at any minute I would be asking to review her source code.

I hope that hers is the new face of big data. Hell, I hope hers is the new face of technology. Her energy, smarts, and passion reminded me how much there was to learn. I made a mental note to check out “Hacker Monthly” and commit to a daily dose of gingko biloba.

The different perspectives at Strata Hadoop World reflect the divide among attendees, and indeed of the big data industry as a whole. Some of us are looking for specific solutions to targeted problems. Others are in search of the next big enterprise stack to replace the old one. But what we all have in common is interest in what’s new, what’s next, and the most straightforward solutions to the data’s growing complexity. It’s a quest on which everyone in the San Jose Convention Center was a fellow traveler.