It’s big, but is it clever?

I’m in Silicon Valley for a month. The Dictionary of Words of the Moment is alive and well, and no table can be passed in a Palo Alto restaurant without hearing the words cloud, social, virtualisation or big data.

In terms of Silicon Valley speak, cloud is now so much a given that it conveys no information, and is thus almost a little passé. Although Hollywood has made a film about social media, while it might still be important, mentioning it at Valley parties does not get you on the cool list.

In fact, I must admit to being a little concerned with The Social Network movie. Yes, I remember those days of lunatic youth when we thought nothing of staying up all night coding, but somehow I have forgotten the fact that there were also beautiful women draped over the sofas while we were doing it…

Funny how time plays tricks on memory — I’m so glad the filmmakers paid attention to those details.

To be cool at the Valley party these days, you have to talk about big data. The problem is, no one can quite tell you what exactly that is. It’s not data warehousing or Business Intelligence — because that is so boring.

In-memory databases come up a lot and are often ascribed magical properties, however all I can seem to learn about this is they are faster due to the avoidance of going to disk. One can get the same answers for unexpected unoptimised queries as one could for boring databases, but quicker.

So just as dumb, but quick dumb. But they are usually mentioned by software marketing people with a twinkle in the eye and an implication that somehow you will also get better answers, but no one can explain why.

So, to be cool and feature in the next Silicon Valley movie, try knocking ’em dead with the following big data gambits: NoSQL and Hadoop.

For those of you that wish to casually toss this word in, Wikipedia will tell you Hadoop is a data-intensive framework that works with thousands of nodes and petabytes. It was originally developed to support distribution for Nutch. If you are no wiser at that point, then bluff.

While Hadoop and NoSQL have perfectly respectable academic, dev and nerd communities behind them, you don’t have to look far to see what happens when marketing departments get their hands on the big data idea.

In university computer departments, some of those researching big data started using the term unstructured data to refer to traditional database-type matching operations performed over what are, in effect, messy databases.

This was a slightly unfortunate term to pick as for many years it has been used to refer to human-friendly data that does not operate well with traditional database operations like email, audio and video. But it’s not a problem, as any self-respecting academic would understand the difference.

The problem arises when the trendy young software marketing executive is sent to talk to the nerds about the next great push to sell more of the old stuff under a new name, and before you know it unstructured, as in messy database and unstructured, as human-friendly are fused, and the idea that a traditional database can handle human-friendly information is stated.

So we’re in for a few months of chaos and crazy statements before the terminology is cleaned up. We have already seen one large vendor of in-memory databases launch with a set of crazy claims, only to back-pedal as they realised marketing had not only confused the customers, but also themselves.

Big data is a generic term and will no doubt have benefits, but it needs clarity of definitions and claims before it can truly assert itself as a revolution rather than an evolution of some rather tired ideas. My mind is open and I await that clarity when the hype dies down.

Recently I went to dinner with a small group containing one of the ultra-famous Valley venture capitalists. He started the evening by declaring his new concern for the environment in an evangelical speech invoking images of polar bears balancing precariously on one paw on a lonely ice cube.

Indeed, he had taken action. He had bought a hybrid car. The guests agreed the need for such thinking and we moved on.

An hour later the same VC mentioned in passing how he was changing his jet as his Labrador kept throwing up in it. I looked around (while trying to appear quite au fait with this kind of predicament, drawing on the parallel knowledge of my seven-year-old and a VW Polo) to realise no one had spotted the irony.

Irony may be a missing element but no one notices in a Valley of elemental Silicon. Here one can make the impossible possible and should not be constrained by having two mutually incompatible ideas in one’s head at the same time, and thus magical things can happen.

But before I sign off, have you heard about this great new start-up that’s using cloud-based, virtualised, social Hadoop to correct global warming…?

Mike Lynch is the founder and CEO of UK software company Autonomy