I am privileged to serve as co-chair of the Cloud Services SIG for a Silicon Valley-based non-profit, the SDForum, which is a great resource for technologists, entrepreneurs, and investors to meet and investigate new technologies. We've been running the SIG since January, and it's been a great experience to see what people are doing with cloud computing (if you're located in Silicon Valley, please come to one of our meetings; you'll enjoy it and learn a lot). In addition, just by virtue of being located in Silicon Valley, I get the opportunity to see lots of great new technologies — like yesterday, when I attended the Amazon Web Services Start-Up Event at the PlugandPlayTechCenter in Sunnyvale.

It is striking about how companies are leveraging cloud computing to create new products or services. I thought I would write about a few of them this week just to give an insight about how people are taking advantage of the characteristics of cloud computing.

Big Data: As you know, I am a big believer in the big data theme — that organizations are moving beyond transactions and into relationships and content, thereby exponentially increasing the amount of data under storage — and requiring much more (and deeper) analytics. Moreover, the traditional tools used to manage data, both from a pure storage perspective as well as a tool perspective (i.e., database engines, etc.) don't scale very well, either technically or economically. At the Cloud Services SIG last month, we had several companies presenting that discussed how they integrate with the cloud to better address the big data problem.

First off, we had a Google representative, who discussed Google Datastore, which is a robust key/value pair storage mechanism designed to provide massive scalability. While not offering the extensibility or flexibility of a relational database, Google Datastore addresses common cloud storage requirements, which are typically very large amounts of relatively simple data.

We then heard from Cloudera, which distributes a supported Hadoop distribution. Hadoop is a great tool to enable parallel processing of very large amounts of data with an aim of performing a relatively simple operation on some portion of the data. Hadoop, which I wrote about a few months ago, has a distributed file system that is redundantly spread throughout a set of servers, which is used to store and retrieve data. A query is launched against the data store, executing a map/reduce function, which then performs some operation on the resulting data set. Hadoop is widely used for very large data sets that outstrip the capacity limits of traditional databases and data warehouses. Incidentally, as I mentioned in my earlier blog piece, Amazon offers Hadoop functionality directly as an AWS offering.