Looking Back at the Year in Cloud Computing

Just about a year ago, I published my 2014 cloud computing predictions. It’s been an amazing year in cloud computing, with a few of these highlights:

• Amazon Web Services continues to grow and innovate. At its November Reinvent conference, the more than 13,000 attendees learned about its many new services, including new Lambda, as well as new instance types (and Intel appeared, talking about how it had designed a new, highest-performing-ever chip just for Amazon).
• Microsoft, charging from behind, seems to be making up lost ground, and, according to all reports, is now a significant player, leveraging its capability to deliver a homogenous hybrid cloud offering.
• Google made it clear that it wants its place in the cloud universe, using price slashing (historically the tool of a new player trying to squeeze its way into a market) as its weapon.
• Both HP and IBM brought new public offerings to market (Helion and BlueMix, respectively), indicating they aren’t planning to forfeit the public cloud computing market by staying stuck in their legacy on-premises businesses (although, to be noted, both provide on-premises solutions that are compatible with their public offerings, thereby providing customer value through consistency and portability).
• The OpenStack Foundation held two Summits with rapidly growing attendance, indicating high level of both vendor and user interest. While OpenStack the platform is still maturing, it’s clear that, as an industry, we have chosen OpenStack as the only meaningful nonproprietary software platform for operating cloud infrastructures.
• More generally, cloud computing is continuing its journey to becoming *the* de facto enterprise computing platform. It’s still early in this journey, but I’m no longer hearing skepticism about cloud’s benefits, as I did even two years ago. Today, the industry — both vendors and users — is trying to figure out how to best extract value from cloud. The discussion about whether cloud computing is real, however, is done. It’s obvious it’s real; now the question is how, not why (although, sadly, there is still too much fixation on what, as in “what is cloud computing,” as though a definition needs to be fixed before getting on with the job).

Given these developments, I thought it would be worthwhile to revisit my 2014 predictions, both as a perhaps-humbling effort to evaluate my foresight, but also to measure how the torrid pace of innovation in the cloud market has supported or outstripped what I expected to see during this year.

So, without further ado, here are my predictions, a self-assigned grade, and a comment on each of my 10 2014 cloud computing predictions.

User Predictions

2014 Prediction 1: More businesses become software companies. Grade: B

Comment: In my prediction, I quoted Marc Andreessen’s statement that “software is eating the world” and went on to say:

The net effect of the ongoing shift to IT-wrapped products and services is that global IT spend will increase significantly as IT shifts from back office support to frontline value delivery. The scale of IT will outstrip on-premise capacity and result in massive adoption of cloud computing.

I would say I was correct directionally, but premature operationally. Most of the IT organizations I’ve interacted with this year are highly motivated to accelerate their application pace of delivery, and many are under pressure from business units to help them roll out new products and services. As I noted in my last blog posting, this is leading many IT organizations to “cross the chasm” toward behaving more like a software startup, with the growth of corporate innovation labs illustrating this trend.

However, most IT organizations are still developing the capability to operate like a software company. They are sorting through key issues like process, toolchains, and employee skills, trying to develop a roadmap to get them to the place they need to be. So … this will come, just not as quick as I predicted.

2014 Prediction 2: Application developers become more important. Grade: A+

In my prediction, I quoted the analyst firm RedMonk’s mantra as “the developer as kingmaker,” and discussed the enormous change in development practices as reflecting the increased importance of developers. The accuracy of my prediction could be seen in the response to my recent blog posting “The Coming War for Developers.” It had, by far, the largest number of pageviews of any post I’ve ever ;done, indicating the intense interest in this topic.

In my prediction, I commented:

It will be interesting to watch mainstream companies address the new importance of developers. Many of them have traditionally downplayed the importance of IT and treated it as a cost center to be squeezed and indeed gotten rid of entirely via outsourcing. When these companies start to ramp up their app development efforts they will confront an expensive resource pool with plenty of job options.

Frankly, it’s fascinating seeing this play out. IT organizations are riven as they try to come to terms with bringing in this new breed of employee. I’ve seen lots of organizational angst as long-time employees resent and resist the arrival of the new breed of developer.

Many IT organizations are throwing in the towel on trying to integrate the old and the new, and are moving to what Gartner calls “bimodal IT”: Type 1 is the old style, which is left undisturbed, while Type 2 is the new, software company-style mode, operating under completely different rules.

I plan to write in the near about the implications of bimodal IT, but for the moment let me say this solution reflects the desperation senior IT executives feel to incorporate and accommodate new employees on their march to operating as a software company, a la the previous prediction.

Application workload placement decisions continue to shift to end users, which will result in more cloud computing adoption. Grade: B

In this prediction I discussed Gartner’s much-discussed prediction that CMOs would control over 50 percent of all IT spending by the year 2017, and what this would imply: much less commitment to on-premises equipment and much more interest in moving quickly, which would result in more public cloud adoption.

I believe I was right that non-IT execs are much less emotionally committed to on-premise (or its near-identical twin, colo) and much more likely to be attracted to the immediate availability and flexibility of public cloud computing. However, I’m seeing much more partnering between business units and IT, with IT leading the infrastructure decisions, rather than business units making the deployment decisions.

However — and it’s a big however — I’m also seeing much more interest from central IT in using public cloud computing. I’ve been surprised in speaking with a number of large enterprise IT organizations, the kind that 18 months ago would have insisted they would never, ever use public cloud computing due to concerns regarding security and compliance, tell me “now that we’ve had a look at it, the public providers actually do have good security.”

One company went so far as to tell me that their internal security staff had concluded that, in fact, the company could probably have a better overall security posture using a public cloud than they could by operating the infrastructure themselves.

So, with respect to this prediction, it’s perhaps the same outcome — use of public cloud — but driven by different decisions makers. Instead of end users deciding for themselves, IT is taking the lead. The end result is the same, though: significant increase in public cloud use.

Private cloud has its moment of truth. Grade: D

In this prediction, I said:

The problem has been that many of these private cloud initiatives have been extended processes, bogged down by budgeting, lengthy vendor assessments, employee skill building and, yes, internal politics. While in the past this kind of delay could be accepted as part of the growing pains of shifting to a new platform, 2014 will force companies to really assess the progress of their private cloud efforts.

What I wrote in 2013 continues to be true in 2014. All too many private cloud initiatives have failed to get traction, and remain mired in a bog of technology confusion, motivation of the various involved parties, and the lack of the operational skillset required to operate dynamic infrastructure.

OpenStack, which appears to have emerged as the technology industry’s victor in the orchestration wars, has suffered due to lack of stability, resulting in an inability of end users to adopt the product easily. One of the most striking things at the last Summit was a chart I saw with figures presented as to deployment sizes: by far the largest number of deployments are 50 nodes or fewer, which is really a POC size, not a production environment. In other words, OpenStack is mostly not moved into production yet. A more stable, more easily implementable, more easily operated OpenStack would help users immensely.

On the other hand, even with a stable orchestration product, IT organizations aren’t out of the woods. Most private clouds are focused on virtual machine self-service, with little integration of end-to-end automation, additional services (e.g., a message queue service to facilitate loosely coupled application topologies), or real scale and elasticity.

I saw Lydia Leong, the Gartner analyst responsible for the firm’s public IaaS, speak recently, and her comment on private cloud was that even when one is in place, developers shun it in preference to public offerings that provide richer functionality and scale.

So 2013 wasn’t the private cloud moment of truth, but it wasn’t private cloud’s moment of success, either. I suspect 2014 will see more of the same as 2013 — good intentions but, at best, mixed results.

Cloud brokerage comes into focus. Grade: F

Many cloud management systems provide (or claim to provide, anyway) the technology to deploy workloads across multiple cloud environments. However, I had something beyond this in mind in this prediction, as I wrote:

The need for cloud brokerage extends beyond the technical, by the way. While a “single pane of glass” using consistent tools and governance across a variety of cloud environments is crucial, managing utilization and cost in those environments is just as important, and that will become increasingly evident in 2014.

There has been some movement toward this vision during 2014 with a number of companies announcing or bringing to market what is commonly termed “cloud marketplaces:” a portal that enables a service catalog of functionality from different cloud providers; some of these marketplaces offer limited capability to predict the cost of running a specific workload across a number of providers, thereby allowing users to use cost as a workload deployment criteria.

This is not anywhere near the level of sophistication I had in mind in discussing cloud brokerage. If you’re going to operate your business with software comprising a significant part of your offerings, you have to have an accurate, near real-time capability to understand the actual cost of running your applications. You have to have the capability to do scenario analysis (e.g., what happens to my computing costs if I get 10 times the application load I predict). This is because the cost of supplemental applications is now a part of your offerings’ COGS, and as such needs to be incorporated into financial calculations to allow accurate marginal cost and profitability analysis.

Frankly, I expected more need by users to adopt the offerings available to perform this kind of analysis. Instead, this capability appears to be experiencing some growth, but nowhere near the kind of explosive interest one would expect for what should be a fundamental need for the business of the future. Maybe I am too visionary, but I sure whiffed on this one.

Vendor Predictions

AWS continues its torrid pace of innovation. Grade: A+

I noted in last year’s predictions:

As AWS crosses the $4 billion revenue mark, it doesn’t seem to be decelerating its innovation progress; far from it. At the recent re:Invent conference I wrote about in my last post, AWS announced four or five major new offerings, and pointed out that it would deliver over 250 total new offerings or service improvements in 2013. Nothing indicates that 2014 will be any different and one can expect many new AWS services and service offerings during the year.

I was dead-on in this prediction, and AWS continues to amaze. This year’s re:Invent conference brought announcement of the aforementioned Lambda service, which allows users to hang code components directly on AWS services, doing away with the need to deploy or operate virtual machines instances entirely.

Re:Invent also saw the announcement of Aurora, a MySQL-compatible, highly available, enormously scalable database service costing just $.29/hour. From my seat, it looked like AWS is going right after Oracle’s fat-margined database business.

AWS also announced a number of IT operational services offering things like key management, code management, and so on, making it clear that its ambition is not just to offer a public cloud service, but to become a fundamental IT systems management company.

It is remarkable how rich the AWS service is, and how Amazon keeps improving and extending it. To get a sense of how innovative AWS is with respect to its core infrastructure, view this re:Invent video by James Hamilton, AWS’s data center guru.

Leong says that AWS is by far the richest and most built-out cloud service and continues to set the bar for everyone else in the cloud computing industry. If I were to make a prediction about what 2014 will see in the CSP industry, I would be a number of providers will throw in the towel when they realize they have no hope of ever offering something anywhere near as capable as AWS.

Google and Microsoft get serious about cloud computing. Grade: A

In last year’s prediction, I wrote that Amazon had had a free run in the commodity CSP business, but I added this:

That unopposed free run is going to end in 2014. Both Google and Microsoft have AWS in their crosshairs and are rolling out serious competitive offerings, designed for an all-out battle royale. Both have, finally, recognized that their initial cloud offerings were inadequate (both, to my mind, seemed like offerings that customers should find superior to AWS, and both companies appeared baffled that the to-them clear advantages of the offerings went unappreciated by potential customers). And both have now, with version 2, delivered directly competitive cloud offerings.

Both Google and Microsoft fulfilled my expectations. Microsoft seems to have recognized that cloud computing is not just something it needs to have as part of its future product set, but that Azure is its future. It indicated this by naming a new CEO who was previously in charge of its cloud service, making sure that its cloud vision is front and center for the company’s executive team. And, as previously noted, Microsoft seems to have gotten going in a major way with Azure experiencing significant growth in 2014.

Also, as previously noted, Google used price competition to wedge its way into the market, illustrating its willingness to use its extremely deep pockets to become a major player in cloud computing.

On the other hand, I continue to hear criticisms of both companies in terms of their offering’s functionality and resilience, so it’s clear that they are both experiencing growing pains as they fight AWS. One can expect to see 2015 continue this battle, as both companies seek to occupy what they perceive as their pride of place in the first order of cloud providers.

Importance of ecosystem becomes clear. Grade: B

Last year, I wrote the following:

Today, the richness of the CSP ecosystems is completely lopsided. Not only does AWS provide a far richer services platform than its competitors, it has by far the larger number of complementary services provided by third parties. In 2014, as more applications get deployed to public cloud providers, the importance of the ecosystem will come into focus.

There’s no question that AWS has the richest set of services; no cloud provider offers anywhere near the broad portfolio of services, not to mention the dizzying array of third-party services offered on the AWS platform. Moreover, many CSPs fail — still — to understand how important ecosystem services are, continuing to bleat on about how “enterprise” they are, pointing to their very expensive kit as a differentiator. Another set of providers appear to realize that complementary services are important, but follow a strategy of trying to get external parties to host their services on the provider’s infrastructure without implementing any integration in terms of account management, billing, or operational tooling. In effect, they appear to confuse hosting a bunch of SaaS offerings with providing a rich ecosystem.

The future of cloud computing is all about applications, and a rich ecosystem enables faster application development and deployment, and easier application monitoring and management. AWS gets that, making it the easiest environment on which to deploy applications. It’s not easy to provide an integrated ecosystem, but its power is amazing.

One could say that there is growing awareness of the importance of a cloud’s ecosystem, but not as much movement toward delivery as one might expect. Therefore, this prediction rates only a B.

VMware realizes vCHS is critical to its future. Grade C

As I said last year:

VMware has been in a funny position with respect to cloud computing. Its undoubted platform advantages inside the corporate data center haven’t been matched by a concomitant public cloud success. For whatever reason (or, perhaps, for a number of reasons), VMware’s public CSP partners haven’t been able to generate large adoption for the VMware flavor of cloud computing.

I continued:

Next year will be extremely important for VMware and its vCHS offering. As noted above, companies are making decisions right now that will set their course for the future. If VMware hopes to play as important a role in public cloud computing as it does in internal data center environments, it needs to be part of those decisions. There’s not a lot of time left to gain a spot on short lists. One can be sure that VMware recognizes how important 2014 will be to its future, and that it is planning an aggressive campaign to maintain its market-leading position.

VMware didn’t really do much to change this state of affairs in 2014. It renamed its hybrid offering, yet again, saying it would henceforth be called vCloud Air. However, it mostly seems to recasting itself as top-to-bottom infrastructure provider, saying the Software Defined Data Center (SDDC) is what IT organizations really need, with VMware being the vendor that can best deliver this vision.

While there is a public cloud component to this, it is presented as one choice among many, and, unmistakably, has the feel of a poor stepsister within the SDDC firmament. To my mind, VMware firmly believes that its (and the industry’s) future is with on-premise computing, and it is planning to be the dominant vendor there.

I’ve had many conversations this year discussing the increasing shift to public cloud computing that had my discussion partner nodding his or her head but disagreeing with my perspective, citing the enormous amount of industry investment in on-premise equipment, with the clear implication that public cloud computing is just a niche use case. We’ll see how this plays out long-term, but it would see that, in the short-term at least, VMware is with the on-premise crowd.

Pricing bloodbath in public cloud computing. Grade A

As I said last year:

One of the key differences between AWS and most of its competition is cost. While much of Amazon’s competition has aligned their pricing with existing hosting models Amazon makes it easy to get started for a few dollars with no commitment at all. In effect, this has meant that Amazon is competing with itself, and, to its credit, it has reduced prices since it first launched AWS. That field of one is going to expand this year with the arrival of Microsoft and Google, and the result is going to be a ferocious price war, with the three of them repeatedly dropping costs.

Well, for sure this is what happened in 2014. We saw a repeated round robin effect — one big three provider would drop prices (often keyed to another provider’s yearly event), with the other two scurrying to match prices. A lull would ensue, until the next set of price drops, whereupon the same dance would once again occur.

The result has been predictable. The lower prices have stimulated adoption in the big three, as many IT organizations have mothballed private cloud plans, recognizing that economics that might have made an on-premise infrastructure viable at one price point are made obsolete by new, lower public prices. Many other users have responded to lower prices as one would expect — with glee, although one commentator seemed to feel that low prices are somehow bad for users.

And, just as predictably, these price cuts have affected other CSPs. Rackspace, for example, announced it would move away from matching the prices of the other players, and refocus its efforts on services that would support higher prices. This year will see other CSPs realize they are unable to compete with the big players and either refocus, a la Rackspace, or throw in the towel on the entire industry.

I believe we will continue to see ongoing price cuts among the big three — perhaps not as large or as dramatic, but steady nonetheless. They all recognize low cost is a way to stimulate demand, and they all view price as a competitive weapon.

Conclusion: Overall Grade B+

In my 2014 predictions conclusion, I said:

In a sense, what has happened in the industry up to this point has been the prologue for the main cloud computing story. Next year represents the beginning of the main story. In 2014, we’ll see cloud computing become the dominant platform for IT from now on. There will be many successes as users learn to take advantage of the new capabilities cloud computing offers, along with challenges to many in the industry — both users and vendors — who struggle make a successful transition to the platform of the future.

My predictions were, I feel, on the whole correct. Where I was wrong was in expecting change to occur more quickly; in particular, I expected people to recognize the implications of what is changing and to implement plans to respond to these changes. I expected, so to speak, more skating to where the puck will be; instead, there was more watching the puck go by and belatedly skating after it.

I suppose that is probably what could have been anticipated. It’s always amazing to see how slowly organizations (and IT organizations, specifically) respond to important inflection points in their environments. Notwithstanding this typical laggardly response pattern, there is an undeniable, palpable sense of dramatic change in the air.

Gartner’s Type 2 IT and IDC’s Third Platform indicate that there is a new day arriving in IT, and everyone will need to respond to it. I firmly believe we are in the most dramatic time of the Information Age, which is the most dramatic period across all of human history — and that the dramatic changes we’re seeing are accelerating. I expect 2015 will be an exciting year for cloud computing.