20 Ways to Measure the Success of Your Growing Cloud Investment

While it’s well known that the cloud helps organizations “do more with less,” the follow up question of “how much?” is far less known. IT rarely asks that question, yet the business does ask, and the business wants answers. If IT is going to better align themselves with business operations, they must be able to objectively quantify the value of the cloud.

Looking to the wisdom of the crowd, we reached out to experts and asked, “How do you measure the value the cloud brings to your business?” As the responses came in we discovered there’s far more than one or even 20 different ways to measure the success of the cloud. A myriad of technical and business dynamics can be a definition of success. More importantly, what may work for one organization may not be appropriate for another. As you look through the following advice, pick and choose what’s right for your organization. Note: Please don’t skip tip number one. It’s appropriate for all businesses.

1: Define what ROI means for your business

“You need to define what ROI means for your organization first, before you identify your KPIs [key performance indicators],” advises Jeff Frankel (@docstarsoftware), executive vice president and principal at docSTAR.

“The KPIs that are used to measure the success of a cloud initiative should similarly be the ones used to measure impact on the overall success of the company itself,” adds Matt Podowitz (@mpodowitz), senior director at Pine Hill Group.

“Before implementing the cloud projects, have business/IT alignment up front to define what the business values are,” says Terence Ngai (@TerenceCNgai), head of cloud go-to-market and solution delivery at HP.

“Company strategy and employee goals should inform which tools and technology you choose – not the other way around,” says Clara Liang (@claraliang), Chief Product Officer of Jive.

Podowitz agrees: “Measuring cloud success using technology rather than business KPIs diminishes the cloud effort to nothing more than a technology project, not a technology-enabled transformation of the business capable of creating real value.”

“The only KPIs for the success of moving to the cloud are how well it achieved your reason for moving to the cloud in the first place,” concludes Nicko van Someren (@good_technology), CTO, at Good Technology.

2: Obviously, Compare costs

“The value of the cloud is around overall cost cutting,” says Michael Weiss (@Oildex), VP, software engineering at Oildex.

“Compare previous expenses around on-premise datacenter management to the expenses of cloud hosting,” suggests Kevin Fishner (@KFishner), director of customer success, at HashiCorp.

For best cost analysis, M.J. Johnson (@threeeyedtoad), director, product marketing, at Akamai Technologies, suggests measuring “reduced hardware and software costs, reduced resources required to maintain infrastructure, plus eliminated on-premise and co-located data centers including leased lines for connectivity.”

“The KPI that measures success in the cloud is more related to cost avoidance. If we add 100 users to an on premise application, what will that cost us in terms of hardware, infrastructure, licenses and help desk costs over a defined timeframe,” adds Dean Wiech (@dwiech), managing director for Tools4ever.

3: Don’t just focus on cost savings

“Many companies make the mistake of only measuring hard cost savings associated with cloud, which frequently leads to misaligned expectations during the implementation phase,” says Charles Moore (@delphix), product marketing, at Delphix.

Keep that in mind as you read the rest of the advice, which addresses mostly non-cost-based ways to measure the success of your cloud initiative.

4: Measure revenue impact

“The closer an organization can tie a cloud project to revenue, the better,” says Eric Shapiro (@ArcTouch), CEO and co-founder of ArcTouch.

“Cloud technologies allow you to more granularly align tech operations KPIs to your current revenue and business KPIs,” adds Scott Maurice (@scottjmaurice), managing partner at vail Partners. “You no longer have to take yearly budget and estimate costs; you can directly assign a technology cost metric to a specific revenue-generating application and by extension, specific revenue.”

5: Focus on customer KPIs

“Deployment of cloud, or any technology for that matter, should involve tight assessment of what it means for end customers of the business,” says Greg Johnsen (@gregjohnsen), CMO of GT Nexus, who suggests measuring the on-time delivery of goods to customers and the number of days you’re holding inventory.

Dave Davis (@DaveDavis), managing director at Redfly Marketing, adds “We’ve found that customer churn rate, average revenue per customer, free to premium up-sell rate, and what we call ‘time to slip away rate’ are what’s worth focusing on from a customer perspective and a financial perspective.”

Additional customer focused KPIs “would include an increase in transaction completion, reduction in cart abandonment, and an increase of spinoff sales for the B2C world,” suggests Steve Prentice (@stevenprentice), senior writer, at CloudTweaks.

6: Calculate ‘time to daily use’

“We like to focus particularly on a KPI we call ‘time to daily use’ which measures how long a cloud product takes to become indispensable to a customer. The shorter this time, the less chance a user will slip away,” says Redfly Marketing’s Davis.

7: Survey user satisfaction

“One of the easiest and probably most underutilized ways [to measure the impact of the cloud] is to conduct employee satisfaction surveys across line-of-business (LOB) apps,” says Todd Schwartz (@GetSkyKick), co-CEO and co-founder of SkyKick.

“KPIs based on surveys not only demonstrate that a process is shorter and cheaper, but that the users are also happier with the new cloud-based set up,” says Max Dufour (@maxdufour), partner at Harmeda.

Schwartz’s experience confirms Dufour’s theory: “What we hear from our partners about their customers is that SaaS-based apps are easier to use, make employees happier, and provide more value to end users.”

8: Gauge market responsiveness

“The KPIs for the cloud are the same they should have always been for IT in the past,”says Jeff Kaplan (@thinkstrategies), managing director of THINKstrategies. “They are greater agility and responsiveness to market demands, and ability to identify and pursue new market opportunities.”

Akamai’s Johnson concurs: “The benefit resulting from instant-on cloud-based capabilities can be measured in time to market, newfound agility to deploy new applications and enter new markets, and increased customer acquisition/decreased customer attrition.”

“Cloud deployed in a global supply chain should enable a manufacturer to know the true cost to serve a market and foster better decisions based on these insights,” says GT Nexus’ Johnsen.

For example, Alastair Mitchell (@alimitchell), co-founder and president, Huddle, has “worked with pharmaceutical companies who’ve managed to dramatically reduce the time to market of new drugs through improved cloud-based collaboration during clinical trials and with universities who’ve been able to connect research teams who are dispersed across the globe.”

“Cloud also enables organizations to move fast and deploy quickly without having to involve finance or purchasing or to think about deployment factors (space, networking, setup, etc.),” adds Gerardo Dada (@gerardodada), VP, product marketing and strategy at SolarWinds.

9: Evaluate speed and control

While speed was a common theme we saw across many KPIs, it had to be balanced with the ability to actually conduct business.

“Success for IT in the ‘cloud’ is a complex mix between allowing speed (pace of new applications, operate at speed of the business, new APIs, new microservices) while still providing IT control (governance around data access and quality, security, identity, uptime SLAs, etc.),” says Chris Purpura (@ChrisPurpura), VP of digital enterprise strategy at MuleSoft.

10: Determine the opportunity cost

“A less obvious KPI for measuring cloud success is opportunity cost, or the cost of the best alternative foregone,” says Ray Bordogna (@RayBordogna), partner and chief strategy officer at LiquidHub.

For example, Steve Herrod (@herrod), managing director, General Catalyst Partners, suggests you ask yourself, “How many employee-hours go toward the non-differentiated application and infrastructure work can be replaced by IaaS or SaaS offerings?”

In addition, the cloud affords companies to rapidly experiment. Without the cloud, the experiment opportunity is lost.

“Leveraging cloud-oriented assets can enable a business to quickly design, implement and test several choices in a timely, cost-effective manner,” continued Bordogna.

11: Don’t overlook continuity of business operations

The ability of the cloud to be always on and operational, without the excessive need for in-house technical staff, is a value add that should not be ignored.

“If a company suffers a server failure, security breach or problem due to simple human error, companies that deploy cloud-empowered data protection solutions are assured that their business remains operating smoothly. It is the continuance of operations KPI that showcases the success of the cloud,” explains Dave LeClair (@LeClairTech), VP, product marketing, at Unitrends.

This continuance extends beyond not only just keeping the lights on, but also to ongoing product deployment.

As MJ DiBerardino (@cloudnexa), CTO of Cloudnexa, notes, “When you are developing a new product, service or platform, the cloud allows you to maintain a continuous deployment model for your development operations.”

12: How does the cloud foster innovation?

“The biggest distraction for cloud adoption is the continuous focus on reduced costs on IT infrastructure. True value is gained from business gain agility with the new found ability to rapidly innovate,” says Larry Carvalho (@robustcloud), research manager and lead analyst – PaaS for IDC.

Jason Dover (@jaysdover), director of product line management, KEMP Technologies, refers to this KPI as “time to innovation (TTI).”

Shortening the TTI requires improving the ease and speed of collaboration.

“Measure connected worker engagement, the resources required to maintain engagement, and retention as it relates to delivering the business goals,” suggests Eric Hanson (@fuze), VP of strategic initiatives at Fuze.

13: Can you identify and solve a specific problem?

“The cloud allows IT professionals to focus on the actual business problem rather than focus on the intermediate technical disciplines,” says Dan Carney (@llnw), VP, operations at Limelight. “This is very healthy as it drives the IT staff [who understands the technology] to engage in business discussions about how the user [who does not necessarily understand or care about the technology] perceives the service.”

In one example, Honeywell, a high-tech company, outfitted its sales staff with a low-tech solution: phonebook-sized catalogs tied together with shower curtain rings. It wasn’t a pretty sight, and it definitely didn’t instill confidence in potential buyers. With the help of ArcTouch, they solved the problem by producing an easy to access catalog application that was connected to Salesforce data.

“With the new solution, Honeywell staffers could instantly capture new business leads, share meeting outcomes, define next steps, and set reminders,” says ArcTouch’s Shapiro. “At a single two-day trade event, the app helped Honeywell capture more than $1 million in new business leads, which more than paid for the solution we developed.”

14: Calculate mean time to solving problems

“The big KPI for cloud infrastructure for internal development is average time between bug report and solution deployment,” says Jeffrey Bolden, managing partner at Blue Lotus SIDC. “Most companies that move toward a cloud culture see reductions of 30 percent to 80 percent in this key metric of IT responsiveness.”

“Time spent on a key process or cycle is another good KPI,” adds Harmeda’s Dufour.

This could be the time spent on back-office tasks such as the time handling and processing invoices, suggests GT Nexus’ Johnsen.

15: Compute the time to develop apps

Neal Bradbury (@IntronisInc), co-founder and VP of channel development at Intronis recommends measuring “provisioning speed, or how quickly a business is able to set up a working application in the cloud.”

“This KPI measurement includes the time to prepare the infrastructure/VMs for dev, QA, staging, and production, which is often a major delay in enterprise shops,” adds Bernard Sanders (@BernardCBolt), CTO of CloudBolt Software.

You need to also include your ability to not have to seek third party support, and instead use the development resources and tools of your choice.

“This is important because skill sets vary and most enterprises don’t have the mobile skills they need,” notes Sravish Sridhar (@sravish), founder and CEO of Kinvey.

“Sure, you need to ask yourself if you’re creating something that will enrich the experience for your customers, but more importantly, are you doing this quickly enough to realize the benefit?” asks Michael Henry (@executionmgmt), CIO at Digital Realty.

16: Measure velocity, not agility

“Agility is difficult to measure, but velocity is not,” argues Ariel Tseitlin (@atseitlin), partner at Scale Venture Partners.

While tip #8, gauge market responsiveness, is valuable, it is still difficult to measure. Other types of speed, which many experts recommend, are far easier.

“Measure the time it takes to procure new servers. Measure the time from idea inception [product requirement] to production deployment. These will all give a good proxy for velocity,” adds Tseitlin. “Measuring the number of new features deployed can give an indication of agility, but that is often a very coarse measure.”

17: Quantify the reputation of the IT group

CloudBolt Software’s Sanders jokingly suggests that “this all important KPI can be calculated by subtracting the number of curse words people use when talking about their IT department from the number of positive adjectives they use.”

On a more serious note, “cloud success is no different than all IT success,” adds Heinan Landa (@OptimalNetworks), CEO of Optimal Networks. “Measure the number and severity of issues that your people have with the service.”

18: Determine the security metrics that measure exposure

“The essential proactive security question is, how easily could someone break in? Building this as a security KPI involves understanding the assets and their networked relationships,” says Dr. Mike Lloyd (@dr_mike_lloyd), CTO of RedSeal. “The ideal outcome and measurement benchmark is risk – the probability of a bad thing happening, times the downside of that event. Reducing risk is a complex business, but for purposes of cloud infrastructure, the essential point is that you must have enough visibility to answer a question like ‘How easily could a bad guy break in and steal assets?’ or ‘How easily could a compromise of one cloud server [or service] spread to damage the rest of my organization?’”

19: Calculate availability

“The cloud should ‘always be there’ – even if part of it is broken, it should look there,” says Sean Molloy (@TenableSecurity), VP, cloud services at Tenable Network Security.

“If you have hardware or system failures in one environment, then you can move those users to another environment without impact to the customer,” says Oildex’s Weiss regarding the benefit of cloud availability.

Availability measurement works best “for cases where cloud consumers leverage the geographic diversity and redundancy options that the cloud offers,” adds Craig McElroy (@contegix), co-founder and CTO at Contegix.

20: Enforce KPIs to determine the success of your SLAs

“KPIs should naturally feed into the service level agreements (SLAs) you have with your cloud provider. KPIs are what will define success or failure in your SLA,” says Setu Shah (@setushah2), business solutions manager at Orange Business Services.

Steve Falkin (@HBR_Consulting), managing director at HBR Consulting, suggests that your performance-based and financially-oriented KPIs should be part of a contractual SLA with the service provider. Such KPIs to measure and back up with an SLA could include availability, recoverability from a disaster, turnaround time for applications and compute resources, plus actual costs and resource utilization versus what was projected.

Conclusion: What’s your KPI for not moving to the cloud?

Even though the cloud seems like such a logical move for so many, that doesn’t mean one can completely ignore all the questions that need to be asked.

“Too often leaders fail to probe deeply enough to uncover the true motivations behind a move to cloud. Probe,” advises Shlomo Swidler (@ShlomoSwidler), CEO of Orchestratus. “Ask ‘why’ repeatedly, until the answers can be measured in terms of cash flow. Along the way, you will have uncovered the KPIs that indicate cloud success.”

DISCLOSURE: David Spark and Spark Media Solutions work with Tenable Network Security.