New Research on What Separates DevOps Leaders from DevOps Laggards

New Forrester research reveals three key differentiators that separate DevOps leaders from DevOps laggards. The study, which was led by Forrester VP/Principal Analyst Diego Lo Giudice and commissioned by Tricentis, found that firms with the most successful DevOps initiatives do a number of things differently from their peers:

• They are transforming software testing into Continuous Testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
• They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator.”
• They are significantly more likely to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

The research also exposed that many CXOs have a severely inflated sense of their firm’s DevOps and Continuous Testing maturity. According to the report, 57% of CXOs believed that their organization was following all the “leader” best practices—but only 26% actually were. According to the report:

“While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers—and their behaviors and attitudes support that assertion.”

Different DevOps Quality Metrics

The study also identified the 20 most important DevOps quality metrics that separate DevOps experts from their less advanced peers. Key findings include:

• Understanding of business risk is the most important determining factor of DevOps maturity.
• Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
• Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

The Risk “Blind Spot”

Risk was a prevalent thread throughout the various parts of the report. Although risk-related metrics did not rank high in overall popularity, DevOps experts measure them significantly more frequently than the non-experts did. In fact, DevOps experts consistently ranked risk-related metrics among their top three most valuable metrics in different phases and categories.

Interestingly, most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Only 15% of respondents say that their test suites reliably provide a good indication of business risk.

Tricentis

From Tricentis’ analysis of Global 2000 companies’ test case portfolios, we’ve found that most test suites actually cover about 40% of the organization’s business risks. However, these test suites have a 67% level of redundancy on average—meaning that over two-thirds of the tests don’t actually increase their business risk coverage. This might be one cause of the disconnect: People assume that more tests mean more business risk coverage, but that’s often not the case.

Another likely cause is that most test results focus on pass/fail status, which doesn’t provide the required insight into whether the release has an acceptable level of risk. As companies move toward Continuous Delivery and automated release processes, real-time insight into business risk becomes critical for promoting acceptable releases as rapidly as possible while stopping potentially damaging ones from proceeding down the release pipeline.

[For additional insights and Forrester’s recommendations, read the complete DevOps Leaders vs Laggards report]

Report Methodology

In this study, Forrester conducted an online survey of 603 enterprise organizations in North America, Europe, and Asia Pacific to evaluate current software testing practices and metrics tracked during survey development. Survey participants included decision makers and individual contributors responsible for their organization’s Agile and/or DevOps development strategies. Questions provided to the participants asked about their firm’s attitudes toward software development automation, risk management, and testing practices, as well as the metrics they track and value in the software development life cycle.

Tricentis