A new research by Forrester exposed that many CXOs have a severely inflated sense of their firm’s DevOps and Continuous Testing maturity.
New Forrester research reveals three key differentiators that separate DevOps leaders from DevOps laggards. The study, which was led by Forrester VP-Principal Analyst Diego Lo Giudice and commissioned by Tricentis, found that firms with the most successful DevOps initiatives do a number of things differently from their peers as they are:
- Transforming software testing into Continuous Testing by adopting five core practices. These include automating end-to-end functional testing and integrating testers into cross-functional teams.
- Almost twice as likely to consider automating the software quality process to be a “critical business differentiator.”
- Significantly more likely to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).
The research also exposed that many CXOs have a severely inflated sense of their firm’s DevOps and Continuous Testing maturity. According to the report, 57% of CXOs believed that their organization was following all the “leader” best practices—but only 26% actually were. According to the report:
“While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers—and their behaviors and attitudes support that assertion.”
Different DevOps Quality Metrics
The study also identified the 20 most important DevOps quality metrics that separate DevOps experts from their less advanced peers. Key findings include:
- Understanding of business risk is the most important determining factor of DevOps maturity.
- Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
- Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.
The Risk “Blind Spot”
Risk was a prevalent thread throughout the various parts of the report. Risk-related metrics did not rank high in overall popularity. But DevOps experts measure them significantly more frequently than the non-experts did. In fact, DevOps experts consistently ranked risk-related metrics among their top three most valuable metrics in different phases and categories.
Interestingly, most firms (80%) believe they deliver within acceptable business risk. Despite this, fewer than a quarter state that their QA and testing processes completely cover business risk. Only 15% of respondents say that their test suites reliably provide a good indication of business risk.
The study finds most test suites actually cover about 40% of the organization’s business risks. However, these test suites have a 67% level of redundancy on average. This means that over two-thirds of the tests don’t actually increase their business risk coverage. This might be one cause of the disconnect. People assume that more tests mean more business risk coverage, but that’s often not the case.
Another likely cause is that most test results focus on pass or fail status, which doesn’t provide the required insight into whether the release has an acceptable level of risk. Today more companies are moving toward Continuous Delivery of real-time insight. Here business risk becomes critical for promoting acceptable releases as rapidly as possible. At the same time, it helps potentially damaging ones from proceeding down the release pipeline.