Digital Transformation and the Cloud have shifted all developers into a new DevOps era. But experts still rely on metrics from a bygone era to verify the quality of written code. Digital Transformation requires new applications and new DevOps environments to generate them. Nowadays, applications are evolving at sustained rates, sometimes falling within a second’s order.
However, DevOps managers still rely on the same metrics to judge the quality of their developments while evolving today in a context with totally different requirements. The analysis firm focused on the 157 managers who had successfully implemented a DevOps strategy to identify those applicable to DevOps processes. By comparing the answers of the other strategists on the panel, the analysts could deduce the metrics that could be considered DevOps differentiators.
Each interviewee provided the list of metrics used daily to qualify their developments. These are classified into four categories, Build, Functional Validation, integration testing, and regression testing. Only the 20 measures considered to be the most important were retained. They are sorted by decreasing value.
In the context of the Build category, the ranking of the measures gives an idea of those considered necessary (box Value Rank), but that does not mean that they are used daily (box Value Usage). Note that the lines accompanied by the acronym “DevOps Differentiator” are the measures found in all companies with a successful DevOps strategy. Automated tests ranked by the importance of risk are at the top of the measures considered to be of great value for 63% of voters. Still, paradoxically, in the field, they could be more used (only 27% among the most advanced companies and 15% for the others). ).
They are essential because they are the only ones to take business risks into account. Developers and testers need to know the impact of changes as code evolves. Note that this measure is present in all companies with a successful DevOps strategy. The metrics considered necessary are often related to code quality. Thus the measurement of “successful code builds,” which comes in second place with 63% of the vote of the managers, follows the “pass/fail rate of unit tests” with 60% and the measurement of the “total number of code defects” considered by 59%.
Around 30% of DevOps managers use all these measures daily. The “StaticAnalysisResults” results of static analysis of the code to discover its weaknesses are part of the measures qualified as DevOps Differentiators by mature DevOps companies. Also on the list of differentiating elements are the following:
- “Test Case Coverage” allows you to know to what extent the needs in terms of an application’s functions are covered.
- The “requirements covered by tests” relate to the number of prerequisites covered by the tests.
- The number of automated and hygiene tests (cleanliness, efficiency, and usefulness).
Of the top four functional testing metrics, the favorite of DevOps experts, with a 69% rate, indicates the percentage of prerequisites covered by the tests. They are only 40% on the ground to use it. For 66% of respondents, the “Critical Defects” metric, which gives an idea of the evolution over time of the functional defects identified, is the second most important measurement, with only 21% using it.
This is followed by the success/failure rate of available tests for 64% of managers (only 46% measure it), then the metric relating to the quality of the application, “Defect Density,” the ratio of the number of defects identified according to the size of the application (62% of voters for 33% of actual users).
The measures taken here mainly relate to integration and API testing. The importance of APIs in the current DevOps context is well established. Distributed architectures and services/microservices make these metrics more than necessary. The first four concern the same tests as those listed in the functional validation phase.
For 75% of the experts, the percentage of the coverage of the difficulties relating to the prerequisites is vital, while only 34% use it. The measurement of the sum of new defects found in API tests is essential for 64% of the people questioned, and only 30% are concerned about it in the field. The quality assessment of APIs is carried out by 39% of experts, while 63% declare it essential.
End-To-End Regression Testing
Three of the four metrics ranked as the most valid by DevOps managers are also considered DevOps Differentiators. And for once, this distribution also corresponds to the practice on the ground. Considered a priority by 70% of DevOps experts, measuring the percentage of automated tests is performed by 36% of people. Equally, 70% of managers consider that the coverage of prerequisites by tests is essential for 40% who measure it. The total number of defects identified is followed by 33% of respondents, while 66% consider it necessary. Finally, counting the number of test cases is essential for 65%, whereas only 41% of the experts carry it out.
The Metrics Trend
In conclusion of this study, three essential points stand out according to Forrester Research:
- Understanding business risk must be considered a vital factor for the success of a DevOps project.
- When a DevOps policy is successful, the manager relies more on contextual measures, such as the coverage of a requirement, than on considerations of the number of times a task is executed (such as the number of tests ).
- The global vision is the most relevant for obtaining quality development.
Most of the time, DevOps experts focus on knowing the measurement of a user experience during an end-to-end transaction and not based solely on specific measures of a particular point.