How to Measure Success in DevOps - TriNimbus

How to Measure Success in DevOps

By this point in our DevOps series of articles, you should be armed with sufficient information and insights to start your positive transformation to DevOps and start your organization’s journey to greater efficiency and effectiveness. Before you go further, however, start applying meaningful metrics to baseline where your organization stands today and keep them running in order to measure future progress.

Most of the metrics discussed here are technical, but we will not ignore equally important “soft” metrics that may not lend themselves to easy quantification. At the risk of being overly repetitive, it is critical to understand that DevOps is fundamentally about process and that goes double for the very human processes of communication, collaboration, responsibility, problem-solving and leadership, which are all vital elements to your DevOps success too.

The Hard Measures of Success

Development to Production Lifecycle Duration

Many consider lead time to be the flagship metric even though it does not take into account the very significant efforts that occur post-deployment. However, if DevOps helps you reduce the time from first code to customer delivery, you have won a consequential battle.

In Trinimbus’ experience, it is less important where you set the start and end markers than being sure you use the same markers pre- and post-DevOps. Also, regardless of whether you use agile, waterfall or some other development methodology, set several sub-markers in between because these are going to help you fine-tune your processes at each iteration.

Rate of Change

There are many ways to measure change (versus churn) in your development, test and operations activities. Choose the ones most relevant to your organization and stick with them for as long as they remain meaningful. These might include the frequency and quantity of new code or features being written, the number of test cases created, run or discarded or the quantity of infrastructure or service deliveries by Ops.

Two things to keep in mind:

  • When selecting such metrics, get staff feedback on their value. This approach often provides a better gauge of their business value as well as ensuring that future improvements in these measures are motivating to those individuals who feel responsibility toward them.
  • Weigh the metrics you choose by the degree of complexity they entail or measure complexity separately as it may change for similar tasks pre- and post-DevOps. For instance, testing could be expected to employ more test automation. Initially, this might slow down test production, but require additional skills from testers.

Rate of Failure and Recovery

You can use a number of sub-metrics to measure how often or how badly software deployments are failing to meet customer expectations. For example, the overall measure could be rolled up from quantities or rates of functional defects, crashes, UI flakiness, documentation mistakes or negative user comments on social media.

To gain better insight into what might be going wrong if deployment failures increase, look for a correlation with the rate of changes and change complexity, both of which could be expected to initially have such an effect.

Also, take into account the average length of time it takes for the organization to recover from a failed deployment. Trinimbus has seen this measure improve even when the initial rate of deployment failures temporarily increases.

Customer Satisfaction

Measuring long-term improvement in customers’ satisfaction with your products relies on mushier metrics, but the usual suspects should still be rounded up: ticket volume, rate of new user adoption and social buzz or feedback.

Additionally, measure now and over time any change in the ability to service customer problems. This could improve for two reasons, one being that your customer service has become more efficient due to DevOps. The other reason may be that DevOps has, as it should, improved product usability such that users have fewer problems that are more easily remedied.

Softer Measures of DevOps Success

When it comes down to calculating if DevOps is working for your organization, it sometimes boils down to measuring the change in the “vibe.” The metrics will be harder to put your finger on and many that you applied making your pre-DevOps baseline may not make sense post-DevOps.

Regardless of the vagaries, per our experience helping enterprises make the DevOps journey, there is ample anecdotal evidence for positive developments:

  • There is a reduction in conflict and stress with a commensurate increase in collaboration and improved communication both quantitatively and qualitatively
  • Knowledge sharing increases, which lays a foundation for cross-skill development and shared responsibility for keeping the wheels moving along smoothly
  • Silos of expertise dissipate in favor of multi-disciplinary teams that center on projects not expertise
  • Staff feel a greater degree of autonomy, which leads to increased motivation and rewarding work experiences
  • Transparency across the organization improves, which boosts organizational cohesiveness

Conclusion

Change is never easy. Measuring the effects of change can be even more difficult, especially when it comes to the fuzzier benefits of DevOps. However, without an attempt to measure the effects of your DevOps transformation, there may be no point in attempting it. Not all the pre- and post-DevOps comparisons are going to be particularly enlightening, but enough of them will that the effort will be worthwhile.