DevOps Success: What to Measure and Why
In the recent State of Agile report, 90% of respondents said their business prioritizes DevOps transformation. Not surprising, since DevOps promises faster software development and release cycles, higher quality software, and faster times to resolve issues. What is surprising is that many teams struggle to show a significant impact on the business. In this article, we’ll consider what the business wants to understand about how DevOps impacts its goals. We’ll also look at common DevOps performance metrics you can use to measure, track, and demonstrate the business value of your team’s hard work.
What the Business Wants to Know About DevOps Impact
DevOps metrics typically show activity; what they don’t show is business impact. For example, if you decrease lead time to code by 2.25 hours over six months, that might be a significant achievement for your team. You might have a small celebration to recognize the achievement. However, when you report your success back to the business, your stakeholders might not be very excited. What does that 2.25 hours mean for them? How does it help them achieve their own aggressive goals? However, if you can show that, by reducing the lead time of code changes, you could deploy new revenue-generating features to your site in less than one day, that will spark interest from your business stakeholders. The key is to focus on the business value, not the activity.
To ensure that what you measure is meaningful to your business, define and prioritize a small set of goals as they relate to your business. Since every business is unique, we recommend that you work closely with your business stakeholders to ensure everyone agrees on the priorities. Once you’ve defined the business goals, identify the things your DevOps team can control and measure to affect those business goals.
Let’s look at a few more examples of how you can link a business goal to a DevOps metric you can control and track:
| Business goal | Potential DevOps measurements |
| Increase customer satisfaction | Attain SLA on shopping cart and order processing |
| Maximize shopping cart conversions on highly competitive e-commerce site | Increase deployment frequency Attain SLA on shopping cart processes |
| Reduce costs | Decrease mean time to restore/recover Deployment success rate Achieve high availability |
Narrowing your focus to a handful of critical goals and measurements has several advantages. It will help you quickly find the signal in all the noise. DevOps technology and processes generate volumes of data. You need to use and then make smart, informed decisions from all the data you gather or you risk slowing things down. Focusing on just a few key metrics will also help your efforts to strengthen a culture of continuous improvement. Quantitative and qualitative metrics that are closely linked to business impact can fuel your efforts to build a team with a pervasive desire to find fresh opportunities to test and learn.
Interestingly, many organizations struggle with this gap between IT and business. To bridge the gap, some organizations have grown DevOps further, aiming to remove the divide between IT and the business. Sometimes referred to as BizDevOps, these variations of DevOps are all about aligning the software development and delivery process more closely with business goals. Whatever approach you take, though, you must find a way to align with the business or your measurements may fail to impress.
What DevOps Measures
Once you’ve defined the business goals, you need to decide what outcomes or metrics your team can influence and measure to impact the goals. DevOps metrics typically measure either throughput or quality. You might think you achieve speed only by sacrificing stability, or stability by sacrificing speed, we believe speed and stability each enable and enhance the other. According to the State of DevOps 2019 report, high-performing DevOps teams outperform low-performers on all measures of software delivery effectiveness, including speed and stability. This research shows that it’s a both/and situation. When you focus on speed, you also achieve greater stability; when you focus on stability, you achieve greater speed.
Speed: Measuring How Fast You Deliver Value to the Business
Speed metrics show how effectively a DevOps team delivers value to the business. If your business is in financial services or retail or other highly competitive industries, speed will probably be a priority. Speed can also be useful for businesses that seek cost reduction, because speed can show greater process efficiencies.
The most common metrics for speed include:
• Deployment frequency: One of two key measures for your team’s throughput, this measures how frequently your team deploys code to production or another significant point in the continuous delivery (CD) pipeline. According to the State of DevOps report, high-performing DevOps teams typically achieve higher deployment frequencies, as much as multiple times per hour (and sometimes as much as multiple times per minute).
This DevOps metric is represented as a percentage of total opportunities to deploy. For example, if there are 10 deployment opportunities in a week, and code deployed only three of those times, the deployment frequency is 30%.
Once you’re comfortable tracking deployments, you might track deployments to production and non-production environments separately. For example, it can be important to deploy frequently and early to QA so that there’s enough time to test.
In addition, you might track the size of your deployments: how many stories, feature requests, bug fixes, or story points are deployed in each cycle or over a given period. For most DevOps teams, the goal will be smaller and more frequent deployments. Smaller deployments are easier and faster to test and release.
• Lead time for changes: The second of two key throughput measures, this metric tells you how much time it takes to move from code commit to successful code deploy. High-performing teams typically achieve a lead time for change of one hour or less.
• Unplanned work: When you track the time spent pausing the CD flow to tackle defects, build breaks, and other unplanned downtime, you can pinpoint problems you may need to address earlier in a workflow. Unplanned work is expressed as a percentage of overall time; most teams strive to keep this number under 25%.
Stability: Measuring the Quality of Value Delivery to the Business
Stability metrics provide an important counterpoint to speed, highlighting the availability and quality of the product delivered. Businesses in industries with intensive regulatory or compliance requirements, such as healthcare, may prioritize stability over speed. Stability can also provide insights into the overall speed and efficiency of a CD pipeline. For businesses in industries that are both highly competitive and highly regulated, such as banks, stability may be given equal priority with speed.
Common metrics for stability include:
• Mean time to restore (MTTR): Measures the mean time it takes your team to restore disrupted service. This is typically expressed as a mean time over some number of episodes. DevOps teams typically see this number decrease over time, especially in the later stages of DevOps transformation. Exceptionally high-performing teams have an MTTR or less than one hour, while less efficient teams or teams working with legacy environments or extensive compliance requirements may have an MTTR between 1 week and 1 month.
• Change success/failure rate: Measures how often deployment failures occur in production that require immediate action. High-performing teams achieve a rate between 0-15%, while low performers have a rate between 40-60%.
• Total number production incidents: Measures the number of incidents that cause disruption during a specified time. This might seem obvious, yet it’s important to not overlook it as it can help you identify areas in the CD pipeline that might need more focus. A high-performing DevOps team should expect this number to track down as the DevOps practice matures.
Proving Continuous Improvement, in Service of the Business
Whether you’re just starting a DevOps practice or fine-tuning your approach, proving return on investment (ROI) to the business can be challenging. To make sure you can show the business impact of your DevOps work, always identify the business goals first, then specify the measures your team can control to impact those goals. Over time, you’ll establish a track record of delivering measurable value to your business. You’ll also prove your team’s commitment to continuous improvement for the business.
