This is part I of Allstacks’ KPI Best Practices Series aimed at helping engineering teams measure what matters, stay aligned with business objectives, and continuously improve their engineering operations. Check out Part II here

Most people are familiar with KPIs (Key Performance Indicators). Yet many of my colleagues struggle to nail down which KPIs matter to their engineering team and their organization as a whole. This blog breaks down how to develop outcome-based engineering KPIs most effectively and focus on the outcomes after the fact. 

The KPI process involves many moving parts. It can sometimes feel like single indicators are either measurable and not truly indicative of performance or well aligned with business goals but not measurable.   As engineering leaders, we strive to translate our engineering activities into business outcomes—a challenge for many that I hope to offer practical advice in overcoming. 

The trick is to determine which KPIs matter to you, your teams, and your organization. This process starts with understanding why your company even has an engineering team to begin with. 

Understanding Engineering Team Roles

Company leaders determine core objectives that all departments work toward. As an engineering leader, it’s important to figure out how your KPIs map directly back to those greater goals. 

At a high level, company goals seek to:

  • Gain profitability
  • Drive growth
  • Deliver customer value
  • Offer great customer experience

Determining KPIs is about taking those significant, if a little vague, measures of success and pinpointing the steps to reach them. These are the outputs that need to be stated clearly, mapped out, and measured.

Types of Engineering KPIs

Let’s discuss what KPIs should measure and how they relate to overarching organizational goals and outcomes.  

1. Continuous attainment targets 

Engineering activity KPIs measure things like:

  • SLAs
  • Uptime
  • Compliance
  • Headcount growth or reduction
  • Bug rates
  • Code metrics

You should continuously check and monitor Target KPIs. These KPIs usually fall entirely within engineering’s control but can sometimes slip through the cracks.

A great example would be your escaped defect rate. Most people know about bugs and their impact on customers, but don’t dig deeper to track where those bugs originate. It’s a common practice to differentiate between bugs found in live code and bugs that get caught during internal QA. When you start tracking this type of target, you establish a new KPI. Your internal QA process may be built around reducing or eliminating the rate of escaped defects.

2. Externally tracked targets 

Examples of externally tracked target KPIs are traditionally business outcomes that include:

  • NPS score
  • Feature utilization

These items measure your application’s perception. Most recommend using externally tracked targets as guiding metrics to validate when the application is running correctly and helping you do a better job when it’s not. 

External targets are typically beyond the direct control of your team. Short of asking your engineers to fill out NPS surveys, it’s impossible to ship a feature that will increase NPS by 1 point directly. However, these types of targets should drive other KPIs or goals in your team. If survey participants overwhelmingly respond that the app is hard to use, then you may set a target to spend 10% of your engineering time on UX improvements. This would be an activity that supports the outcome. 

3. Destination targets 

These longer-term shared business and engineering goals aim to meet business objectives internally or externally. For example, marketing could build a campaign supporting a release, and sales may need a new feature added to break into a new industry. Engineers typically view these goals as the breakpoints in the roadmap. 

Regardless of the time or energy required, your team works to build or deliver something. The source of the work may differ depending on the type of goal, but the general tracking process remains consistent.

Most of us will recognize feature delivery, but this includes significant pure-engineering efforts. Goals like paying off tech debts, upgrading to the next major version of a library, and migrating a production database may not show up on the typical product roadmap. 

Customer success happens when all departments track outputs toward those larger company goals. But how should engineering set up KPIs to make sure customer success happens? Keep reading...


Want to create and visualize the progress of your engineering KPIs in one place? Let us show you how!  

SCHEDULE A DEMO


Best Practices for Building Your KPI Tree and Focusing on Outcomes

If you define your engineering KPIs as “what you need to do and when,” then meeting KPIs can quickly become a swirl of urgent but unimportant tasks. To avoid the swirl, set your KPIs with a purpose. Follow these steps to match outputs with the overall desired outcomes of your organization: 

Know the lineage, all the way back to the original company outcome or goal.

Regardless of what you measure, from running costs to average downtime to the number of comments per pull request, it needs to track back to the company goal. How do costs affect profitability? Does rest get shorter? Does customer experience improve all the time? KPIs need to be specific and linked to overall goals. 

Be clear about how each engineering KPI impacts and supports the overarching company KPIs.

You should develop each engineering KPI with the same rigor as the organizational and company KPIs. For instance, it should be clear that measuring “mean time to recovery (MTTR)” helps meet your SLA target. Further, you should be able to fully explain each KPI by its derivative KPIs. If you only track MTTR but not SLAs for the discovered defects, your target outcome will be lost. 

Ensure each KPI is measurable, or fed by measurable KPIs.

While your KPIs may not have well-defined numerical values or concrete actions, they should be at the next level down. For example, if a goal is to communicate information about new features and their delivery date for marketing to plan promotions, the KPIs below should have measurable metrics. The associated trackable activities for engineering would be to document releases in Jira, outline what was included in the Epic, and create a Confluence page with feature screenshots and full descriptions. Your marketing team can then validate that it meets their requirements.

Make sure teams can control their targets.

A KPI is not a KPI if your team can’t control it. Keep specificity in mind here. You can’t pin broad goals like reducing hosting costs to a single task user. Keeping things general means no one takes ownership. 

Instead, set narrow, achievable goals. Let’s say your data pipeline team thinks they can make things 30% more efficient, and your database team thinks that caching can save you 20% in disk IOP (Input/Output Operations Per Second) spend, translating to a 10% reduction with 80% confidence. For a solid KPI, set your commitment to a 5% reduction in 3 months, and start delegating the lower targets to the teams and individuals that implement the changes. 

You can now act on this broad goal within specific groups and produce defined outcomes. As a result, your teams will likely set internal targets to reach while understanding why and what outcome to expect.

Final Thoughts on Metrics, KPIs and Targets in Engineering

KPIs are less about perfection and more about continuous improvement. Implementing an outcomes-based KPI strategy can help provide the gravitational field that keeps engineering aligned with the rest of the company, instead of getting caught up in the vague pressures of daily operations. 

What are your thoughts on building outcome-based KPIs? How are you tracking against your desired outcomes? If you need a helping hand in improving your engineering operations, reach out to us to see how we can help or schedule a demo to see our solution in action.  

Our industry is ever-growing, and there’s never a shortage of thought-leaders and forward-thinking entrepreneurs to follow and draw inspiration from. Check out our list of 5 top technology leaders to follow in 2022.

Additional Resources to Learn More: 

 

Submit a comment

You may also like

Jamie Howard joins Allstacks as Vice President of Engineering
Jamie Howard joins Allstacks as Vice President of Engineering
11 January, 2022

Allstacks has experienced tremendous growth over the last 12+ months. We’re excited to announce that Jamie Howard is joi...

Solving 3 common software engineering challenges
Solving 3 common software engineering challenges
6 May, 2022

Allstacks “turns the lights on” for software engineering teams operating in the dark to deliver customer value and impro...