This is part I of Allstacks’ KPI Best Practices Series aimed at helping engineering measure what matters, stay aligned with business objectives, and continuously improve their engineering operations. Check out Part II here

Most people are familiar with the concept of KPIs (Key Performance Indicators). Still, many of my colleagues struggle to nail down precisely what KPIs matter to their engineering team and their organization as a whole. 

There are many moving parts, and it feels like any single indicator is either very measurable and not truly indicative of performance or very well aligned with business goals, but not actually measurable. As engineering leaders, we are responsible for translating our engineering activities into business outcomes; this is a challenge for many, and I hope to offer some practical advice for those who need to overcome it.

The trick is to determine what KPIs that matter to you, your teams, and your organization. This process starts with understanding why your company even has an engineering team, to begin with. 

The Role of the Engineering Team

Company leaders determine core objectives that all departments work towards. As an engineering leader, it’s important to figure out how your KPIs map directly back to those greater goals. 

At a high level, company goals often seek to:

  • Gain profitability
  • Drive growth
  • Deliver customer value
  • Offer great customer experience

Determining KPIs is about taking those significant yet vague measures of success and pinpointing the incremental steps needed for your department to reach them. These are the outputs that need to be stated clearly, mapped out, and measured to gauge success.

Types of Engineering KPIs

Let’s discuss what KPIs should measure and how they relate to overarching organizational goals and outcomes.  

Continuous attainment target [Engineering Activities]

Activity KPIs measure things like:

  • SLAs
  • Uptime
  • Compliance
  • Headcount growth or reduction
  • Bug rates
  • Code metrics
Target KPIs should be continuously checked and monitored. They are most often entirely within engineering’s control but can sometimes slip through the cracks.
 

A great example would be your escaped defect rate. Most people are aware of bugs and their impact on customers but don’t dig deeper to track where those bugs originate. It’s a common practice to differentiate between bugs found in live code and bugs that get caught during internal QA. When you start tracking this type of target, you’re establishing a new KPI. Your internal QA process may be built around reducing or eliminating the rate of escaped defects.

Externally tracked target [Business Outcomes]

Examples of externally tracked target KPIs are things like:

  • NPS score
  • Feature utilization

These items are intended to measure the perception of your application. It is recommended to use externally tracked targets as guiding metrics to validate when the application is running correctly and/or help you do a better job when it’s not. External targets are typically items that are beyond the direct control of your team. Short of asking your engineers to fill out NPS surveys, it’s impossible to ship a feature that will increase NPS by 1 point directly. However, these types of targets should drive other KPIs or goals in your team. If the overwhelming response from the survey is that the app is hard to use, you may set a target to spend 10% of your engineering time on UX improvements. The latter would be an activity that supports the outcome.

Destination target [Shared Business and Engineering Goals] 

These are longer-term goals used to meet business objectives. The objectives can either be internal or external. Marketing could be building out a campaign supporting a release, and sales may need a new feature added to break into a new industry. Goals are typically viewed as the breakpoints on a roadmap. Regardless of the time or energy required, your team is working to build or deliver something. The source of the work may differ depending on the type of goal, but the general tracking process is the same.

Feature delivery is the goal that most of us will recognize, but this includes significant pure-engineering efforts. Tech debt payoffs, upgrading to the next major version of a library, and migrating a production database would all be types of goals that may not show up on the typical product roadmap.

Customer success happens when all departments are tracking outputs toward those larger company goals. But how should engineering set up KPIs to make sure customer success happens? Keep reading...


Want to create and visualize the progress of your engineering KPIs in one place? Let us show you how!  

SCHEDULE A DEMO


Steps for Building Your KPI Tree and Focusing on Outcomes

If your engineering KPIs are defined as “what you need to do and in what timeframe,” meeting KPIs can quickly become a swirl of urgent but ultimately unimportant tasks. That’s why your KPIs should be set up with purpose. Here are the steps you need to take to match outputs with the overall desired outcomes of your organization.

Know the lineage back to the original company outcome or goal. Whatever you’re measuring, from running costs to average downtime to the number of comments per pull request, it needs to track back to the company goal. How are costs affecting profitability? Is rest getting shorter? Is customer experience getting better all the time? KPIs need to be specific and linked to overall goals. 

1.Be clear about how each engineering KPI impacts and supports the overarching company KPIs.

Each engineering KPI should be developed with the same rigor that was used to develop the organizational and company KPIs. For instance, it should be clear that measuring “mean time to recovery (MTTR)” helps meet your SLA target. Also, each KPI should be able to be fully explained by its derivative KPIs. If you’re only tracking MTTR but not SLAs for the discovered defects, your target outcome will be lost.

2.Each KPI should be measurable or fed by measurable KPIs.

While your KPIs may not have well-defined numerical values or concrete actions, they should be at the next level down. For example, if a goal is to communicate information about new features and their delivery date for marketing to plan promotions, the KPIs below should have measurable metrics. The associated trackable activities for engineering would be to document releases in Jira, outline what was included in the Epic, and create a Confluence page with feature screenshots and full descriptions. Your marketing team can validate that it meets their requirements.

3.Targets for individual teams should be within their control.

A KPI is not a KPI if it isn’t a target your team can control. Specificity is key here. Broad goals like reducing hosting costs can’t be pinned to a single task owner. Keeping things broad means no one takes ownership. 

Instead, set narrow, achievable goals. Let’s say your data pipeline team thinks they can make things 30% more efficient, and your database team thinks that caching can save you 20% in disk IOP (Input/Output Operations Per Second) spend, translating to a 10% reduction with 80% confidence. For a solid KPI, set your commitment to a 5% reduction in 3 months, and start delegating the lower targets to the teams and individuals that will implement the changes. This broad goal can now be acted on by specific groups and has defined outcomes. Those teams will more than likely set internal targets to reach and understand why they’re doing it and what the outcome should be.

Final Thoughts on Metrics, KPIs and Targets in Engineering

KPIs are less about perfection and more about continuous improvement. Implementing an outcomes-based KPI strategy can help provide the gravitational field that keeps engineering aligned with the rest of the company instead of being caught up in the vague pressures of daily operations. Stay tuned for more information on how to develop and track engineering KPIs in this ongoing series. Part II will focus on the common pitfalls of creating and monitoring KPIs and how to overcome them.  

What are your thoughts on building outcome-based KPIs? How are you tracking against your desired outcomes? If you need a helping hand in improving your engineering operations, reach out to us to see how we can help or schedule a demo to see our solution in action.  

Additional Resources to Learn More: 

You may also like

Predictive Forecasting is a Game-Changer for Engineering Leaders
Predictive Forecasting is a Game-Changer for Engineering Leaders
23 March, 2021

Managing software development is a balancing act. Ensuring teams are aligned on requirements, hitting deadlines, and del...