Engineering Metrics Is The Elephant In The Room

There’s a story, probably apocryphal, about a royal leader who presented an elephant as a gift to those who had fallen out of favor. The unlucky recipients were left with a large and expensive problem. They knew they had a valuable animal, and they knew their health depended on the elephant’s health. However, the cost of feeding and caring for the creature exceeded the value they could realize. Ultimately, they went bankrupt, surrounded by piles of… err, tech debt.

This story came to mind when I talked with a client recently. Their complaint was that “engineering” and “the business” didn’t understand each other. Engineering was the elephant: bulky, expensive to keep, hard to move, and incomprehensible. Business was the poor beneficiary, not understanding how to create value from the incredible resource — they weren’t even sure what they had.

Framing the problem this way is a recipe for failure. Engineering is part of the business, often a large part. Engineering leaders must recognize they need the rest of the business to be healthy, or the elephant will starve. Leaders of the rest of the business must give clear direction to the elephant so it’s doing useful work for the organization, not just wandering off on its own.

GenAI: Was That An Earthquake?

Developer productivity is on everyone’s mind these days. We’ve all heard that gen AI promises to increase developer productivity by 40% or more. There’s a lot of hype, and leaders need to cut through the hype to find reality.

Business leaders both inside and out of engineering have been coming to me with the same question these days: “How can we determine if our developers are more productive with genAI tools?” My response usually doesn’t go over too well: “Take whatever you’re using to measure productivity now, add genAI, and see if those measures go up.”

The truth is that measuring developer productivity is hard. Back in 2003, Martin Fowler gave up, saying, “we have no way of reasonably measuring productivity.” Metrics like lines of code have always been meaningless, and they’re even more meaningless when you can add a prompt like, “make this twice as long.”

A section of a chat session with a gen AI tool. The AI has responded to a prompt with "This script reads a CSV file named data.csv and prints the highest value in each row. Make sure your CSV file is in the same directory as the script or provide the full path to the file." The user has then entered another prompt, "Make the program twice as long." The gen AI returns, "Sure, I can add more functionality to the program to make it longer. Let's include additional features like handling potential errors, adding comments for better understanding, and allowing the user to specify the CSV file name through input. Here's the updated program:". That's followed by a truncated code snippet that reads, "import csv # Function to read a CSV file and output the highest value in each row"

Business Is What Matters

In many cases, metrics are a form of vanity. “Our team is DORA elite” doesn’t mean much if your customer doesn’t want what you deliver, or if you’ve got overwhelming turnover costs due to developer burnout. Just the act of measurement will change what happens at your organization, so a light — and balanced — touch is needed. Save the time and motion studies for processes that get repeated and automated.

The elephant knows how to lift the log and enjoys doing it. Team up with the elephant so you both succeed. Create alignment — make sure the elephant understands where the log needs to be — clear out the obstacles, and let the elephant figure out how to get it there. To learn how to do that, Forrester clients can connect with me or read my report, “Your Focus On Developer Productivity Is Killing You.”

source

Leave a Comment

Your email address will not be published. Required fields are marked *