The danger of surrogate metrics…

May 10, 2007 at 6:11 pm

I was reading a “Joel” post (I like Joel’s writing, but I wish that he allowed comments) entitled “The Econ 101 Management Method“, which I find myself mostly in agreement.


I’d like to expand a bit in the area of metrics – specifically what I call “surrogate metrics”.


Most software development teams are associated with what business guys call a “P&L center”, which in simple terms means that it’s part of the business that will either make or lose money. The measure of whether the group is making or losing money is an example of a metric, and it’s a good metric, in the sense that it measures exactly what it says it’s going to measure.


How important that particularly metric is to a company and what other metrics are also important is a different subject. As is the siren song of metrics in general.


The subject of this post is metrics don’t measure the thing that you want to measure, but are *believed to* correlate with the thing that you want to measure.


Say, for example, that you’re a software company, and you’ve heard through the grapevine that customers are unhappy with the service that they are getting through your support forums. A little research shows that some people aren’t getting prompt answers.


So, you spend some time writing a reporting layer on top of the forum software that tracks the time between when a post first shows up and it has a response from somebody in your company. You run it on some historical data, and see that the response time averages out at 36 hours, which makes you unhappy. You work with your group, tell them that they need to do better, and over the next month, the average response time goes down to 12 hours, and you’re happy that you’ve solved the problem.


Did you do a good job? Is the problem fixed? Discuss…


The answer to my questions is a rousing “who knows?” It’s possible that the problem is fixed, and it’s also possible that it’s still as bad as before. That’s because “response time” is a surrogate measure of the thing that you really care about, customer satisfaction.  You chose it because it was a *easily measurable*.


Which I guess does lead me towards discussing the siren song of metrics in general. There’s a real bias in some business cultures towards measuring a lot of metrics. As Joel points out, this leads to people gaming the system, which is an obvious issue. But even if people don’t game the system, surrogate metrics can, at best, suggest when something is bad, but they can never tell you when something is good enough.


Some people would argue that you should still collect the metrics you can, but I think you just shouldn’t bother with surrogate measures. Measure the things that you truly care about, and don’t mess up your culture and reward system by measuring the surrogates. And if you can’t measure the thing you really care about objectively, if it’s too hard or too expensive, you’ll just have to deal with the the uncertainty.


In my example, if you care about customer satisfaction in your support forums, then you need to ask customers whether their support experience was acceptable. There are lots of ways of doing this, and you can often use the same process to allow customers who had a bad experience to escalate it.


So, what is your favorite real measure and surrogate measure that you’ve seen?