Anti-Pattern: Obsession with Metrics

You’ve probably heard a saying: “It is hard to travel in the right direction if you don’t know where you are.” Great saying per se, but not when used as an analogy to operating in an environment of complexity. In fact, as the long-existing scientific discipline of Dynamical Systems suggests, only relatively simple systems can survive such an analogy. In the case of complex organizations, the actual dynamic is not so easy to properly describe.

This has a lot to do with metrics, especially quantitative ones, as the manner in which they are often used has been the bane of the industry’s journey to Agility. Here are some common problems associated with this phenomenon:

  • Over-reliance on proxy metrics leads to local optimization
  • Every metric creates a certain motivation to optimize the outcomes that are being measured, according to the way they are measured. Often times these end up being negative or even destructive in nature.
  • What may be a good metric today, may not be so tomorrow. A new day sets new goals.

Here’s an example of a fundamentally wrong approach to metrics: “We are adopting Agile. What metrics should we use?”

And here’s a different one, a lot more reasonable: “We have certain operating hypotheses in question. Let’s see what kind of measures would allow us to validate or falsify them…”

Here are some suggestions for the correct use of metrics:

  1. Use metrics along the lines of effective, cross-cutting feedback cycles (see Empirical Mindset and Feedback Loop Markers). If you have a good set of such feedback loops, and they are really cutting across organizational levels and even across the organizational boundary itself, you will be able to mitigate the proxy metric problem.
  2. Don’t collect metrics that don’t impact decision-making or don’t provide ultimate visibility into mission-critical organizational processes. In other words, avoid “vanity metrics”.
  3. Don’t let a tool “decide” for you what metrics to collect. Some metrics are very critical, but may not be provided in the tool you’re using. Also, the data for such metrics may be somewhat harder to collect and calculate the values. This should not deter you from measuring what really matters.
  4. Do not allow metrics to be a substitute for Gemba.

Just a reminder that metrics can be a very sensitive topic, but leaving it unattended will continue to block any real effort in adopting Lean and Agile in the organization.