Wednesday, May 5, 2010

Goodhart's Law and Numerosity

They say "You Can't Manage What You Don't Measure". This isn't quite right, however. You can measure lots of things you can't manage. Rainfall, for example. Software development is like that. You can measure stuff that you can't actually control.

The original Deming quotes are more subtle: there are things which you cannot measure, but they are still important. And there are visible measures that are an attractive nuisance.

What gets lost is that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes" This is Goodhart's Law.

As soon as you try to measure "programmer productivity" or "quality" or similar things, folks will find ways to tweak the numbers without actually improving anything.

Metrics are troubling things.