by Yohan J. John
This is the era of metrics: it seems that if we are to hack a path through the information jungle of the 21st century, we must be armed with an arsenal of scores, quantities, indices, factors, grades, and ratings. Our corporate and governmental overlords seem most comfortable parlaying in the seemingly objective language of numbers.
But can complex social and biological conditions be boiled down to scores? To GDP-per-capita, or a happiness index, or a body mass index? Social and biological metrics are attempt to quantify things that often seem unquantifiable: the overall health of a country or of a person, the ability of a school to educate its pupils, the quality of a consumer product, and even the aesthetic value of a movie, TV show, or musical album.
I've always been uncomfortable with this process of quantification: on the one hand reducing any phenomenon to a single number seems like a major oversimplification, and on the other, the procedures for generating such numbers are often opaque. How exactly is inflation calculated? Or the cost of living? How do Nielson ratings work? Or the Netflix recommendation system? My discomfort with metrics began to crystallize and expand when I was introduced to a somewhat obscure “law” that should perhaps be more widely known outside of the dismal science that originated it.
Goodhart's Law was originally formulated as follows: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The word 'statistical' probably doesn't excite most people. But if we cut to the essence of what is being said, we find a rule of thumb (it's not a real law of nature) that might have implications well beyond the world of economics:
“When a measure becomes a target, it ceases to be a good measure.”
Let's unpack this idea with a few examples. We can start with academic testing. Written examinations are a time-honored way to assess whether a student has learned something. But when tests scores become the metric by which to judge the performance of schools and teachers, then the connection between the test score and the quality of education often breaks down. This is because teachers “teach to the test“, or even cheat in order to raise scores. By making test scores the target, rather than one among many factors that go into the assessment process, the people involved are incentivized to find the path of least resistance that leads to the highly specific targeted outcome. This drive to find the easiest route to higher test scores is what breaks the correlation between test scores and the more general goal of quality education. You can do well on a test because you have a degree of mastery of the subject, or because you were trained in a mechanical fashion to do the specific test, with no care given to your ability to apply what is learned in other potentially important contexts. [1]
