by Yohan J. John
But can complex social and biological conditions be boiled down to scores? To GDP-per-capita, or a happiness index, or a body mass index? Social and biological metrics are attempt to quantify things that often seem unquantifiable: the overall health of a country or of a person, the ability of a school to educate its pupils, the quality of a consumer product, and even the aesthetic value of a movie, TV show, or musical album.
I've always been uncomfortable with this process of quantification: on the one hand reducing any phenomenon to a single number seems like a major oversimplification, and on the other, the procedures for generating such numbers are often opaque. How exactly is inflation calculated? Or the cost of living? How do Nielson ratings work? Or the Netflix recommendation system? My discomfort with metrics began to crystallize and expand when I was introduced to a somewhat obscure “law” that should perhaps be more widely known outside of the dismal science that originated it.
Goodhart's Law was originally formulated as follows: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The word 'statistical' probably doesn't excite most people. But if we cut to the essence of what is being said, we find a rule of thumb (it's not a real law of nature) that might have implications well beyond the world of economics:
“When a measure becomes a target, it ceases to be a good measure.”
Let's unpack this idea with a few examples. We can start with academic testing. Written examinations are a time-honored way to assess whether a student has learned something. But when tests scores become the metric by which to judge the performance of schools and teachers, then the connection between the test score and the quality of education often breaks down. This is because teachers “teach to the test“, or even cheat in order to raise scores. By making test scores the target, rather than one among many factors that go into the assessment process, the people involved are incentivized to find the path of least resistance that leads to the highly specific targeted outcome. This drive to find the easiest route to higher test scores is what breaks the correlation between test scores and the more general goal of quality education. You can do well on a test because you have a degree of mastery of the subject, or because you were trained in a mechanical fashion to do the specific test, with no care given to your ability to apply what is learned in other potentially important contexts. [1]
Good examples of Goodhart's Law crop up because of search engine optimization. Early search engines ranked webpages based in part on their content. This was easily manipulated by dodgy websites that contained vast repetitive stretches of text specifically tailored to web-crawling algorithms. Google's PageRank system was a major innovation because it used links between websites, rather than content alone, as the metric for importance. The logic was that useful pages tend to receive lots of incoming links. But this too can be gamed: webmasters can engage in link trading to move their websites higher up in Google's search results. [2]
If there is more than one way to produce the desired metric, then the agents being measured — the schools, the markets, the websites — will game the system whenever possible to minimize their effort and discomfort. The net effect is a kind of social quantum mechanics: the act of observing a complex social system can disrupt it, and also break down the link between the metric and the system. For this breakdown to occur, specific conditions need to be met. First, the metric must be loose enough to confer a degree of play to the system: there must be more than on causal pathway leading from the system to the metric. High test scores can result from a solid understanding of the subject, or from “teaching to the test”, or from outright cheating. Second, the metric must be used for some 'payout' purpose: the system being measured must feel the consequences of the metric. Websites that get more traffic get more ad revenue.
If a metric is connected to a payout, this creates a form of feedback in the system. This feedback in turn creates a selective pressure, or a drive towards competition, if the payout is a limited resource (like money or attention). The metric is a way for the 'measurer' (the school board, a company, a regulator) to declare its desires to the system, and therefore for the system to adapt to these desires in order to get paid. The goal might actually be high quality education, but because this is very difficult to assess in all its complexity, the easily-gamed metric of average test scores steps in as substitute for the elusive ideal. Some sort of 'objective' guide for getting to one's goal must be better than none at all, right? Perhaps, but be careful what you wish for, because it might not be what you were really wishing for.
In HG Wells's short story “The Truth About Pyecraft” (which you can read for free online here if you are worried about spoilers), a man named Pyecraft asks the narrator for a magical Oriental recipe to help him lose weight. But the recipe turns out to make him weightless, leaving his size and shape intact. After consuming the magical elixir, Pyecraft floats up to the ceiling like a helium balloon. The narrator concludes that Pyecraft has committed the “Sin of Euphemism”: asking for a way to lose “weight” when what he really wanted to do was lose fat. One's weight normally has a fair correlation with the amount of fat in one's body, but in the universe Pyecraft inhabits, this relationship can break down. (Pyecraft has to weigh himself down with lead underwear in order to live a normal life from then onwards.) Euphemism is the use of a pleasant or neutral word to talk about something unpleasant. The “euphemistic” nature of Goodhart's Law is that we use an easy-to-measure number to target something hard to get a handle on. Unless we are very lucky indeed, most of the social metrics we come up with will give the system far too much leeway.
If Goodhart's Law were no more than a specific example of the more general tendency of humans to mess with institutions, then perhaps I would not be that interested in it. But I think that the conditions for Goodhart's Law — a multiplicity of causal pathways, feedback, and competition for limited resources — are not simply hallmarks of social networks. You can also discern these dynamics in neural networks. But in order to understand this, we will need to set up an extended metaphor for how the brain works.
The Neural Citadel
Nowadays we routinely encounter descriptions of the brain as a computer, especially in the pop science world. Just like computers, brains accept inputs (sensations from the world) and produce outputs (speech, action, and influence on internal organs). Within the world of neuroscience there is a widespread belief that the computer metaphor becomes unhelpful very quickly, and that new analogies must be sought. So you can also come across conceptions of the brain as a dynamical system, or as a network. One of the purposes of a metaphor is to link things we understand (like computers) with thing we are still stymied by (like brains). Since the educated public has plenty of experience with computers, but at best nebulous conceptions of dynamical systems and networks, it makes sense that the computer metaphor is the most popular one. In fact, outside of a relatively small group of mathematically-minded thinkers, even scientists often feel most comfortable thinking of the brain as a elaborate biological computer. [3]
However, there is another metaphor for the brain that most human beings will be able to relate to. The brain can be thought of as an economy: as a biological social network, in which the manufacturers, marketers, consumers, government officials and investors are neurons. Before going any further, let me declare up front that this analogy has a fundamental flaw. The purpose of metaphor is to understand the unknown — in this case the brain — in terms of the known. But with all due respect to economists and other social scientists, we still don't actually understand socio-economic networks all that well. Not nearly as well as computer scientists understand computers. Nevertheless, we are all embedded in economies and social networks, and therefore have intuitions, suspicions, ideologies, and conspiracy theories about how they work.
Because of its fundamental flaw, the brain-as-economy metaphor isn't really going to make my fellow neuroscientists' jobs any easier, which is why I am writing about it on 3 Quarks Daily rather than in a peer-reviewed academic journal. What the brain-as-economy metaphor does do is allow us to translate neural or mental phenomena into the language of social cooperation and competition, and vice versa. Even though brains and economies seem equally mysterious and unpredictable, perhaps in attempting to bridge the two domains something can be gained in translation. If nothing else, we can expect some amusing raw material for armchair philosophizing about life, the universe, and everything. [4]
So let's paint a picture of the neural economy. Imagine that the brain is a city — the capital of the vast country that is the body. The neural citadel is a fortress; the blood-brain barrier serves as its defensive wall, protecting it from the goings-on in the countryside, and only allowing certain raw materials through its heavily guarded gates — oxygen and nutrients, for the most part. Fuel for the crucial work carried out by the city's residents: the neurons and their helper cells. The citadel needs all this fuel to deal with its main task: the industrial scale transformation of raw data into refined information. The unprocessed data pours into the citadel through the various axonal highways. The trucks carrying the data are dispatched by the nervous system's network of spies and informants. Their job is to inform the citadel of the goings-on outside its walls. The external sense organs — the eyes, ears, nose, tongue and skin — are the body's border patrols, coast guards, observatories, and foreign intelligence agencies. The muscles and internal organs, meanwhile, are monitored by the home ministry's police and bureaucrats, always on the look-out for any domestic turbulence. (The stomach, for instance, is known to be a hotbed of labor unrest.)
The neural citadel enables an information economy — a marketplace of ideas, as it were. Most of this information is manufactured within the brain and internally traded, but some of it — perhaps the most important information — is exported from the brain in the form of executive orders, requests and the occasional plaintive plea from the citadel to the sense organs, muscles, glands and viscera. The purpose of the brain is definitely subject to debate — even within the citadel — but one thing most people can agree on is that it must serve as an effective and just ruler of the body: a government that marries a harmonious domestic policy — unstressed stomach cells, unblackened lung cells, radiant skin cells and resilient muscle cells — with a peaceful and profitable foreign policy. (The country is frustratingly dependent on foreign countries, over which it has limited control, for its energy and construction material.)
The citadel is divided into various neighborhoods, according to the types of information being processed. There are neighborhoods subject to strict zoning requirements that process only one sort of information: visions, sounds, smells, tastes, or textures. Then there are mixed use neighborhoods where different kinds of information are assembled into more complex packages, endlessly remixed and recontextualized. These neighborhoods are not arranged in a strict hierarchy. Allegiances can form and dissolve. Each is trying to do something useful with the information that is fed to it: to use older information to predict future trends, or to stay on the look-out for a particular pattern that might arise in the body, the outside world, or some other part of the citadel. Each neighborhood has an assortment of manufacturing strategies, polling systems, research groups, and experimental start-up incubators. Though they are all working for the welfare of the country, they sometimes compete for the privilege of contributing to governmental policies. These policies seem to be formulated at the centers of planning and coordination in the prefrontal cortex — an ivory tower (or a corporate skyscraper, if you prefer merchant princes to philosopher kings) that has a panoramic view of the citadel. The prefrontal tower then dispatches its decisions to the motor control areas of the citadel, which notify the body of governmental marching orders.
Let us briefly step away from the citadel metaphor to imagine a concrete situation where different parts of the brain might be saying different things, and therefore having to compete for access to the decision-making tower. Imagine you are in the jungle, and you are somewhat hungry, but also aware that there is a man-eating tiger on the loose. Let's say you find a source of food. Should you start eating, and therefore lower your guard slightly? Or should you stay vigilant, and wait to find a source of food in a safer location? It depends on a variety of factors: on metrics that the brain has either inherited or invented. If you stay hungry for too long you'll become weak, and then it will be much harder to deal with the tiger. But if you distract yourself through the process of eating, you might get eaten yourself.
The brain's metrics might be signals like the volume of animal noise in the jungle (which some say is an indicator of a tiger's proximity), or some estimate of the body's energy levels. Whenever you have two mutually exclusive courses of action, only one of them can win out. And the way the brain decides on one action versus another is by weighing those internal metrics. The brain areas and sub-areas that have proven consistently useful become increasingly able to communicate with the decision-making. In neuroscience we explicitly use the term 'credit assignment' to refer to how a neural network can pick out the neurons with most useful piece of information available, and allow them to have a stronger effect on decisions and actions. One of the mechanisms for this is called synaptic learning: the connections (synapses) between neurons can be strengthened or weakened depending on a variety of factors.
So going back to the citadel, we can make guesses about how payments work in the neural economy. Neurons and groups of neurons receive synaptic credit, which grants them more access to places up and down the organizational hierarchy. Good workers gain the trust of their bosses, co-workers and underlings, and can therefore more easily make their opinions heard. They also gain access to the best quality raw material — data arriving from elsewhere. Meanwhile, workers who aren't saying anything particularly useful or relevant to the context get less attention and less access. Success in the neural economy is much like success in a social network: it all boils down to the number and strength of connections. (Just having lots of connections is not the point, however. Quality and relevance matter. These connections must serve the distinctive goals of the neuron and the networks it is part of.) Whether neural connections are strengthened or weakened is a complex matter, and depends on wider forces including the neurochemical weather conditions.
A troubled neighborhood
Celebrity neurochemicals like dopamine, serotonin, and oxytocin can be thought of as indirect indicators of weather conditions — the temperature, the pressure, the humidity, the cloud cover — and they can have all kinds of effects on the neural citizenry, which tries to adapt to the ever-changing conditions. They can change the way a neuron talks, or modify the rules by which its connections with other neurons get regulated. Just as a meteorologist's instruments serve as a window on what the weather is and how it will develop in the near future, neurochemicals and hormones are used in the brain as metrics for signaling and assessing various situations in the brain and body.
Let's focus on one particular neural weathervane: dopamine. I've mentioned in a previous 3QD essay that dopamine has been mischaracterized as 'the pleasure chemical'. Dopamine doesn't just get released in response to pleasurable activities: it can also be released by negative experiences or by situations that are unpredictable. But dopamine does seem to have a powerful ability to change the decisions made by the brain as a whole. Drugs of abuse often affect the dopamine system. So researchers initially thought that drugs like cocaine or methamphetamine were just so pleasurable that the addicts simply couldn't resist them. The story turned out to be much more complicated.
Dopamine is not itself the chemical that engenders the subjective experience of pleasure. There may not be any single chemical that operates in such a simplistic way. What it might be is a kind of metric that the brain uses to represent some (still debated) class of potentially useful information. Dopamine has a strong ability to change the synapses between neurons. The chemical and informational highways that lead up to the dopamine cell neighborhood seem to allow dopamine cells to notify the rest of the brain of situations that are potentially worth paying attention to, and perhaps remembering for later.
So if you experience something pleasurable, like a taste of honey, you will probably want to know how you can repeat this experience. In order to do this you need to find out which pieces of information preceded the taste of honey. Perhaps it was the sounds of bees. Or the sight of a particular tree. A burst of dopamine seems to help form a link between certain kinds of information and their consequences. Once these links are made, the next time your circumstances resemble the happy honey discovery episode, you can begin to respond, perhaps ensuring that you get more honey than last time, or paying closer attention, so you can discover the precise source of the honey. More proactively, you may also be able to re-create those circumstances — by getting into beekeeping, for instance.
Remember that dopamine is not a direct mapping from honey to the brain. It is embedded in a complex causal web woven by the blind spider of natural selection. Evolutionary forces are under no compulsion to construct signaling pathways that convey one and only one thing. Natural selection is a satisficing process, so it creates situations that are good enough, but not necessarily the most efficient. Dopamine signaling was evidently good enough as a signal of potentially useful information for evolutionary forces to allow it to change synapses and bias our behavior. The raw data generated when honey arrives at the tongue and later the stomach travels along multiple roads that lead into the brain. One of those roads leads to the dopamine neighborhood. But the full subjective experience does seem to involve the consequences of all those other data streams.
When a drug starts to make trouble on the streets of the dopamine zone, it is very likely to be doing so through causal pathways that are distinct from the ones linking honey with dopamine. Because there is more than one way to get a dopamine cell to sound the alarm, dopamine's correlation with usefulness (or pleasure, or unpredictability) can easily break down. Drugs may have a pleasurable component, but addicts often talk about how the pleasure can fade away at some point, leaving only the compulsion to feed the addiction. It is at this point that we can import Goodhart's Law into our picture of the neural economy. Perhaps dopamine is a metric that the brain can use to assign credit to useful information, giving the neural sources of that information more access to the halls of power. But if dopamine stimulation becomes a target, it ceases to have a strong relationship with usefulness. It nevertheless retains its more mechanical ability to restructure neural networks through synaptic change. Direct dopamine stimulation might cause the brain's motivation system to skip the 'liking' stage and go straight to the 'wanting' stage.
For drug addicts, it seems that the feedback and competition in their neural networks has created a Goodhart-like scenario, in which behavior becomes focused on the self-reinforcing chemical stimulation of dopamine cells, rather than on a more holistic stewardship of brain, body and society. Circumstances that prevent such a holistic approach may actually plant the seeds of addiction. The famous Rat Park experiments suggest that rats will not become addicted to drugs of abuse — even if they do consume them occasionally — if they are placed in a stimulating environment with sufficient food and opportunity for social interaction. Social isolation may contribute to the overcast neurochemical skies that are most conducive to drug abuse. It may create a situation where chemical stimulation is the only signal that is loud and clear, with nothing healthy to compete with it for access to the decision-making centers.
The head and the heart
Addiction to drugs is just an extreme version of a 'neural Goodhart' situation that can happen to anyone. One of the things the human brain allows for is rational theorizing about the state of the world. Whatever else these theories are for, at the very least they should occasionally help a person navigate in the world. But if rationalizing per se becomes the focus of the theorizing, rather than usefulness or agreement with reality, then we might end up delusional, or paranoid, or in thrall to a conspiracy theory. In such situations apparent internal consistency becomes the only metric for the worth of an idea. Rationalizing can become less useful as a guide to behavior if it becomes the only target of behavior. A group of rogue neurons that specialize in rationalizing might hijack one of the brain's credit assignment centers, giving it the ability to pay other groups to adopt its dubious standards.
We might come up with more examples if we replace 'rationalizing' with other neurochemical signals, ideas, theories, or patterns of behavior. Almost anything can become a 'totalitarian' obsession and lead to the detriment of other modes of thought and forms of life. We can propose a neural Goodhart's Law, which we might as well call Good-head's Law: a neural or conceptual metric might cease to correlate with mental and bodily well-being if it becomes the dominant prism through which experience is viewed.
In allocating power based solely on some reductive measure of goodness or truth, a totalitarian government destroys diversity and create stereotypical internal yes-men. We might go on to define a fundamentalist as a would-be totalitarian: someone who takes very seriously some narrow and rigidly defined set of metrics of personal or social health, and attempts to remake the self or society in slavish accordance with this idealized image. On the personal level, the quantified self movement might confront such issues head on. Metrics for personal health — cholesterol levels, body mass indices, blood sugar levels — capture very broad and often poorly understood statistical correlations. Imagine a person whose goal is to be healthy, and rigidly defines health in terms of these numbers. This person might conceivably find specific drugs or exercise regimens that bring these numbers into their allegedly optimal levels. But there are always mental and social consequences to any change in behavior. If the drug and exercise regimen leads to anxiety or social isolation, it may have various adverse effects on health.
A person who follows a Mediterranean diet might have health indices in the right places, but the good health that accompanies these numbers may also depend on specific causal chains, such as the psychological, social and environmental conditions that frame the person's life. A person might be able to manipulate his or her health indices into the supposedly healthy zone, but in a more forced or anxiety-inducing way that could be counterproductive. Along with Good-head's Law we can propose a Good-heart's Law: health metrics might cease to correlate with good health if they become lifestyle goals.
I hope it's clear that almost everything I've written here is wildly speculative. Not all metrics are going to break down when they become targets — whether they're implicit metrics used by the brain or explicit metrics invented by governments and private companies. Some of them have a very close relationship with the things they are measuring. If you are interested in reducing the number of atoms in your body, then your mass in kilograms is an appropriate metric. If you are interested in controlling how hot a room is, then temperature is the ideal metric.
The Goodhart scenario arises in very specific situations. First, the metric must be trying to capture something that is hard to define — there must be some gap between what the measurer is actually interested in, and what is directly measured. Second, there must be more than one causal pathway that can give you the targeted value for the metric. In other words, there must be no necessary or one-to-one relationship between the metric and the phenomenon being studied. (This is almost always the case in complex networks.) Third, the metric must be used to give feedback in a competitive situation: to allocate limited resources —money, credit, energy, attention — in a neural or social network. [5]
The letter of the law and the spirit of the law
Goodhart's Law and its (admittedly speculative) biological corollaries serve as a warning for us as we concede more and more power to the algorithmic management of our lives. People working in government, in the business world, and in academic research are increasingly confronting problems that have to do with network level phenomena: at the highest levels we are grappling with complex social, economic and ecological networks, and at the lowest level we are dealing with equally complex networks of neurons and proteins and genes. New metrics will increasingly be deployed to make sense of the ever expanding pool of Big Data. Already on Wall Street there are anonymous trading algorithms doing odd things for unclear reasons. Perhaps in the near future governments and large companies will scan social networks (or the bio-sensors that some people will voluntarily place on themselves) looking for signs of incipient public outrage. Their metrics and algorithms will be marketed to us as 'scientific' and 'objective' approaches to concepts that used to be described in the wishy-washy terms used by artists, humanities professors, and social workers.
Goodhart's Law tells us that if we are not careful about what we target in a complex network, we might end up with something that is not at all what we actually wanted. Perhaps it is simply an amusing quirk of networks — something that future researchers will do away with using better metrics. On the other hand, Goodhart's Law could be a warning for how technocrats might inadvertently lead us into an Orwellian world by convincing society to dispense with those old wishy-washy terms for what we actually want. “Liberty, equality, fraternity, peace… what do they mean? Give me a number!” If everyone is forced to adopt the simultaneously oversimplified and opaque language of metrics, then we may gradually lose the ability to articulate what is missing.
Perhaps the lesson we can learn from Goodhart's Law is that there must be a difference between the spirit of the law and the letter of the law, and between the spirit of measurement and a specific metric [6]. In a democratic culture, a specific law is a concrete expression of some goal for the self and society. In a scientific culture a metric is a concrete quantification of some multifaceted state of a self or a society. Behind a specific metric there is often a vague awareness of a more complex reality: after all, this awareness is usually what motivates the creation of the metric in the first place. I think it is important not to abandon this awareness, however vague it is, once a metric arises to dispel it. The post-Enlightenment legal systems that structure our lives also arose from a vague awareness: a spirit or sensibility that drives us to become fitter, happier, more productive, and more perfect personalities and polities. We might call these forms of awareness the scientific spirit and the humanistic spirit. They both seem to arise from a common source: the impulse to transcend the material, social, political, and conceptual boundaries of the present moment. This drive towards transcendental novelty also gives rise to art, literature, poetry and music — those wellsprings that seem to provide us with our most resonant expressions of what a self or a society ought to look like. Perhaps our allegiance must be with this mysterious spirit that animates our arts, our laws, and our measurements, rather than with any particular — and therefore limited — manifestation of it.
______
Notes
[1] In India, the prestige of the Indian Institutes of Technology has lead vast numbers of students to prepare for the entrance exam in intensive coaching camps. These coaching centers train students to get into the IITs, but do not prepare them at all for the qualitatively different (and more difficult) academic challenge of the coursework that awaits them. Many of them quickly go from elation at having made the grade to depression and sometimes even suicide.
[2] Several examples that illustrate Goodhart's Law, including an amusing one from the Soviet Union, can be found here.
[3] For an excellent historical discussion of the various metaphors that have been used to describe the mind and brain, see John G. Daugman's essay Brain Metaphor and Brain Theory [pdf].
[4] The trick of seeing the human society as a macrocosmic body is used to great effect in a section of the song 'Maya' by the Incredible String Band.
The great man, the great man, historians his memory
Artists his senses, thinkers his brain
Labourers his growth
Explorers his limbs
And soldiers his death each second
And mystics his rebirth each second
Businessmen his nervous system
No-hustle men his stomach
Astrologers his balance
Lovers his loins
His skin it is all patchy
But soon will reach one glowing hue
God is his soul
Infinity his goal
The mystery his source
And civilization he leaves behind
Opinions are his fingernails
[5] Economic indicators often check off all three boxes, which is presumably why Goodhart's Law turned up in economics.
[6] The Indian mythologist Devdutt Pattanaik makes an interesting point about the spirit of the law and the letter of the law in a television presentation on the differences between the two great Indian epics, the Ramayana and the Mahabharata. In his opinion the Ramayana, which takes place in an innocent age, depicts a society that still remembers the spirit of the law. The Mahabharata, by contrast, describes events at a later and more corrupt age, when society has forgotten the spirit of the law, and clings to the letter of the law.
[Image from Wikipedia: Seventeenth-century plan of the fortified city of Casale Monferrato. The citadel is the star-shaped structure on the left.]