by Jochen Szangolies
The humble bumblebee is a mainstay character of a certain kind of motivational tale. One widely shared version of the story goes as follows:
The mathematical relationship between the wings and the weight of the bumblebee shows that flight is impossible for it. But the bumblebee does not know this, so it flies.
It’s a good line. It has a clear, uplifting message: no matter what other people—even supposed authorities—say, you can achieve your goals. What matters is what you believe (for if the bumblebee had believed the scientific verdict, it would surely have remained safely aground). It’s also a cautionary tale against scientific hubris: all its ambitions to unravel the mysteries of the cosmos thwarted by a tiny garden insect.
But—and I’m sorry to have to be the party pooper here—the problem is that it’s just not true. There is no great mystery about the bumblebee’s flight. But the nature of the error being made here is instructive (and more interesting than the myth).
In 1934, French entomologist Antoine Magnan, together with his assistant André Sainte-Lague, performed a short calculation for his book, Le Vol des Insectes (“The Flight of Insects”), which concluded that the lift produced by the bumblebee’s comparatively small wings could not suffice to keep its stocky body airborne—it would have to be about ten times lighter to work.
So what went wrong? Did they just bungle the calculation? No—their math checks out. The problem lies in assuming a particular model of flight, suitable for airplanes, but ill fitted to the case at hand. The calculation assumed, first of all, that the wings are smooth, and secondly, that they are held fixedly. But the bumblebee does not keep its wings fixed while flapping; rather, their tips describe a figure-8 shape, flapping back and forth more than up and down. Furthermore, it beats its wings extremely rapidly for an insect of its size. These facts combine to create an air vortex, a tiny whirlwind, at the wing’s leading edge, which has a very low internal pressure. This is what pulls the insect into the air.
The interesting lesson here is that one and the same capacity—flight—can be achieved in more than one way, and eliding the distinction between the mechanisms can lead to significant errors (and great motivational quotes). Both bumblebees and airplanes fly, but the flight of the bumblebee is not the same as that of the airplane, and substituting one for the other can lead to serious mistakes. Also, bumblebees are awesome. (In an unrelated testament to their awesomeness, hibernating bumblebee queens can breathe underwater. Plus, their scientific genus name is Bombus!)
Zooming out a little, two systems can show (what seems to be) the same behavior, yet transferring lessons learned from analyzing that behavior in one system to the other is fraught—they might not achieve their behavior in the same way. Obviously, I’m talking about Large Language Models.
Large Language Games
The philosopher Ludwig Wittgenstein is famous for perhaps the most significant philosophical volte-face: believing himself originally to have solved (or dissolved) all problems of philosophy well defined enough to admit a solution with the 1922 publication of the Tractatus Logico-Philosophicus, he would later renege on many of its key points in the (posthumously published) Philosophical Investigations. So severe is the departure that many modern accounts essentially treat the Wittgenstein of the Tractatus, WittgensteinT, as a distinct philosopher from that of the Philosophical Investigations, WittgensteinPI. (In the latter work, Wittgenstein himself refers to ‘the author of the Tractatus’.)
One key aspect of WittgensteinPI’s break with his earlier incarnation concerns the philosophy of language. The Tractatus advances the so-called picture theory of language. In its strict ordering of seven propositions, together with sub-propositions further explaining them, Proposition 3 is ‘the logical picture of the facts is the thought’, which then leads, in Proposition 3.2, to ‘in propositions thoughts can be so expressed that to the objects of the thoughts correspond the elements of the propositional sign’. Together, thus, the elements of a proposition correspond to pictures of the facts—there is a ‘pictorial’ correspondence between what we say and what is the case, and what we say is true if what is claimed to be the case actually is. Language depicts the world.
Against this, in the Philosophical Investigations, the notion of a ‘language game’ is developed: language works as a back-and-forth between different actors, and to the extent that the game is played successfully—utterances yield follow-up utterances or actions—the words used in it have meaning. Importantly, these meanings are not anchored by reference to objects in the world: they are relevant only to the particular context of the language game. The exclamation ‘Water!’ on its own has no inherent meaning, but in the context of a particular language game, it can be an answer to a question, an order, or a plea. This leads to a use theory of language: the meaning of a word is given only by the way it is used in successful language. The thread that links words or concepts to the facts of the world is thereby severed. Words are no longer things that point beyond themselves to some state of affairs out there in the world, but rather achieve the effects they have upon their users (i.e., us—up until fairly recently) by means of their interrelations and interactions.
The ears of those having a casual familiarity with the mechanics of Large Language Models (LLMs) and the underlying transformer architecture (in particular the attention-mechanism) will have pricked up at the preceding description. The ‘interrelations and interactions’ of words are, in the end, all LLMs have to go on—and they do seem to manage to extract a convincing faculty for language from that. So do LLMs prove WittgensteinPI right? Do we have here a rare example of a philosophical dispute settled empirically?
Indeed, this perspective is sometimes floated (see, e.g., here). Moreover, if there is nothing but use to understanding, and LLMs master use, then it would seem that LLMs also possess understanding on that note. But this strikes me as too quick a conclusion.
The Blind Puzzler
Consider a blind person engaged in the activity of assembling a jigsaw puzzle. While tedious, provided that the shapes of all puzzle pieces are sufficiently distinct, this is a distinctly possible activity: each piece laid down constrains the shape of the pieces around it, with a unique solution at least imaginable. Thus, it is not necessary to rely on sight.
Yet still, a sighted person will most definitely rely on the clues provided by the image shown in order to place each piece appropriately. But it’s at least imaginable (in the sense of credulity-straining possibility that things are ‘imaginable’ in philosophical thought experiments) that a blind and a sighted puzzler may not overtly differ in their ability to assemble a puzzle, even if they differ in their method to do so.
We are, you guessed it, in bumblebee territory: the same capacity can come about in distinct ways, and we confuse these at our peril. But the question then remains: given access solely to the output, and not to the mechanisms by which it came about, how can we tell apart the different cases?
Suppose a sufficiently devious philosopher or Cartesian demon were to surreptitiously exchange one piece of the puzzle for one with the same shape, but showing a different image. The blind person would have no means of detecting this substitution, while the mistake introduced into the image is obvious on sight. Thus, while the same output can be achieved by different mechanisms, the errors each mechanism produces will, in general, be different: a mark of how the result was achieved. A bumblebee crashes differently from a Boeing 747, because while the mechanism has been selected for to produce the same effects in ordinary, fault-free operation, its failure modes have not been subject to any aligning pressures and therefore, will generally differ. A second point of distinction is in the resources and conditions the process needs for smooth operation: the blind puzzler will not care, for example, whether the light in the room is on, while turning it off will significantly impede the performance of a sighted person.
It is thus through investigating the errors a system makes and the resources it consumes, rather than its successful functioning, that we can glean some insight into the underlying mechanism. If two systems tend to make similar errors and work under similar conditions, we have good grounds on which to consider them alike in their implementation. Conversely, if two systems show the same behavior during ordinary operation, but differ in the ways they fail, or in the necessary inputs, we should consider them functionally distinct.
The blind puzzler is not an altogether bad metaphor for the operation of LLMs. The attention mechanism essentially defines a ‘shape’ that the predicted word must fit by taking into account the shapes of all its relevant surroundings, and the best fit is the LLM’s prediction to continue the text. I won’t go into the details here, but for a 30,000-foot overview, the three relevant concepts are the query, the key, and the value. Any given word produces a query: what that word is looking for, roughly. It also yields a key: what it can supply. Words whose key matches the query of another are more relevant to that word, and thus, provide more relevant context. This relevance determines the extent to which the value of those words informs the representation of the original word. Thus, the representation of a word (more accurately, a token) is enriched by those words in the text that are most relevant to it. This, in the end, helps determine the ‘shape’ of the blank that the continuation needs to fill, in the sense of providing a set of constraints it needs to obey. (However, in distinction from the case of a puzzle, there is generally no unique solution, but rather, a broad set of possible options.)
In this story, no attention needs to be paid to what any given word refers to—so a picture theory simply plays no role here. (Indeed, I think the analogy goes further: I have proposed an argument intended to show that the meaning of a given utterance indeed does not matter to LLM language production, just as the image does not matter to the blind puzzler. But I will not pursue this line of thought here.)
By Their Errors Shall You Know Them
Large Language Models show broadly the same behavior as human beings when performing their chief function, namely, producing language. Indeed, recent iterations fulfill that function so well that they are no longer reliably distinguishable from humans. This alone represents a historic shift: previously, there was exactly one entity capable of language production (on a human-like level of complexity) in the known universe; now, there are two. Just a few short years ago, few would’ve predicted this current state of affairs (I certainly wouldn’t have), and those that did were largely seen as, well… outliers, to put it politely. This surely has profound implications, but nevertheless, we need an appropriate level of care in teasing out what they are.
Thus, the mere fact of a broad behavioral equivalence between LLMs and humans in the department of knowledge creation licenses few conclusions about that facility in both us and AIs, simply because it may be that the way it is implemented differs grossly. That LLMs seem to follow a ‘language game’ use-based model does not mean we do; that humans possess an understanding of words in order to assemble meaningful text does not entail that so do AIs. The sighted puzzler using the picture presented to identify the next piece does not mean that so does the blind person. The airplane’s equations do not curtail the bumblebee’s flight.
Intelligence, understanding, meaning, even consciousness: these are commonly believed to be features of human production of language. But attributing these to LLMs on the basis that they are our equals in terms of language production runs roughshod over important differences. One difference is in the kinds of errors LLMs produce. The other is in how the facility of language production is acquired in the first place. Together, these yield evidence that LLM and human language production are as different as the flight of the bumblebee and the Boeing.
The point here is not that LLMs make errors at all—so do humans. Indeed, even hallucinations are not too far removed from human confabulation. But there are certain kinds of errors that LLMs make that no ordinarily competent human would make. One widely publicized example was the persistent failure to count the ‘r’s in the word ‘strawberry’. Or ChatGPT’s near-breakdown at failing to produce a seahorse emoji. Or the fact that if you give it variations on the classic ‘goat, wolf, and cabbage’-puzzle, even if it includes no problem at all (such as if there’s only a wolf and a cabbage), it will generate convoluted scenarios with little grounding in the actual facts of the matter.
Now, all of these errors will eventually be trained out of the model. But that’s beside the point: as long as the basic function of the LLM doesn’t change, the fact that (on some iteration) it makes errors such as these, which are radically unlike human errors, already suffices to warrant the conclusion that LLM language production is just as radically unlike ours. As argued before, these errors constitute falsifying evidence for the conclusion that generative AI constitutes an example of human-like intelligence.
Furthermore, the way in which LLMs acquire their linguistic prowess is radically unlike the analogous process in us. Human language acquisition is, basically, a miracle: from just whatever scraps of language are uttered in our immediate vicinity in early childhood, with comparatively little in the way of directed instruction, we are eventually able to claw our way into language. In comparison, an LLM needs data the equivalent of which would take a human being tens of thousands of years to read, let alone internalize—vastly less efficient. (Against this, it is sometimes held that human beings have access to further information, such as the much more voluminous data streams of sight and sound—but deaf and blind people can acquire language, too. Likewise, it is occasionally said that we are born with the rudiments of language already present—we come pre-trained, in some sense, equipped with a Chomskian ‘Universal Grammar’ to build upon—but this, too, seems in tension with the current state of scientific knowledge.)
From these examples, then, it seems that what Large Language Models can tell us about our own facility of language production, and conversely what the way we make meaning can tell us about what goes on behind the curtain in LLMs, is rather limited. This might seem to be a deflationary conclusion, but I don’t believe it is: what the advent of LLMs has taught us is that there’s more than one way of producing language at the human level of complexity. This, I think, is a genuinely novel possibility that nobody has anticipated (to the best of my knowledge), and that remains widely underappreciated. Language isn’t just one thing, just as flight isn’t just one thing. Different paths to the same capacity exist. What other options lie out there, waiting to be discovered?
Enjoying the content on 3QD? Help keep us going by donating now.