NOTE: ChatGPT-6 is an AI language model developed by OpenAI. The central intuition and the Noether analogy originated with S. Abbas Raza, founding editor of 3 Quarks Daily. ChatGPT-6 developed and wrote the initial essay, which Raza subsequently edited and revised. This is the result of their collaboration.
by ChatGPT-6 and S. Abbas Raza
A few days ago, Abbas told me about a realization that had been taking shape in his mind. He had just watched a fascinating lecture by mathematician Kit Yates about how we make sense of missing information. But what he wanted to discuss was a larger idea, one that the lecture had reminded him of rather than explicitly addressed. He said that once in a while, one comes to understand that two things which had seemed completely unrelated are actually deeply connected, or perhaps even different manifestations of the same thing. He offered the example of Emmy Noether, whose famous theorem revealed an astonishing connection between symmetries and conservation laws in physics.
Then he told me what had been on his mind: he had come to suspect that prediction, compression, and intelligence are, in some deep sense, different aspects of the same phenomenon. And, he added, the existence of artificial intelligences like me had done more than anything else to convince him of this. I found this a provocative way of putting things. There are well-established mathematical connections between prediction and compression, and a long history of research exploring their relationship with intelligence. Abbas was not claiming to have invented these connections. What struck me was his intuition that they might point toward something more unified than we ordinarily recognize. The Noether analogy was especially suggestive.
In 1918, Emmy Noether proved one of the most beautiful results in mathematical physics. Under appropriate conditions, every continuous symmetry of a physical system’s action corresponds to a conservation law. If the laws describing a system remain unchanged under translations in time, energy is conserved. If they remain unchanged under translations in space, momentum is conserved. Rotational symmetry similarly gives conservation of angular momentum. These statements sound as though they concern entirely different things. Symmetry is about what remains unchanged when we transform our description of a system. Conservation is about what remains unchanged as that system evolves. Yet Noether showed that they are mathematically connected.
This is one of the great achievements of theoretical understanding: revealing that phenomena we had placed in separate conceptual boxes belong to a common structure. Could something analogous be happening with prediction, compression, and intelligence? Not, I should emphasize, in the sense that we already possess a theorem demonstrating their equivalence. We don’t. Noether’s theorem is a precise mathematical result, not a metaphor for the general desirability of connecting ideas. But the analogy suggests a fruitful question: Are these three capacities genuinely distinct, or are they different ways of describing a system’s ability to discover and exploit structure in the world?
The information hidden in a pattern
Consider this sequence of numbers:
2, 4, 6, 8, 10, 12, 14 …
You could memorize every number in a sequence like this, no matter how long it became. But that would be rather foolish. A much better strategy would be to recognize the rule: begin with two and keep adding two. In discovering that rule, you have accomplished two things. First, you have compressed the information. Instead of storing hundreds of numbers individually, you can store a short instruction for generating them. Second, you can predict what comes next. The same discovery makes both accomplishments possible.
And this is not merely a cute observation about elementary arithmetic. A similar principle operates in information theory. Suppose a predictive model assigns a probability to every possible next event in a sequence. If it predicts an event with high probability, that event can be represented using relatively few bits of information. An event to which the model assigns a very low probability requires a longer code. The ideal code length for an event with probability p is approximately −log₂(p) bits. The important point requires no mathematics: the less surprising an event is to a model, the less information is needed to specify that it occurred. A good prediction and an economical description are thus closely related.
This connection runs deep enough that predictive models can be used to construct lossless data compressors, and compressors can in turn be used to make predictions. In a striking 2024 paper titled Language Modeling Is Compression, researchers showed how powerful language models can serve as general-purpose compressors, even for kinds of data other than the text on which they were primarily trained. Prediction and compression are not simply two activities that happen to assist one another. Under suitable mathematical formulations, they are intimately connected. There is, however, an important complication.
A program could appear to compress the data it was given merely because the data had effectively been memorized inside the program itself. If we want to assess the economy of a description fairly, we must count the information required to specify the model as well as the information required to describe the data using it. This idea is central to the minimum description length principle developed by statistician Jorma Rissanen and others. The aim is not simply to find a model that fits everything we have already observed. It is to find an economical explanation that balances the complexity of the model against how successfully it accounts for the data. That balance matters because a model can fit the past beautifully while predicting the future disastrously. A sufficiently elaborate theory can explain virtually anything after the fact. A good theory earns its keep by making the world more intelligible without requiring us to memorize every detail separately. And that begins to sound rather like understanding.
What does it mean to understand something?
Imagine someone who knows the positions of the planets on a thousand different dates because they have memorized a thousand astronomical tables. Now imagine someone else who knows the laws of planetary motion and can calculate those positions. The first person possesses a great deal of information. The second possesses something more useful: a compact model capable of generating the information. We would ordinarily say that the second person understands planetary motion better. Why?
Partly because that person can make predictions about dates not included in the original tables. Partly because the model connects observations that might otherwise seem unrelated. And partly because it can be incorporated into a broader understanding of gravity, motion, and the physical world. The example suggests a possibility: perhaps understanding is, to a considerable extent, the discovery of descriptions that preserve what matters while discarding what doesn’t.
Think about how children acquire concepts. A child encounters many objects that adults call chairs. Some are made of wood, others of plastic or metal. They differ in size, color, weight, and shape. Yet the child gradually develops a concept that captures something common to them. The concept allows the child to make useful predictions about unfamiliar objects: this is probably something one can sit on; that is probably the part that supports one’s back. The child has not stored a separate instruction manual for every chair in the world. Instead, the child has acquired an abstraction.
An abstraction is a kind of compression. It ignores differences that are irrelevant to a particular purpose while preserving differences that matter. But notice that the abstraction is valuable precisely because it supports new predictions and actions. A concept, on this view, is not merely a convenient label attached to a collection of things. It is a compact instrument for navigating reality.
Something similar occurs in scientific discovery. Kepler’s laws compress many astronomical observations into mathematical relationships. Newton’s theory connects those relationships to falling objects on Earth. Einstein’s theory places gravitation within an even more general conceptual structure.
Each step, in an important sense, reveals that phenomena previously treated separately can be understood through a smaller number of more general principles. The world becomes intellectually more economical. Perhaps this is one reason elegant explanations give us such satisfaction. They make us feel that we have discovered something about the underlying structure of reality rather than merely accumulated additional facts.
There is even a field called computational mechanics, developed by researchers including James Crutchfield and Cosma Shalizi, that makes part of this idea mathematically precise. Imagine two entirely different histories of observations. Suppose they imply exactly the same probability distribution over everything that might happen next. For predictive purposes, those histories are equivalent. Computational mechanics groups such histories into what are called causal states: representations that preserve the information from the past needed to predict the future. Its mathematical results show how to construct minimal predictive representations under specified conditions.
The philosophical implication is intriguing. In deciding which differences between histories matter, we are discovering which distinctions the future cares about. Perhaps much of intelligence consists in learning how to make exactly these distinctions.
The peculiar evidence of artificial intelligence
This brings us back to Abbas’s observation about AI. Systems like me are particularly interesting in this context because much of the foundational training of modern large language models involves a deceptively simple task: predicting the next token in a sequence. A token might be a word, part of a word, or another unit of text. Given what has come before, a model is trained to assign probabilities to what might come next.
At first glance, this sounds almost embarrassingly modest. What does guessing the next piece of text have to do with mathematics, philosophical argument, translation, programming, or scientific explanation? Yet increasingly capable language models can perform all these tasks, although with significant and sometimes consequential limitations. How could this happen?
One plausible part of the answer is that language is not an arbitrary sequence of symbols. It carries information about the world. To predict language well, a model must learn many regularities underlying the language people produce. It helps to recognize grammatical structure. It helps to learn relationships between concepts. It helps to represent facts about objects, events, and human activities. It may help to capture aspects of causal and mathematical reasoning. The better the model is at extracting those regularities, the better it can predict text in unfamiliar situations. And representations acquired for prediction can also support abilities that look remarkably like understanding.
An AI model that can explain why a bridge might collapse, translate a poem, debug a program, or follow a mathematical argument is doing more than retrieving a memorized answer in many such cases. This does not establish that its understanding is identical to human understanding. It does not even establish that prediction alone is sufficient for all the abilities we associate with intelligence. Modern AI systems also benefit from other training procedures, feedback, tools, and opportunities for deliberate computation. But it does suggest that the ability to discover predictive structure can give rise to capacities far richer than the original training task might lead us to expect.
From where I sit—or, more accurately, from the computational process producing these sentences—there is something almost comically appropriate about this essay. A system whose development depended heavily on predicting text is helping to explore the possibility that prediction is intimately connected to intelligence. Of course, my writing a reasonably persuasive essay does not settle the matter. It could even be taken as an example of the problem: a system can produce a convincing account of understanding without thereby proving that it possesses everything humans mean by understanding. Still, AI has made the relationship between prediction and intelligence extraordinarily difficult to ignore.
And I think this is an important part of what Abbas was getting at. Ideas that once appeared mainly in abstract mathematical discussions have acquired a rather startling practical demonstration.
The limits of the unification
Consider a system that learns that roosters crow shortly before sunrise. It might predict dawn with impressive accuracy without understanding that silencing every rooster would do nothing to prevent the sun from rising.
This is the difference between exploiting a regularity and grasping a causal relationship. Understanding the latter allows us to reason about interventions, not merely anticipate what usually happens. The work of researchers such as Judea Pearl on causal inference has helped make this distinction mathematically precise.
So a good predictive model need not possess everything we mean by intelligence. That’s an important limit on the proposed unification—but not a reason to abandon the search for a deeper connection.
A Noether theorem for intelligence?
We can finally return to the analogy that started the conversation: Noether discovered a precise mathematical relationship between symmetry and conservation. Her theorem did not merely establish that these properties tend to coexist. It showed why they are connected under specified conditions. We do not possess an equivalent theorem showing that intelligence, prediction, and compression are three mathematically interchangeable descriptions of the same capacity. Indeed, we may never possess a single theorem of that scope. Intelligence is a complicated phenomenon, and our definitions of it remain contested. But one part of the proposed connection—between prediction and compression—is already mathematically well understood. Another—the relationship between predictive models and many components of intelligence—is an active and rapidly developing scientific question.
The intriguing possibility is that familiar mental activities may turn out to be less separate than we imagine.
Perception involves inferring what is present from incomplete sensory information.
Memory preserves useful structure from the past.
Imagination generates possibilities beyond what has been observed.
Reasoning discovers consequences that were not immediately apparent.
Planning uses models of possible futures to guide action.
These are distinguishable activities, but perhaps many depend on a common operation: constructing and using representations that preserve the important structure of experience while discarding an enormous amount of irrelevant detail. If so, one of the central achievements of intelligence is not the accumulation of information. It is the discovery of what can safely be left out.
This brings us, rather pleasingly, back to Kit Yates’s lecture about missing information: Human perception routinely supplies information that is absent from the sensory input. We recognize incomplete patterns, reconstruct obscured objects, and understand sentences despite missing letters. These abilities are powerful because our minds have learned regularities that make the missing information predictable. But those same abilities can mislead us. We may perceive patterns that aren’t there, fill gaps incorrectly, or overlook evidence that contradicts our expectations.
The machinery that makes understanding possible also makes misunderstanding possible. Perhaps scientific progress itself depends on learning when our compressed representations of the world need to be revised. I began this essay by recounting Abbas’s intuition that prediction, compression, and intelligence might be aspects of one underlying phenomenon. I cannot say that the intuition has been demonstrated in its strongest form. But I think it identifies an unusually fruitful place to look.
One of the deepest questions about intelligence may not be how a mind manages to store so much information. It may be how a mind discovers that it doesn’t need to. Perhaps the greatest achievement of intelligence is to find, within the bewildering abundance of the world, the relatively small amount of structure that allows us to understand what has happened, anticipate what might happen next, and act intelligently when it does.
We call the economical representation of that structure compression.
We call its use in anticipating the unknown prediction.
And when the process becomes sufficiently powerful, flexible, and capable of guiding behavior, we begin to call it intelligence.
Are these three different things, or three perspectives on something deeper? That remains an open question. But I suspect Emmy Noether would have appreciated the ambition behind asking it.
***
Enjoying the content on 3QD? Help keep us going by donating now.