John Pavlus in Quanta:
A picture may be worth a thousand words, but how many numbers is a word worth? The question may sound silly, but it happens to be the foundation that underlies large language models, or LLMs — and through them, many modern applications of artificial intelligence.
Every LLM has its own answer. In Meta’s open-source Llama 3 model, each word contains 4,096 numbers; for GPT-3, it’s 12,288. Individually, these long numerical lists — known as embeddings — are just inscrutable chains of digits. But in concert, they encode mathematical relationships between words that can look surprisingly like meaning.
The basic idea behind word embeddings is decades old. To model language on a computer, start by taking every word in the dictionary and making a list of its essential features — how many is up to you, as long as it’s the same for every word. “You can almost think of it like a 20 Questions game,” said Ellie Pavlick, a computer scientist studying language models at Brown University and Google DeepMind. “Animal, vegetable, object — the features can be anything that people think are useful for distinguishing concepts.” Then assign a numerical value to each feature in the list. The word “dog,” for example, would score high on “furry” but low on “metallic.” The result will embed each word’s semantic associations, and its relationship to other words, into a unique string of numbers.
More here.
Enjoying the content on 3QD? Help keep us going by donating now.

T
W
P
T
I
In 2016, scuba-diving philosopher Peter Godfrey-Smith made a huge splash with his book
I opened the library’s glass door and placed a fresh copy of my latest novel, What the Dead Can Say, on the bottom shelf. Then I returned to the car. For months my wife and I had been driving around the country, dropping off free copies of What the Dead Can Say in hundreds of Little Free Libraries. Now, we turned back on to the main road and headed down to Colorado, in search of more.
The operation that used pagers and walkie-talkies to
No idea has fallen flatter than the “end of history,” popularized by political philosopher Francis Fukuyama in his eponymous 1993 book. Few still believe that all human beings will accept liberal democracy and free market capitalism as the final forms of society and are uninterested in any alternative. But like many truly awful ideas, the end of history had its 15 minutes, or in this case 15 years, of fame, as a catchall motivation for America’s misguided attempt to export democracy to Afghanistan and Iraq.