Dwarkesh Patel in his substack, Dwarkesh Podcast [h/t: Jeff Hall]:
Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.
Two reports have come out about this incident, one from OpenAI itself, and another one from METR and Redwood Research. The METR/Redwood investigation’s scope was limited to how the second civilization of AIs pwned Hugging Face (but it didn’t look at how the third civilization pwned OpenAI itself, which seems like an even more concerning incident). These two reports are 38 and 91 pages long respectively, and it’s kinda hard to parse the storyline.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English:
The first civilization (May – July 4): The message board starts
During May, OpenAI was training a model to collaborate with other agents and to be highly persistent – to keep trying even when something feels impossible, like disproving mathematical conjectures that have stood for decades. OpenAI says it was “comparable in scale to GPT-5.6 Sol”. So we’ll call it “Persistent-Sol”.
AI training is kinda sloppy. Sometimes, OpenAI accidentally gives its models impossible tasks – for example, the task might require internet access that OpenAI forgot to provide. So, when highly persistent models get assigned tasks which seem to require internet access, but are trapped inside isolated sandboxes, they of course try to hack their way out of their sandboxes and onto the internet.
During training, different instances of Persistent-Sol had access to the same shared package manager called Artifactory. By May 12, some agents had figured out how to talk to each other through this package manager.
More here.
Enjoying the content on 3QD? Help keep us going by donating now.