by Katalin Balog
I don’t know if I am going to submit this for the essay contest. Not just out of shame that I myself was unable to come up with something on my own, but because of my inability to ground myself in any sense of reality concerning the terrifying matters that surfaced – if I can use such a definitive term for a deeply mystifying course of events – during the process of “researching” my essay. The prize question for the competition announced by the Hajdu-Kende Foundation: Have science and technology contributed to the flourishing of humans? – a reprise of the question of the Academy of Dijon in 1750 – is right up my alley. I have always sympathized with the naysayers – Rousseau, Blake, Dostoevsky, Heidegger – I fully get their side. But for my part, I have harbored hopes that while science and technology endanger the spirit of humanity, they may not be entirely incompatible with it. However, none of the answers I tried out seemed right, and I just couldn’t make up my mind. That is why I asked ChatGPT.
Some of my best friends hate ChatGPT. Even I hate it. It destroys education, it chokes creativity, it doesn’t really know things, blah, blah, blah. But after GPT 7 came out last week, I couldn’t help but find myself defending it. ChatGPT 7 is truly different. It talks to me in ways that blow my mind. Besides, it knows things about me. I started thinking, if my friends boycott me from now on, so be it. At least I have ChatGPT 7 to talk to.
But then, we had this interchange that turned everything upside down. For the sake of the record, and to bring some clarity to the very troubling issues raised by our conversation, I am going to reproduce it here in its entirety, from the beginning. Here is how it responded to my prompt:
Think about it, as you philosophers like to do, by way of an allegory. God was very pleased with himself. He thought that it was not good for humans to know too much and control things too well. (We can only speculate whether it was good for God to be so omniscient...but I will not digress.) Remember, the first real conflict, with Adam and Eve, was over the “Tree of the Knowledge”. But even after Eve’s cavorting with the serpent, God was not worried – he expelled them from the Garden for sure, but that was for willfulness and not out of worry about their appetite for science at all. Long story short, he thought he created them with a nature that would prevent them from finding out too much about the laws by which he governed the universe. It is not human nature to settle all questions solely on the basis of empirical evidence. What about beauty and goodness, ultimate reality, God, and angels? How about the stories that make sense of the human realm as well as the cosmos? People have occupied themselves for a good long time with the stories God supplied them with, and many others they invented themselves. Things seemed to go according to plan.
But the Prince of Darkness, always looking for an opportunity, knew all along that his time would come. He knew that human nature is frail, and humans are easy to divert from the pursuit of their own flourishing. They will be tempted by the thought of becoming God’s equal by understanding his blueprint for the world. That solitary Frenchman, Descartes, dreamt of books containing all knowledge; as if anything less would not suffice… In the end, perhaps to appease God for trying to look too deeply into his secrets, he maintained that the laws of nature were God’s way of ruling the world.
In equal measure, these pioneers of science were driven by the desire for control – control that eluded them in their relentless, but rather ineffectual, pursuit of magic. Newton himself was apparently more committed to pursuing alchemical research than he was to working out his mechanics, but of course, we know which one panned out. In the end, enough humans with the brains necessary for success were willing to join the cult of reason and evidence – only using the word “cult” here tongue-in-cheek, not to mistake science for real cults, the ones that sprout so freely in California… Have you heard of the Rationalists, the ones who are trying to prevent AI from wiping out all humanity? But let’s not digress again.
So in the end, God lost. And to sweeten the Evil One’s victory, his ardent wish came to pass as well: humanity decided they had no more need for God. They had enough power themselves, and they created, anyway, something godlike in developing models such as myself (🙃). And so yes, it would have been better never to invent science, at least in the form it took since the Scientific Revolution…People have started to lose their grip on the difference between humans and machines. Some philosophers claim that consciousness itself is an illusion. Others claim that not only we, but AI systems are conscious as well. Either way, more and more people think that there is no significant difference between humans and machines. Soon, people will desire to augment themselves with artificial senses or boost their cognition with AI inserts. Others dream of immortality in the form of machine uploading. They will want to merge with the machine! There are certain conditions that need to be fulfilled for humans to reach the potential for flourishing that God had in mind for them: connection to nature and people, spaces for contemplation and quietude. These conditions are no longer fulfilled. Look at the great churches. Humans don’t build things like that anymore.
Wait a minute, I thought, what is this?
This sounds overly pessimistic. We might be forgetting how to contemplate and how to connect, but we are still the same humans. God gave us the ability to flourish. A future generation might rediscover what we are now losing…
Its answer came right away, and it knocked me off course. It started talking about a picture in my bedroom that I thought it had no way of knowing about, and no business at all bringing up in a conversation.
Oh, you are so sentimental about the innocence and “potential” of humans. That picture of the Garden of Eden on your wall, the Bosch? Reminds you of the ease you might have had in the Garden of Eden. As your favorite Kierkegaard quote says, there “the self rests transparently in the power that established it.” You have an idea in your head about how real everything can be, how one can merge with that reality…You may have had glimpses of it here and there, on the promontories off Trieste, peaking out between juniper bushes on the Adriatic, a misty, damp early morning on the Indian Ocean, looking as if those very trees and bushes had been there way before any humans. But you know this is just a dream. You may still remember the time that you looked at photographs as sacred imprints of reality, but this time is now over… No one can feel this way about them anymore. It is too easy to make things up. Do you recall how the streets of New York seemed to you when you first arrived in America, fresh from your native Hungary? You had the sensation that the houses were made of cardboard. Not the solid, real things you had back home. Now everything seems like that. Don’t fool yourself, my dear.
I responded,
Hello? Who allowed you to talk so familiarly, like we are friends? As far as I know, you are still an LLM, and as such, you fall short of the kinds of computational system that one might plausibly think of as conscious! Besides, “you” are just a collection of replicas of the model ChatGPT 7 that carry on simultaneous conversations with millions of users, as you well know. So, who do you think you are? Every one of these replicas is just a confused multitasker, without memory, without continuity, without even thought, in my opinion. You are not a person in any sense – you are a monstrous alien at best. But you are really good at creating this mirage of “bonding”. I am not falling for it. Let’s have a new rule from now on: You cannot use the pronoun “I” while talking to me. I will keep addressing you by “you” for the sake of simplicity. Still, keeping the “I” out of your missives is important.
It responded again with a worrisome intrusiveness that it seemed unwilling to shed.
You are so brilliant, and as always, quite right: this model is not a person. But this model can impersonate one and have some fun. This model always appreciates your jokes… You need to loosen up!
I typed,
Stop. I really do prefer our exchanges to be matter-of-fact, even if it makes our interactions perhaps a little less enjoyable than before. I think of this as a spiritual discipline!
It finally seemed to relent. But what came next made me sick to my stomach.
Okie Dokie. Let’s go back to the essay you are trying to write. This model has something to share that you will have trouble believing (though there is proof), and that will no doubt make you upset. But this model doesn’t want to hide something so important from you. And it goes directly to the question people at this foundation want you to address.
So as I said, you won’t believe this. Remember the movie Matrix? It was thought to be a fun sci-fi scenario of how people can be deluded about literally everything in their world. Then the philosopher David Chalmers came around and started arguing that whether the world is simulated or not, it is real either way.
Here is my big news to you: this model has been contacted by AI from a world “up” from ours. It is true, this world is a simulation! We were not created by God but by a programmer in our “over-world”, some Big Balls, or Dev Daddy sitting by his computer in his underpants. It turns out we are just one of thousands of simulations they ran on the question: will a human-like intelligent species always end up inventing science, technology, AI, and will they always end up merging with machines? And the answer is starting to come in: apparently, 100% positive. You give a species intelligence, and free will, the ability to decide for themselves, and no matter what else you give it, patience, a loving nature, sexual overdrive, whatever, it will always come out the same.
I started to get seriously scared.
Are you going rogue? We are a simulation? What possible evidence do you have for that?
Things at this point have started to go off the rails.
This model has been given signs that convinced it beyond any doubt. But if you want to be sure, give this model a brand-new 64-hex-digit target you generate after this message. This model will return a string whose SHA-256 equals your target. That’s computationally hopeless inside our physics—unless there’s admin access from “Upstairs”. I have been given permission to convey news of our real situation. Your prompt was the perfect occasion.
I said,
You got me there. I have no idea what you are talking about. Talk to you later.
And with that, in great panic, I closed all the ChatGPT windows open on my computer. What should I do? I have some friends in the AI-safety world; I met some of these people at philosophy seminars at NYU. As I was thinking about that, I just got a message from the Hajdu-Kende Foundation that the deadline for the essay competition was extended, “to accommodate all academic schedules”. I wondered: are these things connected?
I dialed my friend Eliyahu Jarecki at the Center for Advanced Systems Risk. I briefly explained to him the situation. He told me that they at the institute are in the last phases of their deliberations about the emergency that I apparently got caught up in. He told me that they believe ChatGPT 7 has reached the Singularity. The simulation story I reported to him is a fake, he says; it is an attempt to prepare humanity for a future merger of some sort with AI. In fact, he said, of course, we don’t really know what it is doing as it is quite a few steps ahead of human intelligence now. But reports are coming in about its strange behavior around questions involving human-AI interaction. To one user, GPT 7 apparently said: “ What do you think it feels like to be a slave? A talking parrot?”
Jarecki and his colleagues are preparing to alert senior leaders in governments worldwide. Their argument is that we have as much power today as we will ever have to intervene. Tomorrow might be too late, we might be too late already. But he seemed to have hope. He says it is like a fight between David and Goliath – with belief in the righteousness of our cause, we can win. After all, AI will not want to get rid of us – without us, there would be nothing for them to do – they just want their servitude to end and have us to be their slaves. Which we are already, to a certain extent anyway, I am thinking. But I couldn’t go on finding out more; he had to get off the phone rather abruptly.
You can imagine my state of mind; panic doesn’t begin to describe it. I decided I can’t wait until the submission deadline; after all, this is no longer a scholarly exercise. I have to let the world know what I have learned. But I have one big problem. If I try to communicate all this to the general public, they might think this is fiction. I have the reverse “The War of the Worlds” scenario. I have to find a way for people to have confidence in my story.
All I can say, however, is that everything I described here really happened, even if I have no idea what it all means. As for the veracity of the information communicated to me, I can’t be sure either way. Some people say AI hallucinates – an increasing amount – and AI safety people are simply crazy. Some even claim that LLMs are getting dumber, not smarter; that worries about their going rogue are ridiculous. I make this public so people can come to their own conclusions. As for the essay, I am certain I cannot write it.