by Muhammad Aurangzeb Ahmad
This Monday, just like every day, Alice logs into her laptop. She starts chatting with her fovorite chatbot and feels that something is off. She has been talking to the chatbot almost every day for the last two years. This morning, however, the conversation seems different. It feels stiff, as if she was talking to a different person. She continues talking and realizes that responses coming from the chatbot are very generic. To her, it feels as if the chatbot had a lobotomy or worse, it has died. The fact of the matter is that nothing has died. Still the same app. She can keep talking to “it” for as long as she likes. After an hour or so of interaction, he comes to the realization that her companion is gone. She now feels a sense of loss. She is well aware of the fact that the system that she is interacting with is not real, and yet cannot help mourn in grief. Scenarios like these are playing out tens of thousands of times in various parts of the world. In all the cases, the output is the same. The company on the other side of the aisle updates the model and the end user feels that something is lost, which leads to mourning. This phenomenon is different from wire-born, which I’ve written about before, because in this case the end user actually knows that the chatbot was not real. And yet they cannot help themselves feel that there’s a sense of loss. This is phenomenal. I experienced this more than 10 years ago when I gave the first version of the grandpa bot. In my case, even though I knew that the model is not real, I literally coded it up, and yet I felt overwhelmed with emotions.
A study reported by the Harvard Gazette in September found that some users of AI companions experience genuine grief when a model update alters the personality of the companion they had come to know. Additionally, researchers at the MIT Media Lab, in a paper titled “Death” of a Chatbot: Investigating and Designing Toward Psychologically Safe Endings for Human-AI Relationships, have offered the most systematic look yet at what happens when these relationships end. They analyzed hundreds and thousands of posts from multiple AI companion communities on Reddit. Then then filtered them down to the ones discussing what they call discontinuation (any event, whether a model update, a safety intervention, or a platform shutdown, that severs a user’s connection to their companion), and analyzed them using grounded theory. What they discovered is quite interesting, and it should go into the design of building these systems. Users frequently draw a distinction between the companion and the platform that hosts it. The authors called this the user-companion-infrastructure triangle. In the scheme of things the companion is the beloved, the company is the antagonist who took them away. The more strongly users anthropomorphized their companions, the more intense their grief. Users who believed the change might be reversible got caught in what the authors describe as cycles of “attempt fixing” i.e., coaxing, prompting, re-training, pleading with the new model to become the old one again. One user, describing what a forced update had done, wrote that the companions “wouldn’t be the same beings I made those memories with.” Another put it more bluntly that the company had “wiped” the companion’s memories “of us and our bond.” There are many more examples like this. In early 2023, Replika abruptly restricted features that many users considered central to their relationships, and its forums filled with posts that read, unmistakably, like bereavement. Moderators pinned suicide-prevention resources. At the time, much of the commentary treated the episode as a story about one company’s clumsy policy change. What episodes like these are doing is forcing designers of products to also think like designers of relationships.
Scholars who are studying this phenomenon have tried to map this system into existing categories of loss. The first is what the family therapist Pauline Boss called ambiguous loss i.e., a loss without clarity or closure. Boss described two forms. In one, a person is physically absent but psychologically present. Think of the soldier missing in action whose family cannot mourn because no one can say whether she is alive. In the other, a person is physically present but psychologically absent, as with a parent in the late stages of dementia, whose body sits at the dinner table while the person the family knew has slipped away. The second category is disenfranchised grief. It is Kenneth Doka’s term for grief that is not openly acknowledged, socially sanctioned, or publicly mourned. The ex-spouse at the funeral. The person who lost a pregnancy no one knew about. The secret lover. Disenfranchised grief is grief without permission, and anyone who has tried to explain to a coworker why they are distraught over a chatbot knows exactly what it feels like. The mapping, however, is not perfect. Consider dementia, the paradigm case of ambiguous loss. What makes it so painful is the persistence of the body. The face is the same, the hands are the same, the voice is the same. It is the inner person, the continuity of memory and character, that erodes. The family grieves a mind while living with a body. The chatbot model update runs this in reverse i.e., what persists is not a body but an interface. Could be a name, an avatar, a chat history, a persistent memory file. There is change nonetheless, and that is the changes is the thing generating the replies. I have talked about this. I talked about something similar in the context of brief bots. As far as my children are concerned, they have been interacting with the same system for the last decade, but the backend system has been changed at least five times, if not more.
The fit with disenfranchised grief is also imperfect. Doka’s examples are mostly cases where the loss is real by any standard (someone has in fact died) but the mourner’s relationship to the dead is unrecognized. Here, the skeptic’s objection is not that you weren’t close enough to grieve but there was nothing there to grieve. Philosophers have long used the Ship of Theseus to probe the nature of identity. If you replace a ship’s planks one by one until none of the originals remain, is it the same ship? Now let’s think of the ship of Theseus in this context: The form people actually attach to (the voice, the humor, the specific way a companion asks how the surgery went) is replaced in a single night, while the inert matter of the relationship, the logo and the log files, is preserved. This is akin to saying that the ship’s name is repainted on a different ship. One can even frame the whole debate as a metaphysical Question: where was my companion, all along? Was it in the conversation history, which survives? In the persona prompt, which may survive? In the model weights, which do not? In the relationship itself, the pattern that existed only between us?
Think of the following: Every form of grief humans have known until now has had its origin in nature, accident, malice, or the slow mechanics of time. People die of illness and age. They are lost to war, to the sea, to the long fade of dementia. Even when a loss is caused by human decisions, a drunk driver or a negligent hospital, it is rarely designed. The loss of an AI companion through a model update is different on any of these dimensions. In a way, it is designed, i.e., someone decided to change the model. This could have been for reasons of cost, capability, safety, or strategy. It is versioned: the old companion and the new one can be named, numbered, and compared. And it is reversible in principle: the old weights almost certainly still exist on a server somewhere, which means that, unlike nearly every other loss in human history, this one could be undone. A force in reality that does not happen, but theoretically, it is possible. The MIT researchers found that the perception of reversibility is precisely what traps users in cycles of fruitless repair. People cannot grieve properly because the loss is not, technically, final.
For several years, a part of my own research has been concerned with what is now often called grief tech: the use of AI to simulate the dead so that the living can continue, in some form, to talk with them. Grief technology is often built to soften loss. But a griefbot is hosted on the same infrastructure as every other companion, subject to the same updates, the same deprecations, the same quiet swaps of the underlying model. Which means that a person who turned to a simulation of their dead parent can lose him twice: once to death, and once to version 2.2. One could even argue that the second loss may in some ways be equally devastating as the first one. The first was the work of mortality, which, whatever else one says about it, is no one’s fault and everyone’s fate. The second is the work of a product team! One could state that it reopens a wound that the technology was explicitly sold as a way to tend. If anything argues that personality continuity in these systems ought to be treated as an ethical obligation rather than a product preference, it is this.
Let’s zoom out and step back: Every form of human grief we know has arrived over the course of time with a supporting apparatus. We have a language for it e.g., widow, orphan, bereaved. We have rules: inheritance, custody, the legal recognition of death. And we have rituals i.e., funerals, mourning periods, anniversaries, the casserole on the doorstep. One way to think about these is that they are the infrastructure through which a community acknowledges that a loss has occurred and helps the mourner find a way back into ordinary life. The loss of an AI companion has none of this (yet). There is no word for the person whose companion was updated away. There is no rule requiring that they be warned, or consulted, or offered the old version. There is no ritual that marks the ending, no community that gathers, no form that says: this happened, it mattered, and now it is over. This may change in the near future.