by Muhammad Aurangzeb Ahmad
Everyone grieves in their own way. For me, it meant sifting through the tangible remnants of my father’s life—everything he had written or signed. I endeavored to collect every fragment of his writing, no matter profound or mundane – be it verses from the Quran or a simple grocery list. I wanted each text to be a reminder that I could revisit in future. Among this cache was the last document he ever signed: a do-not-resuscitate directive. I have often wondered how his wishes might have evolved over the course of his life—especially when he had a heart attack when I was only six years old. Had the decision rested upon us, his children, what path would we have chosen? I do not have definitive answers, but pondering on this dilemma has given me questions that I now have to revisit years later in the form of improving ethical decision making at the end-of-life scenarios. To illustrate, consider Alice, a fifty-year-old woman who had an accident and is incapacitated. The physicians need to decide whether to resuscitate her or not. Ideally there is an advance directive which is a legal document that outlines her preferences for medical care in situations where she is unable to communicate her decisions due to incapacity. Alternatively, there may be a proxy directive which usually designate another person, called a surrogate, to make medical decisions on behalf of the patient.
Given the severity of these questions, would it not be helpful if there was a way to inform or augment decisions with dispassionate agents who could weigh in competing pieces of information without emotions coming in the way? Artificial Intelligence may help or at least provide feedback that could be used as a moral crutch. It also has practical implications as only 20-30% percent of the general American population has some sort of advance directive. The idea behind AI surrogates is that given sufficiently detailed data about a person, an AI can act as a surrogate in case the person is incapacitated, making decisions that reflect what the person would have taken if they were not incapacitated. However, even setting aside the question of what data may be needed, data is not always a perfect reflection of reality. Ideally this data is meant to capture a person’s affordances, preferences, and more preferences, with the assumption that they are implicit in the data. This may not always be true, as people evolve, change their preferences, and update their worldviews. Consider a scenario where an individual provided an advance directive in 2015, yet later converted to Jehovah’s Witness—a faith that disavows medical procedures that involve blood transfusions. Despite this profound shift in beliefs, the existing directive would still reflect past preferences rather than current convictions. This dilemma extends to AI-trained models, often referred to as the problem of stale data. If conversational data from a patient is used to train an AI model, yet the patient’s beliefs evolve over time, data drift ensures that the AI’s knowledge becomes outdated, failing to reflect the individual’s current values and convictions.
Many of the challenges inherent in AI, such as bias, transparency, and explainability, are equally relevant in the development of AI surrogates. For instance, an AI surrogate must be able to justify its recommendations—how does it arrive at a particular decision, and can it offer a rationale that aligns with ethical considerations? Moreover, if the model fails to account for the decision-making patterns of minority groups, it risks exacerbating existing disparities. This issue is already evident in hospice and palliative care, where marginalized communities experience greater pain, increased caregiver burden, and poorer end-of-life experiences overall. Unlike other areas of AI deployment, end-of-life decision-making presents a unique challenge—there is no opportunity to test alternative outcomes after the fact. Another critical concern is language dialect bias. Large language models, for example, have been shown to exhibit biases against African American Vernacular English (AAVE), potentially leading to incorrect or skewed interpretations in end-of-life scenarios. Additionally, AI-driven decision-making does not operate in a vacuum—it is susceptible to external influences, such as cost-saving measures, insurance policies, and hospital efficiency goals. Given the opacity of many AI systems, it is conceivable that such pressures could subtly or overtly shape AI-generated recommendations. The history of healthcare is rife with examples of AI deployment where profit motives took precedence over patient welfare, leading to the use of unreliable or even harmful models in contexts where AI was neither warranted nor beneficial. However, this does not imply that AI surrogates are merely another case of technological overreach. There are notable success stories where algorithmic decision-making has demonstrably improved outcomes—for instance, AI-driven organ matching systems have enhanced transplant success rates by optimizing donor-recipient compatibility.
While taking into account the cultural and religious background of individuals is important, it may also clash with legal imperatives. Consider the case of an AI surrogate that has to make recommendation on behalf of a brain-dead pregnant women. A review of brain death protocols in American hospitals revealed that the overwhelming majority of them, around 94%, do not have any guidance on fetal management following maternal brain death. Here the wishes of the woman may clash with state directives e.g., certain states may overwrite the patient’s directive if there is a possibility of fetus viability. How one views such scenarios would depend upon where one stands on the issue of bodily autonomy and personhood of the fetus. There are unlikely to be clear cut answers even from an AI. Moreover, the AI’s decision may conflict with the moral compass of the physician. In case the surrogate decision maker, whether human or AI, causes the physician to experience moral distress, the physician may transfer care to a different physician. In case of absence of a clear directive, one has to take the cultural and religious beliefs of the patient into account as in the case of the Jehovah’s Witness example above. However, there is a fine line between coercion and recommendation. There has been some discussion regarding how to deal with situations where a Jehovah’s Witness patient may change their mind about blood transfusion in the middle of a crisis. That said, the counterargument is that in such a midcrisis reassessment the decision of the patient may be influenced by coercion from a physician. An AI surrogate would have to navigate the same terrain – suggest but do not coerce, take into account the patient’s values and not necessarily that of the dominant cultural group. This is an acute problem in large language models which often reflect the values of the dominant cultural group while neglecting minority ethnic and cultural perspectives.
There have been numerous efforts to explore the requirements of building such a system. As I have ventured into this domain, I have come to grasp the truly herculean nature of the task. Now, as I contemplate the current essay, I open my drawer and again find the last handwritten note my father left before departing this mortal abode. More than a decade has passed, yet I have repeatedly returned to the questions surrounding decision-making at the threshold of life and death—whether through modeling mortality predictions or witnessing firsthand the delicate interplay of fate and choice in the trauma ICU. There are seldom easy answers. In 2017, when I first began working in healthcare AI, one of my earliest projects involved predicting mortality within a given timeframe—determining, for instance, whether a critically ill patient would survive the next six months. Such predictions carry weighty real-world consequences, informing pivotal decisions: whether to transition a patient to hospice care or persist with treatment. Not long before that, I had created the first iteration of Grandpa Bot—a digital reconstruction of my father, designed for my children to engage with. Though seemingly disparate, these three inquiries are bound by a common thread: the pursuit of knowledge to better inform ethical decision-making at the fragile boundary between life and death. We may need all the help that we can get to navigate these water and AI crutches may help.