by Muhammad Aurangzeb Ahmad
The French poet Jean de La Fontaine has a famous quote that “A person often meets his destiny on the road he took to avoid it.” We find echoes of this phenomenon in global literature, whether it’s Oedipus in the Greek Myths, Rostum and Sohrab from Iran, or the story of Kamsa and Krishna in the Hindu tradition. There are elements of self-full filling prophecy that we are seeing in the world of predictive modeling. Consider the use of AI and machine learning models to predict risk of mortality in an ICU setting. Some of these models have extremely high accuracy and precision. They do in milliseconds what it would take a team of clinicians hours to synthesize. The predictive power of such models need to be contextualized however: A mortality prediction model is trained on historical data i.e., on what happened to patients who looked like this, had these labs, were managed in this way. But the historical data does not merely record biology, it also records medicine as it was practiced. This includes all its established patterns, its habits, its inequities, and its mistakes.
Consider a well known finding that has been often used as a cautionary tale: in a certain historical ICU dataset, patients with a diagnosis of asthma had lower predicted mortality than otherwise similar patients without it. This seems absurd, asthma is a serious respiratory condition. When researchers looked closely, they realized that the problem was not about asthma biologically but it was about care. Asthma patients were more likely to have their respiratory distress recognized early. They arrived with better documentation, better advocates, better access to specialists who knew them. The asthma diagnosis was not a protective biological factor. It was a marker of a particular kind of patient i.e., one who had navigated the healthcare system in a way that produced better documentation, faster escalation, more attentive management.
When a mortality prediction model learns from this data, it learns the pattern correctly. Asthma is, statistically, associated with better outcomes. However, if we deploy that model, it will assign lower mortality risk to asthma patients. The danger is that this may cause clinicians to be less vigilant about them, which will over time close the gap that the model detected, and possibly reverse it. This is not an isolated quirk. Researchers have formally characterized a class of prediction models that are harmful self-fulfilling prophecies: their deployment harms a group of patients, but the worse outcomes of these patients do not diminish the measured accuracy of the model. The model remains “accurate” in the narrow sense of predicting what will happen. This is because it is now partly causing what will happen even as it causes harm!
There is a second problem that we need to address: Mortality prediction models do not predict mortality directly. They predict mortality as it was recorded in the data they were trained on. This means that they predict the outcomes that accrued to the kinds of patients who were treated the way those patients were treated, in the institutions where those patients were treated, at the historical moment when the data was collected. When the training data reflects a healthcare system that did not treat all patients equally, the model learns those inequalities as facts about the patients rather than facts about the system. A classic example of this is a risk stratification algorithm used to allocate resources in care management. The algorithm exhibited significant racial bias ie.g., at a given risk score, Black patients were considerably sicker than White patients. The algorithm used healthcare cost as a proxy for health need. But unequal access to care means that less money is spent on Black patients than on White patients who are equally sick. The model correctly learned the relationship between race and cost. It incorrectly treated that relationship as a relationship between race and health. In the study the researchers concluded that if we fix this disparity, percentage of Black patients receiving additional help increases from 17 to 47 percent.
Mortality prediction models may face the same problem. When a model learns that certain demographic groups have historically had worse outcomes in an ICU, it does not know whether those outcomes reflect biological differences, disparities in care quality, implicit bias in clinical decision-making, structural inequities in access to treatment, or some combination of all of these. It learns the outcome. It assigns the risk. And then clinicians, seeing a high mortality risk score for a patient, may be more likely to move toward comfort care and away from aggressive intervention. We may unintentionally end up in a scenario where we deny a patient care they might have benefited from. The above discussion should not be taken to mean that mortality prediction models are simply harmful and should not exist. This would be wrong, both empirically and practically. These tools exist because clinical need is real, the limitations of unaided human judgment, and the genuine benefits of well-designed decision support systems.
Every mortality prediction model contains, embedded in its architecture, a threshold: a score above which a patient is classified as “high risk” and below which they are not. This threshold looks like a technical parameter. It is set by optimizing a tradeoff between catching the patients who will die and not over-alarming clinicians about the patients who will not. But it is not purely a technical parameter. It is a value judgment about what kind of errors matter more. Setting the threshold low i.e., catching more dying patients at the cost of more false alarms, implies that missing a deteriorating patient is the greater harm. Setting it high implies that alarm fatigue and the costs of over-treating are the greater harm. A patient who has clearly expressed that they do not want aggressive intervention at the end of life is poorly served by a model whose threshold was set to minimize missed deteriorations. A patient with a serious but treatable condition who receives high-intensity care and survives is well served by the same threshold. The model does not know the difference. It may apply the same logic to both.
I believe that the problem is not the AI the tools themselves, but rather the epistemological confidence with which they are sometimes deployed. Thus, when a clinician sees a 78% mortality probability attached to a patient, that number carries the authority of algorithmic certainty in a way that a clinical impression does not. The clinicians may doubt their ‘s own judgment, in deference to the model. This is what the literature calls automation bias i.e., the tendency to over-weight algorithmic outputs relative to independent human assessment. It should be noted that the self-fulfilling prophecy in clinical AI is not only technical but also cognitive. The model shapes what the clinician sees and then they act on what they see. The patient’s outcome reflects what the clinician did. The model’s prediction is confirmed.
I am not arguing that we should not use these tools. I am arguing that we should use them the way we use any instrument that operates near the boundary of what it can reliably know i.e., with care, with skepticism, and with a clear-eyed understanding of where they tend to fail. A model trained on ICU patients at a large academic medical center does not automatically transfer to a community hospital in rural Mississippi or a safety-net hospital serving a majority-immigrant population. We would not deploy a drug approved in one population without asking whether it works in another; we should not deploy a prediction model without asking the same question. And when a model performs significantly worse for Black or Hispanic patients, that is not a footnote. It is material information that should be visible at the point of care, with the same ease as the score itself. When a model states that the risk of mortality is 74%, what it actually means is something closer to: we think this patient’s risk falls somewhere in this range, based on patterns in historical data that may or may not apply here. It is a more truthful statement, and it leaves room for the clinician’s own judgment to operate.
Having that room to navigate is critical. Somewhere along the way, through time pressure, cognitive load, and the accumulated authority that numbers acquire when they appear on official screens, it becomes possible for the score to quietly replace the patient in a clinician’s working attention. Not through any single decision, just through the gradual substitution of the legible for the complex, the measurable for the irreducible, the profile for the person. The deepest requirement here is not technical but rather it is a commitment. It is institutional, cultural, finally personal commitment to the idea that the work of medicine is not to manage a distribution but to care for an individual. The mortality score is a tool in that work. It is not a substitute for it.