by Malcolm Murray
In a recent study we conducted on AI risk estimation, something potentially interesting about the nature of LLMs emerged as a side effect. In the risk assessment studies we typically run, a group of experts are asked to provide estimates of various parameters and how they vary with different levels of use of AI. These parameters can then be stitched together to form a risk model that estimates how much risk increases when AI is introduced into the picture. A classic example is LLM-enabled cyber risk, i.e. how much AI is helping cybercriminals to conduct more sophisticated and successful attacks. This has received additional attention recently given the announcement of Mythos.
In this recent study, we experimented with also using LLMs as experts, given the paucity of human experts with expertise at the intersection of AI and specific risk domains. This worked very well for creating more risk models at greater scale, and the risk models appeared to be of similar quality to the human-created models. However, interestingly, as a side-effect, this also yielded some potential insights into how LLMs function differently from humans. In creating their estimates, LLMs seem to have their own biases, distinct from human ones. This is something that should be studied further.
In the future, many, if not most, professions will likely have some kind of combination of human and LLMs in the mix. Some might become fully automated and handed over to LLMs, and some might remain the province of humans exclusively. The vast majority of jobs, however, will likely see some combination of the two. As Noah Smith has argued, there will likely be room for humans to continue to be competitive in the labor market given our comparative advantages. As Ricardo’s theory of comparative advantage suggests, even an actor that is worse at everything can still specialize productively if constraints and opportunity costs exist. Humans may therefore remain economically relevant even if AI surpasses us across most dimensions, as long as there are opportunity costs to allocating the next incremental piece of compute to AI instead of humans, and the marginal cost of compute is non-zero.
For this human-AI collaboration to work well, however, it will be key that the division of labor plays to the strengths of both sides. This brings us to the question of biases. Humans are commonly considered to have a range of biases, such as our tendency to ascribe good deeds to someone we already hold in high esteem (the halo effect) or our tendency to disproportionately overweight things that more easily come to mind (the recency effect). We don’t know if these are bugs in the human operating system or necessary features from our evolution, but we are now in any case stuck with them.
When it comes to AI, we have traditionally thought of it as a completely objective actor. Leaving a decision to the machine would typically be seen as a way to reduce any sneaky biases in the decision, making it more objective. There were cracks in the armor already with earlier versions of AI in the 2010s, as famously seen in court and hiring decisions. Since the systems had been trained on human data, with all its shortcomings, the machines propagated those same errors.
Recent studies such as the one I referenced above are however suggesting that LLMs might be not just propagating human biases, but perhaps even developing their own idiosyncratic biases. Our study served to estimate the increase in risk as AI capabilities grow. Both human experts and AI experts were asked to estimate the same parameters, for the same risk scenario. Some of the results can be seen in figure 1 above and figure 2 below.
In figure 1, we see growth in capabilities captured on the X and Z axes, in terms of higher performance on two AI benchmarks, and the resulting increase in risk on the Y-axis, with human experts on the left and LLM estimators on the right. The X and Z axes show tasks of increasing difficulty radiating outward from the baseline in the middle. The two graphs show distinctly different patterns. In the left (human) chart, we see risk growing smoothly away from the baseline in the middle. With each additional task that the AI can perform, the risk increases, and it does so in a smooth and continuous fashion. On the right, however, in the LLM estimator chart, we see a much more spiky pattern. Risk goes up and down even as performance nominally increases; it plateaus and then increases again. The interesting thing is that, since this is forward-looking, we don’t know which of them are right or wrong. It seems likely that humans are over-imposing a pattern, i.e. are forcing an increase in risk with progressively more difficult tasks, even when not warranted. That would be a common human bias of seeing patterns where there are none – apophenia (such as the “hot hand” in basketball). On the other hand, it seems unlikely that the LLMs are correctly estimating the risk either. While it is commendable that they consider each task in isolation and don’t fall prey to imposing an overall pattern, some of these tasks are demonstrably more difficult than others and we should see more differentiation in the risk levels than we do. So both humans and LLMs may be flawed, but in different ways. A combination of the two, a centaur as it was called in chess, might be the best of both worlds. OpenAI’s recent model disproving an Erdős theorem, where the LLM did things humans just wouldn’t do, is a good example.
Another potential bias is at play in figure 2. Here we again have human experts on the left, LLM estimators on the right, and total risk on the Y-axis. The X-axis is here showing two bars for human estimators and two for LLM estimators, one for “SOTA” – current state of the art level of AI performance, and one for “Saturated” – when the LLMs have saturated the benchmarks, i.e. can complete every task in them. Again, we see two distinctly different patterns. When estimating the risk increase, the humans swing for the fences, and estimate a manyfold increase in risk from today’s levels. The LLMs are much more conservative and forecast a much smaller increase. Again, we don’t know who is right or wrong here. The LLMs might be correctly conservative, or they may be uncomfortable extrapolating so far out from today. This is something I notice in my work as a superforecaster also. When I present an LLM with a similar forecasting problem that I’m working on, it is often markedly more conservative than I am. On the other hand, what I believe is going on with the humans is that they fall prey to the fallacy of what I call “envisionability”. Basically, if we can envision something, we assume that it must have a nonzero probability. As a cyber expert, you are deep in the weeds and it is easy to picture in your mind how AI will help cybercriminals with every step of an attack. That combination of many small increases quickly compounds to a huge overall increase in risk, which may or may not be realistic. The LLMs may not be prone to the same fallacy.
In sum, therefore, it seems there might be a high likelihood that we will get to human-LLM centaurs on some professional tasks. In order to enable this, it would be beneficial to ascertain where the two complement each other. The past decades have thanks to Kahneman, Tversky and Thaler (and their many successors) unearthed a cornucopia of bugs in the human operating system in the form of biases and fallacies. Some of those biases might be replicated in the LLM training data, but it seems there could also be other, distinct biases. Hopefully, with the rapidly increasing focus on model behavior, we might get a Kahneman for LLMs that can help us understand them better.
Enjoying the content on 3QD? Help keep us going by donating now.