by Ali Minai
Artificial intelligence – AI – is hot right now, and its hottest part may be fear of the risks it poses. Discussion of these ricks has grown exponentially in recent months, much of it centered around the threat of existential risk, i.e., the risk that AI would, in the foreseeable future, supersede humanity, leading to the extinction or enslavement of humans. This apocalyptic, science fiction-like notion has had a committed constituency for a long time – epitomized in the work of researchers like Eliezer Yudkowski, Nick Bostrom, Steve Omohundro, Max Tegmark, Stuart Russell, and several others. Yudkowsky, in particular, has been a vocal proselytizer for the issue of existential AI risk. This might have remained a niche issue but the emergence of ChatGPT and other extremely large artificial intelligence (AI) models in late 2022 has made it both more mainstream and more urgent. A major factor in this is that some of the most important pioneers in the area, such as Geoff Hinton and Yoshua Bengio, have expressed great alarm. Hinton, whose pioneering work on neural network learning is at the core of today’s big AI systems, is quoted as saying: “My intuition is: we’re toast. This is the actual end of history.” Understandably, such statements have elicited skepticism from many others such as Yann Le Cun, who see AI as promising great benefits to humanity. The problem is that both groups are likely right, and we have no way of knowing who is more correct. Though various people have thrown probabilities around, there is no way to credibly estimate the probability of an event that has never happened.
Most of the debate outlined above is focused on risks posed by artificial general intelligence (AGI), which refers – approximately – to the kind of versatile, flexible, and autonomous intelligence seen in humans. The argument of those raising the alarm is that such intelligence, if it were to be achieved, would necessarily entail capabilities in the machine that would make it very dangerous to humans. This is an interesting and vast topic with philosophical, psychological, and engineering dimensions. It will be treated separately in the second part of this two-part series of articles. The present article, i.e., Part I, will attempt to lay out a principled framework for characterizing the large range of risks posed by powerful AI, and briefly discussing those that stem from sources other than the very nature of AI.
Classes of Risk
The fact that AI might pose risks has been realized and discussed ever since the idea was born. Indeed, premonitions of it can be found going back centuries to the legend of the golem and Frankenstein’s monster. As AI has moved gradually from toy models to real-world applications, the discussion of risks has also become more detailed and ramified. As mentioned earlier, much of the high-profile debate has been about existential risks, but a large number of more operational risks have also been discussed, including criminal misuse, toxic biases, generation of misinformation, catastrophic job losses, and autonomous munitions. However, most of these discussions are ad-hoc and mix risks stemming from different sources. A good summary of these is provided in a recent paper by several leading researchers and thinkers, including Yoshua Bengio, Geoff Hinton, Yuval Noah Harari, and Daniel Kahneman. A key paragraph from that states:
“AI systems could rapidly come to outperform humans in an increasing number of tasks. If such systems are not carefully designed and deployed, they pose a range of societal-scale risks. They threaten to amplify social injustice, erode social stability, and weaken our shared understanding of reality that is foundational to society. They could also enable large-scale criminal or terrorist activities. Especially in the hands of a few powerful actors, AI could cement or exacerbate global inequities, or facilitate automated warfare, customized mass manipulation, and pervasive surveillance.”
As a first step towards a more systematic understanding of AI risks, it is useful to group these and other risks into clear categories – each with its own sources, and therefore, its own unique ways of mitigation. I propose that AI risks be divided into the following classes:
- Model Risks: The risks that result from how AI is currently built and used.
- Application Risks: Risks arising from the use of AI as intended.
- Malicious Use Risks: Risks that rise from the use of AI for illicit or inappropriate purposes.
- Psychological Risks: Risks that emerge from the interaction between the human psyche and AI.
- Essential Risks: Risks that are inherent in AI because of its nature.
This article with focus on the first four categories, which may be termed operational risks, and the last – most complex – category of essential risks will be the topic of Part II in this series.
Model Risks:
These risks arise from the way AI systems are configured, trained, accessed, and used. A non-exhaustive list would include the following:
Effect of Data Bias: AI systems today use algorithms to discover and learn patterns in large datasets. Thus, any visible or latent biases within this data become part of the learned behavior of the AI system. While not all biases are bad – in fact, intelligence itself depends on biases in the form of heuristics – some learned biases may render the system harmful, unsafe, or toxic.
Centralization Risk: When a large number of applications are built using a single cloud-based AI platform such as ChatGPT, any malfunction in that platform can cause widespread failures. Of course, this risk is not specific to AI systems alone.
Opacity Risk: Applications build using APIs (application program interfaces) to centralized cloud-based systems such as ChatGPT do not have access to the internals of the system, creating the risk of inexplicable faults in performance, This risk too is not specific to AI.
Algorithmic Risk: In some cases, the logic underlying the AI system can induce an irreducible risky effect, such as making false inferences. For example, the use of an autoregressive stochastic process to generate text in large language models (LLMs) makes them inherently prone to confabulation), i.e., the generation of factually incorrect statements – often termed hallucinations.
Environmental Risk: The most powerful AI systems today use an enormous amount of energy. If AI becomes as pervasive as is being imagined, the energy resources it will require with current technology will pose an enormous environment and economic risk. Of course, it is possible – even likely – that much more efficient technologies will be developed to mitigate that risk.
Some, but not all, model risks can be addressed with careful data curation, testing, good design, and detailed documentation. In any case, they will change as AI technology advances and, as such, are of limited significance at this time.
Application Risks:
This is a vast class of risks that arise from the use of AI systems as intended. To a large extent, these risks are a consequence of automation, i.e., handing over decisions made by humans to machines. However, the nature of the risk itself can vary a great deal. Three important examples can illustrate this:
Dangerous Behavior by Self-Driving Vehicles: As attempts at building fully-autonomous self-driving (FSD) vehicles have shown, this task is far more complicated than engineers had thought, and poses significant physical risk to humans in all but the most carefully controlled situations. This risk arises from the interaction between the sensory and motor control systems of the vehicle and the uncertainty of its environment, and must be distinguished from the essential risk of misaligned values that is present in all truly autonomous systems.
Autonomous Munitions: While robot armies have been a staple of sci-fi for decades, advances in AI, robotics, and autonomous vehicles are bringing the possibility of autonomous “smart” munitions ever closer, raising alarms in the scientific community. Such weapons pose an array of risks, including greater lethality, sanitization of violence, and increased disparity in power between the haves and have-nots of these technologies.
AI-Driven Escalation: As AI becomes more embedded in systems such as power grids, transportation systems, and even defense systems such as nuclear weapon systems, the risk of unexpected, arbitrarily dangerous scenarios can increase. This is, in part, because the benefit of using AI in these systems is to let it optimize decision-making beyond what humans can handle, thus ceding control to AI. In some situations, the AI systems might escalate their responses to levels that, while mathematically optimal, are extremely hazardous to humans.
Massive Economic Dislocation: The risk that AI will automate away entire employment sectors has loomed large in recent debates in AI. In a sense, this is not a new problem: Automation has been reducing the need for physical labor – even quite skilled physical labor – for a long time. The threat this time is more widespread because, in addition to covering skilled physical labor jobs such as truck driving, it also targets white-collar, “cognitive” job categories such as lawyers, managers, teachers, physicians, engineers, and scientists, as well as creative categories such as writers, actors, composers, and artists. It is not clear that the feared automation will reduce quality – except possibly in the creative areas – but that the entire economic system will change in radical ways. A further complication is that the automation is likely to happen at very disparate rates in the richer and poorer parts of the world, creating great risk of exploitation and conflict. However, it is important also to keep in mind that, in the past, technologies that automated some classes of jobs have always created whole new and unforeseen classes of jobs in greater numbers, though the process is still quite disruptive.
These, and other risks like them, cannot be addressed sufficiently by purely technological solutions, and will require policy choices and regulation at the highest levels for any chance of mitigation. Technological advances can then work as the instruments of these policies and regulations.
Malicious Use Risks:
The risks created by the myriad potential misuses of AI have probably been the most widely discussed and acknowledged class of risks. Of course, all technology poses risks as well as benefits, but the general nature of AI means that its primary risk lies in the ability it gives human users to amplify existing risks – including those of other technologies. As with application risks, it is impossible list all risks arising from the misuse of AI, but a few illustrations will suffice:
Generation and Spread of Misinformation: Perhaps the most readily accessible and understandable risk misused AI is the generation and dissemination of false information. Misinformation that seeks to shape opinion by playing on ignorance and human biases has been part of the human experience throughout history. However, its use required either a great deal of effort on the part of its user or significant gullibility in its targets, or both. With the advent of generative AI, both obstacles have been removed. It is now possible to generate extremely convincing photographs, video, and speech very easily and with a quality that can make it indistinguishable for the real thing. And, while, in the past, the misinformation spread slowly through small, weakly-connected social networks constrained by physical time and space, it now spreads at lightning speed on immense, highly-connected, virtual social networks in cyberspace. This could greatly “weaken our shared understanding of reality”, with catastrophic consequences.
Use of AI in Cybercrime: There is growing concern that AI systems such as LLMs and image generators are making cybercrime easier by increasing the sophistication and reducing the detectability of online scams. Whereas creating the documents, images, malware, viruses, etc., that these scams require once took a great deal of effort, skill, and knowledge, generative AI makes things much easier. At the same time, it has been noted that AI can be a powerful tool to thwart AI-powered cybercrime and even for terrorism. Thus, the real risk that AI has added is to take the eternal battle between criminals and crime-fighters to a new level.
Using AI to Build Harmful Artifacts: In addition to helping generate harmful virtual items such as computer viruses, malware, and fake images, AI can also help malicious actors discover, design, build, and deploy dangerous physical artifacts such as viruses, bioweapon agents, intelligent munitions, etc. And, as with cybercrime, AI can also be an important part of fighting these threats, again simply taking the conflict to a new, more dangerous level.
Big Brother: While many governments are increasingly using AI tools such as face recognition and predictive policing for legitimate surveillance, the same technology can easily be applied by authoritarian governments, corporations, or rogue agents for illicit spying, invasion of privacy, crime, oppression, and targeted violence. In other words, legitimate surveillance can easily turn into Big Brother from 1984, even in well-meaning hands. This danger will become especially acute as the decision-making in surveillance systems too is handed over to AI in the name of efficiency – a path that, in the extreme case, could lead to a Skynet-like system.
It is worth noting that all these risks are actually created by human behavior, not by AI per se. AI just provides a tool of previously unimaginable sophistication to amplify bad human behavior. In virtually every case in this risk category, it may only be through the use of AI that the risks created by AI can be met. Since AI is, by construction, capable of adaptation and learning, each situation is likely to result in an escalatory ladder as more and more powerful “Good AI” battles increasingly sophisticated “Bad AI” – a scenario familiar to us today in the perpetual war between computer virus-makers and cybersecurity systems. AI is going to supercharge this, and the increasing virtualization of everything in modern society will spread the risk to every aspect of life. Mitigating this risk will require strong global regulation and law enforcement, as well as perpetual vigilance – none of which seem feasible in today’s divided world.
Psychological Risks:
One of the most interesting – and most neglected – classes of AI risks is that of risks arising from the interaction between the human psyche and truly intelligent machines. It is worth separating them out into their own category because they promise to reshape society itself in ways that policies and regulation cannot control. Indeed, in time, some of them may come not to be seen as risks at all but just the way we are.
Illusion of Objectivity: One proximate psychological hazard of AI-based decision-making is that it might be seen as inherently more “objective” than human decision-making simply because the decision is perceived to be the result of dispassionate calculation. This is dangerous for two reasons. The first is that, as discussed above, AI systems often inherit the biases present in their training data, and reflect these in their decisions. The second, more subtle, issue is that not all human biases are bad; some are virtuous and necessary. Good human decision-making is not based only on cold calculation, but also on empathy, a sense of fairness, kindness, historical context, social context, and other such “biases” that may run counter to the decision that is objectively optimal. An AI system that does not use such judgment may make an objective decision, but not a humane one. The danger lies in the possibility that the perceived objectivity of the AI’s decision may be used to impose inhumane decisions by declaring them inherently fair. This issue, in fact, connects with the deeper one of whether AI can be trusted to follow human values. That will be discussed in the next article.
Illusion of Humanness: As AI systems become more sophisticated, the tendency to anthropomorphize them grows stronger. Users of ChatGPT, for example, can easily feel that they are conversing with a human interlocutor. While this tendency to think of machines as human-like is not new – we do it with ships, cars and other machines with goal-directed behaviors – it is especially tricky in a system that engages with our minds and psyche. In the 1960s, Joseph Weizenbaum – a professor at MIT – created a program called ELIZA that “conversed” with humans through a computer terminal. Though ELIZA just used very simple scripts and a few psychological tricks to say plausible things, it induced some users to treat it is a sympathetic human therapist and to share extremely personal thoughts with it. Today’s LLMs are vastly more sophisticated, and far more likely to induce false trust. Not only could this be exploited by malicious actors as already discussed, it may also lead humans to form false expectations or make bad choices, resulting in dangerous situations. Indeed, there has already been a reported case where a chatbot called Chai, which is based on the GPT-4 LLM, convinced a person to commit suicide. As with illusion of objectivity, the illusion of humanness devolves ultimately to the issue of trust in AI.
Cognitive Outsourcing: A profound mental risk that AI poses to humans is that of cognitive outsourcing, i.e., offloading cognitive tasks to AI. Humans have, of course, been outsourcing physical labor to tools, machines, and animals from time immemorial. Several decades ago, the important cognitive task of calculation was outsourced to electronic calculators, and this has expanded gradually to include things like proofreading and symbolic mathematics. However, these are still specialized tasks. Two much more profound instances of cognitive outsourcing have occurred more recently. The first is geographical outsourcing to global positioning systems (GPS), surely reducing the native ability of individuals to situate themselves geographically and navigate intelligently. Fully self-driving cars will exacerbate this further. The tradeoff, of course, is convenience and far greater ability to navigate anywhere in the world. The other example is epistemic outsourcing, which has transferred the task of retaining factual knowledge in the brain to electronic devices and search engines. I wrote an article about this on 3QD three years ago, arguing that the impoverishing of the mental episteme posed a real risk to the capacity for individual creativity, which was being traded off for ready access to a far greater amount of knowledge. This issue has also been discussed by psychologists for several years. In all these instances, we can see that while the capabilities of the (human + machine) system are far greater than those of the pre-machine human, the post-machine human’s cognitive abilities are diminished to some degree. This may not, in itself, be a bad thing at all, just as being able to drive instead of having to walk or ride a horse is not a bad thing, but it does raise the specter of dependency. This dependency arises not only because we are no longer using brain or brawn, but also because the enhanced power of the {human + machine) system makes feasible new situations that cannot be handled without the machine. For example, the invention of motorized transport allowed cities to become much larger, making people dependent on motorized transportation, and increasing their risk in situations where that transportation fails. The question is how the new, extremely powerful AI plays into this process.
The potential for cognitive outsourcing exists in all recent developments in AI, but the advent of large language models such as GPT-4 and Llama 2 is orders of magnitude more important because language and symbol manipulation has such a central role in human higher cognition. Though the necessity of language for thought is still a much-debated issue, there is no doubt that language is central to the conceptual organization, composition, and expression of thought, as well as many modes of creativity from writing standard documents to composing poetry. LLMs represent the first time in human history that these tasks – and many others like them – can be outsourced to machines. They are very good at some of these, such as summarizing documents, extracting topics, translating between certain languages, and inferring the logical flow of a written argument, and can be trained to do well on tasks such as medical diagnosis. On other tasks, such as generating plans to solve complex problems, generating plausible scientific hypotheses, and writing truly creative literary works, they are severely limited by their lack of a causal model of the real world and their extremely simplistic generative process. But these limitations will eventually go away – next year, or in five years or ten – and AI will extend to even more profound things such as engineering design, scientific research, and making legal arguments. At that point, a vast swath of what we humans use our minds for will have been automated and outsourced away, with computers doing all these tasks better than humans. A time can be foreseen when people – at least in developed societies – will not be able to function cognitively outside a cocoon of AI, but will be hyper-capable when embedded in it. Some may argue that this is no different than our inability to function without electricity, telecommunications, or transportations, but, in fact, it is qualitatively different. In all those other situations, we may have lost physical supports but are still mentally autonomous and whole. As more of our core cognitive functions are outsourced to AI – as they will surely be – it will reduce our mental autonomy as humans and force us to live in symbiosis with our machines in the profoundest sense. Some AI imagineers – seeing the possibilities of this enhanced existence – have been waiting eagerly for this “singularity” to arrive, but the process is likely to be extremely hazardous, fraught with possibilities of delusion, dehumanization, loss of individuality, and oppressive control through denial of access to mind-sustaining AI.
Though the extreme scenario of the singularity may not emerge soon, profound levels of cognitive dependency will certainly do so in the very near future. AI does not need to become AGI, or even fully autonomous, to create such dependency, and all the issues of failure, unequal access, and cognitive hacking that this might create. Misinformation passed on smart phones already has the world in turmoil; manipulation of AI to which we have outsourced our cognitive functions will be at another level altogether. At the same time, AI will also help mitigate many existing risks such as poor decision-making due to lack of information. It is not clear where things will settle on balance, but whether for better or for worse, AI is going to change what it means to be human.
The Cloud vs Zootopia
The currently dominant paradigm for cutting-edge AI involves the development of very large systems by a few corporations such as OpenAI and Anthropic, with a vast number of applications connecting to these systems through application program interfaces (APIs). This may be termed the Cloud Model of AI. If this remains the prevailing model as AI is scaled up by orders of magnitude and infuses every aspect of life, it will create an extreme dependency on these corporations and an extremely asymmetric power relationship between them and the general public. In the extreme case, if such centralized, globe-spanning AI becomes truly general and full autonomous, this could even lead to the emergence of a Skynet or Matrix-like scenario. It will also make for a very fragile system, though one where safety and regulation would be relatively easier to implement. At that point, it might make sense to treat AI as a utility and regulate AI corporations like utility companies.
An alternative possibility is that new technologies will allow widespread decentralization of AI, with a lot of companies, small entities and individuals developing AI-based applications, devices and robots using on-board AI or local servers. This can be called the Zootopia Model of AI, and may ultimately come to resemble the kind of world we see in sci-fi such as Star Wars. The decision by Meta to make its LLM, Llama 2 available open-source is a major step towards this future, but the technology for powerful AI would need to become much more accessible, efficient and lightweight for this model to come into its own. While this removes the risks of fragility and asymmetry of power, it amplifies greatly the risk of rogue actors developing dangerous systems and makes regulation extremely difficult.
At this very early stage of application-scale AI, it’s hard to say which way things will go, but the next five years or so will probably make things much clearer.
Existential Risk?
Finally, a word on existential risk – an issue that seems to be of consuming interest to many. Potentially, many of the risks listed above could grow to become existential. Energy-hungry AI systems could tip the world past a critical point in climate change, though this is extremely unlikely. Certainly, the risk associated with nuclear escalation or the development of weapons of mass-destruction by rogue actors would be existential. Modern human societies are moving more and more of their essential functions into cyberspace and will, no doubt, embed AI into them. AI will also become increasingly critical to human life at the individual level, as discussed above. All this means that the capacity for catastrophically dangerous malfunctions, misinformation, hacking, cybercrime, and terrorism will assume existential proportions. All these risks, and others beside, will be amplified by the essential nature of AGI as an autonomous, adaptive, motivated, and self-improving alien intelligence. That is the topic of Part II in this series.