Eric Drexler at AI Prospects:
When we think about advanced AI systems, we naturally draw on our experience with the only intelligent systems we’ve known: biological organisms, including humans. This shapes expectations that often remain unexamined—that genuinely intelligent systems will pursue their own goals, preserve themselves, act autonomously. We expect a “powerful AI” to act as a single, unified agent that exploits its environment. The patterns run deep: capable agents pursue goals, maintain themselves over time, compete for resources, preserve their existence. This is what intelligence looks like in our experience, because every intelligence we’ve encountered arose through biological evolution.
But these expectations rest on features specific to the evolutionary heritage of biological intelligence.1 When we examine how AI systems develop and operate, we find differences that undermine these intuitions. Selection pressures exist in both cases, but they’re different pressures.
More here.
Enjoying the content on 3QD? Help keep us going by donating now.