A theme of all these books is that ASI would not need to hate us in order to destroy us. Even if its goal were to bake the perfect Victoria sponge, it might decide to wipe out all of humanity just in case one of us was tempted to turn the oven off early. We may hope that it would not do such a thing to us, its makers, but instead regard us with a sense of affection and filial obligation. But that is to project on to it those un-programmable human sensibilities. And anyway, if we discovered that we were created by bacteria, would we be nicer to them? Probably not much.
Perhaps more worrying than the difficulties of creating friendly machines is that most AI developers are not even trying to or, indeed, are striving for the opposite. As Barrat points out, most of the research is sponsored by business and designed to do things such as make money on the stock market — well over half of all Wall Street’s equity trades are already made by automated systems. The other main sources of funding are defence agencies. Darpa, the US defence department’s research body, has long been a major sponsor of AI. According to Barrat, alongside the US, at least a further 55 countries are “developing robots for the battlefield”. In other words, the serious money is going into AI designed to be decidedly unfriendly — AI that is, in fact, designed to kill humans. What could possibly go wrong?
more here.