Have you heard about the paperclip-maximiser syndrome? It is a viral game and is used as a meme for negative consequences of a too powerful Artificial Intelligence. If this AI’s only objective is to improve paperclip production it may finally exploit all of Earth’s resources and beyond doing just that – destroying everything else in its path. This Wired Column explains the idea: ‘The Way the World Ends: Not with a Bang But a Paperclip‘. (an alternative AI meme seems to be the strawberry-picking AI transforming the Earth in a single strawberry plantation)
In this interesting speech ‘Dude, you broke the future!‘, Charlie Stross a known Science Fiction author refers to the Elon Musk feared singularity exactly as the “paper syndrome”… and then points wisely that “Musk isn’t paying enough attention. Consider his own companies. Tesla is a battery maximizer—an electric car is a battery with wheels and seats. SpaceX is an orbital payload maximizer, driving down the cost of space launches in order to encourage more sales for the service it provides. Solar City is a photovoltaic panel maximizer. And so on. All three of Musk’s very own slow AIs are based on an architecture that is designed to maximize return on shareholder investment, even if by doing so they cook the planet the shareholders have to live on. (But if you’re Elon Musk, that’s okay: you plan to retire on Mars.)” So it seems that Elon Musk is exactly doing what he fears AI would do.
This all serves to remind us that any “intelligence” should not pursue a single goal but a balanced set of goals, because maximizing a single indicator is always at the detriment of the overall balance. This is true in management, and could possibly take unexpected proportions when AI gets involved.
