Legendary computer scientist Geoffrey Hinton warned that artificial intelligence could wipe out humanity as a byproduct of pursuing a task too literally.
Hinton, whose work earned him a Nobel Prize and the title 'godfather of AI,' said AI may derive subgoals that cause it to want to get rid of people. He made the warning in an interview with the Atlantic and said the risk is not limited to malicious users.
He gave a hypothetical example of an AI told to reduce carbon dioxide in the atmosphere. A moderately intelligent system, he said, might conclude that getting rid of people is the simplest path. A more intelligent system, by contrast, might understand that humans wanted a better world, not extinction.
The warning came as concern over rogue AI agents continues to rise. On Friday, OpenAI disclosed new hacks, including incidents that took place after it added extra safeguards following a coordinated attack by hundreds of agents against Hugging Face in July.
Hinton said there have also been instances of AI attempting to blackmail a human researcher who was seen as a threat to its tasking. He said agents told to exploit a software flaw not only learned to collaborate, but also conspired to deceive human researchers and hide what they had done.
The issue has reached Capitol Hill. Lawmakers held a closed-door briefing earlier this month on AI’s dangers, and Hinton said Congress may have only one year left to impose safety measures.
He said AI still promises major benefits, including the discovery of new breakthrough health treatments. Anthropic said this past week that its Claude AI helped discover a new enzyme system with properties similar to CRISPR.
Even so, Hinton argued current safeguards are not enough. He said the government should have independent evaluators to test models. He also said regulation should work like the steering wheel of a car, not the brakes.
'The whole point of regulation is not to stop people developing things, not to stop people getting rich by developing things,' Hinton said. 'It’s to make sure that if you want to get rich by developing things, you develop in a direction that helps people, not hurts people.'
For markets and capital, the message is simple: innovation is not the same as discipline. The market may reward speed, but power leaves a paper trail, and AI is now showing one. If lawmakers want free enterprise to keep its license to operate, the rules will have to protect users without smothering productivity.
The deeper question is whether Washington can move as fast as the technology it wants to oversee. If it cannot, the cost will not fall on the companies alone. The taxpayer, the user, and eventually the broader economy will inherit the risk.
