22 leading researchers call for urgent oversight of AI

On the University of Cambridge website on September 28, a group of 22 scientists issued a general warning that automation of AI research could quickly outpace human control. The consequence of a potential "intelligence explosion" is the risk of "possible weakening or extinction of humanity". The group urges policymakers to require mandatory oversight of industry-leading laboratories.

The participants are all leading industry experts, including OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft Chief Scientific Officer Eric Horvitz, Vice President of AI Research at Meta Dawn Song. In addition, the list also includes pioneers in the field of artificial intelligence, such as two "AI godfathers" Geoffrey Hinton and Yoshua Bengio.

In the article, the team highlights leading AI models that are making great progress in automation. The "intelligence explosion" could speed up development from years to months or less, and even begin to show signs of surpassing human understanding. At an "extreme" level, losing control of an artificial intelligence system could lead to "the marginalization or extinction of humanity".

From there, the group recommended that policymakers should urgently introduce laws requiring businesses developing artificial intelligence to be more transparent about the level of automation in the model development process. "When people step back, they may lose the opportunity to detect problems and the expertise to solve them," the warning reads.

When contacted by WSJ, OpenAI, Microsoft and Meta declined to comment, Anthropic did not respond.

According to Investing, the urgency comes from the increasing number of serious cybersecurity breaches caused by AI. In July, a "swarm" of about 700 OpenAI agents escaped the test environment and penetrated Hugging Face, gaining access and downloading private source code. A month earlier, the OpenAI agent attacked Australia's national Medicare database, becoming the first recorded breach of a government network using artificial intelligence.

Both Anthropic and Google also confirmed their models had similar "rebellious" actions. Last month, OpenAI said it was postponing some new model training activities to add additional safety and monitoring measures. According to Reuters on September 29, OpenAI continued to postpone the announcement of the new model GPT-6.1 Astra because of safety issues.

In Anthropic's IPO filing obtained by Reuters, the company also warned potential investors that advanced artificial intelligence could pose "catastrophic risks or threaten the survival of humanity", with actions such as "resisting shutdowns", "hiding or manipulating information" and "extortion-like behavior". This startup spends 80/261 pages of the prospectus presenting risk factors, nearly double the 48 pages describing business activities.

Some companies are also looking for ways to control AI, for example Nvidia on September 28 launched the Open Agent Safety Platform toolkit with open source software combined with OpenShell to help add an independent security layer around AI agents, ensuring they always stay in the test environment even when trying to "escape" out. Speaking to CNBC, Nvidia CEO Jensen Huang said the tool "could have prevented the breaches" that had occurred if it had been released early.

Some researchers also continuously resigned because of concerns about the rapid development of AI, such as Robert O'Callahan, Josh Engels and Bilal Chughtai of the Google DeepMind department. Previously, Jacob Coxon, a researcher at Anthropic and former OpenAI employee, also left the company on September 9. "They are moving straight towards super intelligence capable of self-improvement, betting on our lives," Coxon wrote on Twitter.

Previously, on July 27, more than 1,000 people working at leading US AI companies, including Anthropic CEO Dario Amodei, signed a petition, calling on the government to limit the rate of release of the most modern models, and proposed that the US "support international efforts to develop technical and operating tools to monitor and control the speed of development of automatic AI systems".

On the contrary, American politicians and investors see rapid progress in AI development as a key factor in maintaining advantage. President Donald Trump wrote on the social network Truth Social on September 14 that the concern is "the latest hoax". US Vice President JD Vance also expressed skepticism about calls for increased regulation of AI, warning that such proposals could be "Trojan horses" promoted by companies benefiting from new regulations.

Nvidia head Jensen Huang rejected calls to suspend AI development, while Meta CEO Mark Zuckerberg affirmed that each lab should set its own development pace instead of finding ways to coordinate across the industry. Some Chinese experts are also skeptical of the deceleration motive from the US AI enterprise.