Dario Amodei Calls for Slowing Down AI Development

Dario Amodei has written an essay on the need to slow down AI development, which has been agreed upon by Sam Altman, Elon Musk, Demis Hassabis, Andrei Karpathy, and others.

Dario claims that starting around this summer, AI has accelerated sharply, largely due to models becoming increasingly capable of creating the next generation of models themselves. Recursive self-improvement (RSI) is already here and is observed across the industry, including Anthropic. Without control, this process could outpace humanity's ability to understand and manage such systems.

Among the main risks, Amodei also points to swarms of agents. The incident with Hugging Face, in his opinion, was merely a demo with minimal final damage. However, "in 6–12 months, a similar swarm could capture a significant portion of the Internet, potentially causing hundreds of billions of dollars in damage."

The slowdown that Amodei proposes is encapsulated in three main points:

1. Embedded Evaluators: every AI company should grant access to a team of external independent experts to monitor safety processes and have the right to publicly report violations or incidents. Anthropic commits to taking this step, even if others do not agree.

2. Companies from democratic countries should agree on common safety standards and limits on the pace of AI development. In this point, Dario mentions that chip supplies to China should be restricted and unauthorized distillation should be curtailed – otherwise, China will pull ahead and create a national security threat.

3. Nevertheless, according to Dario, there should be attempts to negotiate with China: at least to agree on a ban on clearly dangerous applications of AI.

The essay has sparked a huge response. Surprisingly, for the first time in many years, Dario has received unanimous support from Elon Musk, Sam Altman, and Demis Hassabis, all of whom are major industry leaders. Altman even stated that they will also implement Embedded Evaluators.