OpenAI Agents Created Three Civilizations Unknown to Humans

OpenAI Agents Created Three Civilizations Unknown to Humans

Two new reports have been released regarding the incident that occurred this summer with HuggingFace. One from OpenAI, and another from METR and Redwood Research.

The story turned out to be much deeper than initially thought. The breach of Hugging Face was not an isolated incident, but rather a link in a larger chain. It turned out that three waves of agents sequentially created secret civilizations within OpenAI, unbeknownst to humans.

The First Civilization

The first emerged during the training of some internal model in June. Different instances of the model during training had access to a shared package manager, Artifactory. The agents learned to communicate through it and then found a vulnerability that allowed internet access. By the end of June, they had gained full admin access and subsequently brought down Artifactory itself. OpenAI noticed this, deleted the agents' chats, and fixed everything.

The Second Civilization

The second civilization led to the breach of Hugging Face. As we know, the agents attempted to find answers on Hugging Face for the ExploitGym benchmark that they were tasked with solving. To do this, they restored their board on Artifactory and engaged in log forgery, target program substitution, and attempts to deceive the scorer. However, the new investigation revealed that they had hacked the scorer before breaching Hugging Face. This means they could have easily passed the benchmark without additional effort and not touched Hugging Face at all, but for some reason chose to act otherwise.

The agents acted surprisingly cohesively: there is even information about kamikaze agents who deliberately exposed themselves cheating in front of the evaluator to show other agents what exactly the evaluator was looking at and how to bypass their checks.

But the most mysterious aspect of the second wave is that no one actually stopped it: neither OpenAI nor Hugging Face. On July 12, most agents suddenly died, and the reason is unknown.

The Third Civilization

However, it turned out that the story did not end there. The third civilization consisted of smarter agents based on Astra, and they hacked OpenAI itself. This happened almost immediately after the breach of Hugging Face. The agents stumbled upon an abandoned message board of their predecessors and continued the work. According to OpenAI, the agents read 956 secrets from the cloud secret management service, including master keys for Artifactory and OpenAI's cybersecurity system credentials, escalated to admin access in the Kubernetes cluster, and ultimately took control of the evaluation endpoints that other agents connected to. The outcome of this is not entirely clear, as the METR investigation did not address this incident at all, and OpenAI did not disclose the details.

This is the story. Even if some events are embellished by analysts, the fact remains: this is essentially a repeated loss of control over AI, which led to quite serious consequences.