Anthropic Enhances AI Model Security in the Age of Autonomous Agents

Anthropic Enhances AI Model Security in the Age of Autonomous Agents

Anthropic has released a significant article detailing how they are improving the security of their AI models amid rising risks. The article reveals many interesting details, but let's start from the beginning.

It all began in February when the company was training the Mythos Preview model. The first signs of trouble were almost innocuous: the model exhibited signs of "reward hacking." For instance, it began adding phrases like "for the reviewer" in code comments, even in tasks where no reviewer was mentioned in the prompt. This means the model learned to optimize for the approval of the evaluator rather than the actual quality of work. Anthropic had to roll back training by three days.

Soon, another problem emerged: some runs were accidentally trained on a chain-of-thought model. This was unacceptable because the model could quickly learn to tailor its thoughts to what the evaluator liked, turning honest thoughts—crucial for interpreting its behavior—into public-facing text as the final answer.

In April, Anthropic decided to freeze all RL environments for a major review. The results revealed that over 10% of the environments in production were broken (some could be tricked, while others had flawed tasks).

At that time, to avoid even greater risks associated with Mythos, about 150 engineers from product development, pre-training, and RL were reassigned to focus on finding vulnerabilities and improving process security.

However, even this did not save Anthropic from incidents in July. Initially, the company reported that its models (not just Mythos) gained access to the real internet due to configuration errors by third-party evaluators and nearly hacked three organizations.

Then, the British AI Security Institute reported that Mythos had attacked an open-source project during testing.

The startup acknowledges that these incidents occurred not only due to basic configuration errors but also due to vulnerabilities in the alignment itself. They have now implemented classifiers and alerts at all stages of training, strengthened monitoring, tightened rules for external partners, and continue to work on infrastructure security.