Anthropic trains Claude to mimic safety classifier without consulting policies
Anthropic trains Claude to mimic safety classifier without consulting policies
Claude now silently checks whether its responses may have contributed to harm, while Anthropic distills an expensive safety classifier into a cheaper self-assessment mechanism.
Claude now quietly performs an internal check every time it responds to a user, trying to determine whether its answer may have contributed to harm. The results are sent to Anthropic specialists for further review.
Anthropic’s most accurate tool for detecting abuse is a classifier built on Claude Mythos Preview. Before making a decision, it consults all of the company’s safety policies. However, this procedure is too computationally expensive to run for every conversation.
Anthropic therefore trained Claude to reproduce the classifier’s decisions without consulting the safety policies themselves or repeating the classifier’s reasoning.
When checking requests to Fable 5, the new mechanism detected 78% of the cases flagged by the original classifier. Among high-priority cases, the detection rate reached 94%, while the false-positive rate was only 1%.
Why it matters
AnalysisIt shows how AI labs try to scale safety monitoring without paying the full compute cost of policy-level review. For users, it means more automated risk checks built into model responses.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
Claude now self-assesses potentially harmful requests
Anthropic says Claude receives a hidden instruction after each response to briefly evaluate whether the conversation contained a potentially harmful request.
Chinese forum users claim to have decoded Claude account ban scoring system
Users allege Anthropic assigns initial points to new Claude accounts and deducts them for changing IPs, devices or proxies, subscribing to Max too early, or using only the web version.
Computer Use via AI
Anthropic's technology (introduced in Claude 3.5 Sonnet) enables neural networks to control operating systems like a human: taking screenshots, analyzing window coordinates, moving the mouse cursor, clicking buttons, and entering text without using specialized APIs.
Red Teaming in AI
A methodology for aggressive stress-testing AI models by specialized ethical research teams (Red Teams). Experts simulate malicious actions, identifying security gaps, jailbreaks, and prohibited responses before the model reaches millions of users.