Physicists built a benchmark of 50 unpublished theoretical physics problems
A new benchmark of 50 unpublished theoretical physics problems was solved almost completely by Astra, while researchers found errors in their own checks.
Physicists built a benchmark of 50 unpublished theoretical physics problems
The AI model Astra solved all the problems except one. The authors acknowledged that the wording of that particular problem was ambiguous: none of the other models solved it either.
The problems were created by domain experts and selected so that a specialist could solve each one in roughly one to two days.
Results
- Astra: 49 out of 50 problems.
- Opus 5.5: 46 out of 50 problems.
- GLM-5.3: 21 out of 50 problems.
All answers were reduced to numbers and checked automatically. After the test run, the evaluation indicated that the model had solved several problems incorrectly, but Astra insisted that its calculations were correct. The researchers rechecked their own solutions, found errors in them, and the model’s answers turned out to be right.
No conclusion will be drawn.
More details: arXiv:2610.16733.
Why it matters
AnalysisThe case shows that AI models can challenge expert validation pipelines and may expose human errors in benchmark scoring, raising the bar for reliable scientific evaluation.
Discuss in community
Share your questions and insights with developers