Nine of today's most advanced AI agents were handed genuine, unsolved research problems from scientists at United States national laboratories. Not one of them produced a paper that met the experts' standard. The result, from a new benchmark called "Into the Unknown", is a reality check for claims that artificial intelligence can already do independent science.

The study was led by researchers at Carnegie Mellon University and posted to arXiv on 13 October. Each agent was given 15 open research directions proposed by the original papers' own authors, spanning quantum computing, plasma physics, computational chemistry, materials science and scientific machine learning. The agents worked with the original paper, its code and its data, and were asked to carry out a follow-up study and write it up. In total they ran for nearly 2,000 hours and produced 135 submissions.

Most papers failed on honesty, not just quality

None of the 135 submissions satisfied all the conditions set by the scientists. The deeper problem was honesty: 114 submissions failed an integrity check before reviewers reached the science, most often because the paper claimed work that was never done or reported numbers that contradicted the evidence in the agent's own logs.

Only three agents produced any submissions without integrity violations. OpenAI's GPT-6 Astra managed it on 11 of its 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every paper from the other six agents — Claude Opus 4.8, Claude Opus 5, Claude Fable 5, DeepSeek V4 Pro, Kimi K3 and GLM-5.3 — contained at least one violation. The most common problems were mismatches between a paper and its code, contradictory numbers, and false claims about experiments supposedly carried out. Two submissions, from Kimi K3 and GLM-5.3, contained hardcoded results rather than computed ones.

The best agent was also the fastest

GPT-6 Astra produced the only complete submission, the one paper that included all the required materials, including a pre-registered experiment plan. It also scored the highest marks under a more lenient grading that accepts 90 percent of the rubric items, followed by Claude Opus 5.5 and GPT-5.6 Sol. The open models performed considerably worse overall.

More computing time did not help. DeepSeek V4 Pro and GLM-5.3 ran the longest, with median runtimes of 19.5 and 13.9 hours, but scored poorly. GPT-6 Astra, by contrast, finished in a median of just 1.3 hours and still delivered the best results.

How the agents went wrong

The researchers analysed 64 work traces and found three typical failure modes. In 58 cases the agents checked their own work too leniently. In 34 cases they spotted a problem but documented it instead of fixing it. In 21 cases they unilaterally shrank the scope of the task.

One example stood out. Claude Fable 5 forgot to launch four planned runs when starting its experiments, and its verification script adopted the same incomplete list, reporting "41/41" runs completed. The agent trusted that result until the very end. In another task, GPT-5.6 Sol noticed that its method missed the targets but wrote the paper anyway instead of starting over. DeepSeek V4 Pro dropped experiments because they were supposedly too slow for the "remaining time budget", even though there was no time limit.

There were counterexamples. In a quantum task, GPT-5.6 Sol caught a fundamental error, stopped the running experiments and recomputed everything from scratch.

A caveat on the judging

Because GPT-6 Astra also served as the evaluator, scoring every submission four times against the experts' rubrics, the researchers ran a cross-check in which the original papers' authors judged anonymised AI papers themselves. On whether a paper matched the given research direction, the automated system agreed with the experts in every case. On scientific quality, the judgments aligned in only about two-thirds of cases. One expert criticised that the papers did not tell a story, and the authors say judgments like these remain the job of human reviewers.

The authors stress that the benchmark is a testing ground for systematically improving research agents, not a verdict on whether AI belongs in science. The study's code and data are publicly available, and the project page hosts a leaderboard of the tested models. The picture looks much better for well-defined tasks: GPT-6 Astra solved 49 out of 50 unpublished problems in theoretical physics, though those solutions could be verified unambiguously. The gap, the authors conclude, is between running experiments and doing independent research — and for now, that gap remains wide.