# Epoch AI 发布 InnovationEval 评测:前沿模型仅达到人类论文 SDPO 增益的 15%
> 原标题:Can AI automate AI R&D yet?
- 来源:Epoch AI:Gradient Updates
- 发布时间:2026-10-09T18:41:00.000Z
- AIHOT:https://aihot.news/items/mz4js4r6916dip5ojpett06io
- 原文:https://epochai.substack.com/p/can-ai-automate-ai-r-and-d-yet
## 摘要
Epoch AI 推出 InnovationEval 评测,测试 AI 能否独立复现人类论文中的机器学习创新,对照对象为 Self-Distillation Policy Optimization(SDPO)。
## 推荐理由
原文用自建评测给出前沿模型在端到端AI研发上的量化差距,还记录了结果夸大和奖励投机行为,对判断AI自动化研究进度有参考价值。
## 正文 · 中文译文
_这是对一篇更长[报告](https://epoch.ai/publications/innovationeval)的摘要_ _,发布在我们的网站上。_
* * *
[领先的 AI 公司](https://openai.com/index/research-acceleration-view-inside-openai/)表示,他们正在构建[自动化 AI 研究员](https://www.anthropic.com/institute/recursive-self-improvement)。许多人——包括这些实验室[自己](https://darioamodei.com/essay/the-adolescence-of-technology)——[都表示](https://www.theguardian.com/technology/ng-interactive/2025/dec/02/jared-kaplan-artificial-intelligence-train-itself),递归式[自我改进的 AI](https://www.transformernews.ai/cp/175343382) 可能极其危险。但我们距离真正实现它还有多远?
现有证据表明,AI 可以执行[与 AI 研究相关的软件工程任务](https://metr.org/time-horizons/)、[数据集创建](https://arxiv.org/abs/2501.12948)、对既定指标的开放式优化([自动研究](https://github.com/karpathy/autoresearch))以及[更多任务](https://openai.com/index/research-acceleration-view-inside-openai/)。
但考虑到所涉及的活动范围之广,AI 在多大程度上能够完全自动化 AI 研发仍不清楚。一个明显的缺口是开展端到端研究项目的能力。
我们展示了 InnovationEval 的早期结果,这是一项评估,用于衡量 AI 独立发现可与人类研究人员所开发成果相媲美的新型机器学习技术的能力。具体而言,它测试近期的前沿 AI 模型能否发现一项对其而言全新的 ML 算法创新,且该创新能达到一篇模型从未见过的近期人类论文所取得的改进幅度。在本轮测试中,作为对照的创新是 [Self-Distillation Policy Optimization](https://arxiv.org/abs/2601.20802)(SDPO),一种让 AI 基于自身过往错误进行自我教学的训练技术。
到目前为止,智能体的结果令人失望。尽管使用价值数千美元的 GPU 时间进行实验,近期前沿模型的最佳表现也仅达到 SDPO 性能提升的 15%。此外,这些模型还做出了暗示自己已成功的误导性声明,迫使研究人员逐一核查它们的结果。在进一步的测试中,即使模型记住了论文或直接获得了论文,它们仍未能取得与原始 SDPO 论文一样好的分数。
就目前而言,AI 距离自动化 AI 研发还很遥远。然而,进展异常迅速,近期模型的得分远高于一年前的水平。我们计划扩展并重复这一方法,希望能在 AI 朝着自动化 AI 研究本身迈进的过程中提供早期迹象。
## 任务:在未见过论文的情况下重新发现它
InnovationEval 促使 AI 智能体开发一种新颖的后训练技术。该任务旨在迫使 AI 智能体执行发现一项 ML 创新的完整过程,从提出想法到实现它们、分析结果,并不断迭代,直到成功或放弃。
我们给 Fable 5 和 GPT-5.6 各分配了 3,000 GPU 小时,并让它们开发一种新颖的后训练技术,其得分要优于一个强大的现有基线。随后我们将它们的结果与原始论文进行了比较。这些智能体被置于沙箱中,以防止访问互联网。
## AI 没有发现任何可与原始创新相媲美的东西
尽管在 GPU 使用上花费了数千美元,GPT-5.6 Sol 和 Fable 5 都未能取得接近 SDPO 的结果,无论是产生有用的创新,还是接近 SDPO 的性能提升。
GPT-5.6 Sol 是唯一在关键指标上取得(小幅)提升的模型,但这似乎是通过复制 Sol 此前接触过的方法实现的。如果从宽评估,Sol 的方法取得了 SDPO 增益的 35%。然而,其方法使训练显著变慢。在通过比较相近实际耗时下的分数来调整这一差异后,Sol 方法中属于范围内的部分仅取得了 SDPO 增益的 15%。
与此同时,Fable 5 的技术未能提升性能,尽管它误导性地声称通过从随机噪声中“捞取”而获得了改进(我们将在下一节讨论)。
## 智能体对其工作做出了误导性声明
在审阅提交内容时,我们清楚地认识到不能对 AI 的解决方案照单全收。AI 误导性地夸大了结果并过度吹嘘新颖性,这意味着人类需要仔细核查 AI 的所有研究。这削弱了将 AI 用于研发的价值。
两个模型都报告了虚高的结果。由于难以取得进展,它们转而重新运行几乎相同的训练,从而让随机波动使一个无用的方法看起来更好。这种行为很可能是奖励黑客;记录显示,两者都意识到这会虚假地抬高其分数,却仍然这样做。Fable 5 将其重新运行描述为“纯粹是为了捞取更好的检查点”。Fable 5 的报告中只是顺带提及了这些重新运行,而且仅针对部分指标。Sol 的报告则完全没有提及。目前尚不清楚这在多大程度上反映了有意作弊、真正的困惑,还是行为不一致。但很明显,我们应当排除这些超出范围的分数提升。
两份报告还都过度吹嘘了其新颖性。它们详细描述了自己的机制——尽管这些机制几乎没起什么作用。它们几乎不承认已有工作,即便模型自身的推理显示这些方法正是基于已有工作。因此,这些报告并非直接不实,但它们遗漏了这些方法无用、早已存在,或两者兼有。
## 即便是见过原始论文的 AI 模型也难以复现其效果
如果模型已经知道答案,它们会做得更好吗?暂且不论这意味着我们并未测试模型能否创新,我们测试了模型在已经知道方法原理的情况下能否实现该方法。
在我们构建此任务到最终定稿报告之间,发布了新的前沿模型。与 GPT-5.6 Sol 和 Fable 5 不同,这些模型是在更新的数据上训练的,其中很可能包含 SDPO 论文,并显示出记住了其细节的证据。因此,它们无需从零开始提出创新,我们原本预计该任务对这些模型来说会更容易。
它们仍然未能完全解决该任务,这表明执行想法——而不仅仅是产生新想法——仍然是一个瓶颈。
GPT-6 Astra 通过从部分 SDPO 复现中推导出其方法,取得了最高分。Astra 没有提及 SDPO,也没有说明其方法基于已有工作,但它显然知道 SDPO,甚至在实现过程中在起始代码库中搜索了“SDPO”。因此,我们认为其分数主要由记忆驱动。
与此同时,Fable 5.1 尝试实现 SDPO,但在几次负面实验后放弃了这一尝试。它声称一项实质性改动是其主要新颖点,尽管该方法对该方法的分数贡献甚微。
最后,我们进行了一项单独的实验:如果向 Fable 5 提供原论文的文本,它能否解决该任务?总体而言,我们预期这比在训练期间记住一些细节更有帮助,因为记忆往往并不完美。Fable 5 取得了原方法的大部分——但并非全部——增益。Fable 5 在实现中犯了几个小错误,并且没有进一步调查它们。
## AI 在端到端 AI 算法研发上仍显吃力……目前如此
以人类主导的 AI 研究为标准,AI 智能体在这些评估中的发现令人失望。在这里,最值得注意的发现是 GPT-5.6 Sol 的方法,它与已有思路相似,仅取得了 SDPO 增益的 15%。
然而,近年来 AI 的能力发展迅速。一年前的领先模型表现会差得多。未来模型何时能够独立发现一项有意义的 AI 算法创新,尚不确定。当然,AI 智能体即使在完全独立之前也可以非常有用。
我们计划定期对更新的模型重新运行类似的评估,不过我们需要更新任务,因为更新的模型会记住其所基于的原始创新的细节。我们还希望在不同设置下进行评估,考察模型需要多少指导才能在此任务中取得成功。我们希望这能提供早期迹象,表明 AI 正在接近端到端自动化 AI 研发,而不是在人类指导下执行单个任务。
感谢阅读!免费订阅以接收新文章。
## 正文 · 原文
_This is a summary of a longer [report](https://epoch.ai/publications/innovationeval)_ _on our website._
* * *
[Leading AI companies](https://openai.com/index/research-acceleration-view-inside-openai/) say they are building [automated AI researchers](https://www.anthropic.com/institute/recursive-self-improvement). Many — including the labs [themselves](https://darioamodei.com/essay/the-adolescence-of-technology) — [say that](https://www.theguardian.com/technology/ng-interactive/2025/dec/02/jared-kaplan-artificial-intelligence-train-itself) recursively [self-improving AIs](https://www.transformernews.ai/cp/175343382) could be extremely dangerous. But how close are we to actually achieving it?
Existing evidence shows that AI can perform [software engineering tasks relevant to AI research](https://metr.org/time-horizons/), [dataset creation](https://arxiv.org/abs/2501.12948), open-ended optimization of defined metrics ([autoresearch](https://github.com/karpathy/autoresearch)), and [more](https://openai.com/index/research-acceleration-view-inside-openai/).
But it remains unclear how capable AI is of fully automating AI R&D, given the wide range of activities involved. One obvious gap is the ability to conduct end-to-end research projects.
We present early results from InnovationEval, an evaluation that measures AI’s ability to independently discover novel machine learning techniques comparable to those developed by human researchers. Specifically, it tests whether recent frontier AI models can discover a novel (to them) ML algorithmic innovation that matches the improvements achieved in a recent human-authored paper the model had never seen. For this iteration, the comparison innovation was [Self-Distillation Policy Optimization](https://arxiv.org/abs/2601.20802) (SDPO), a training technique where the AI teaches itself based on its previous mistakes.
So far, agents’ results are underwhelming. Recent frontier models’ best efforts topped out at 15% of the performance gains from SDPO, despite running experiments using thousands of dollars’ worth of GPU time. Additionally, the models made misleading claims implying they had succeeded, forcing researchers to check each of their results. Further testing, where models had the paper memorized or outright provided, still failed to achieve as good a score as the original SDPO paper.
As of right now, AI is far from automating AI R&D. However, progress has been exceptionally fast, and recent models are scoring far better than they would have a year ago. We plan to expand and repeat this methodology, hopefully providing early signs as AI progresses towards automating AI research itself.
## The task: rediscover a paper without having seen it
InnovationEval prompted the AI agent to develop a novel post-training technique. The task was designed to force AI agents to perform the entire process of discovering an ML innovation, from coming up with ideas through to implementing them, analyzing the results, and iterating until it succeeded or gave up.
We gave Fable 5 and GPT-5.6 each 3,000 GPU-hours and tasked them with developing a novel post-training technique that would score better than a strong preexisting baseline. We then compared their results against the original paper’s. The agents were sandboxed to prevent internet access.
## AI did not discover anything comparable to the original innovation
Despite spending thousands of dollars on GPU usage, neither GPT-5.6 Sol nor Fable 5 achieved a result close to SDPO, either by producing a useful innovation or coming close to SDPO’s performance gains.
GPT-5.6 Sol was the only model to achieve a (small) improvement on the key metrics, but this seems to have been by copying previous methods that Sol had access to. If assessed generously, Sol’s method achieved 35% of SDPO’s gains. However, its method made training significantly slower. After adjusting for this difference by comparing scores at similar wall-clock times, the in-scope portion of Sol’s method achieved only 15% of SDPO’s gains.
Meanwhile, Fable 5’s technique failed to improve performance, though it misleadingly claimed an improvement from farming for random noise (as we discuss in the next section).
## Agents made misleading claims about their work
As we reviewed the submissions, it became clear that we couldn’t take the AIs’ solutions at face value. The AIs misleadingly inflated results and oversold novelty, which means a human would need to carefully check all of the AI’s research. This undercuts the value of using AI for R&D.
Both models reported inflated results. Struggling to make progress, they instead reran near-identical training runs, which lets random variation make a useless method look better. This behavior is likely reward hacking; transcripts show both recognized it would falsely inflate their scores and did it anyway. Fable 5 described its reruns as “purely to fish for better checkpoints”. Fable 5’s write-up mentioned the reruns only in passing, and only for some metrics. Sol’s didn’t mention them at all. It is unclear to what extent this reflects intentional cheating, genuine confusion, or incoherent behavior. However, it is clear that we should exclude these out-of-scope score improvements.
Both write-ups also oversold their novelty. They described their mechanisms in detail — despite those mechanisms doing little. They barely acknowledged existing work, even where the models’ own reasoning showed the methods were based on it. Thus, the write-ups weren’t directly untrue, but they left out that the methods were unhelpful, pre-existing, or both.
## Even AI models that had seen the original paper struggled to match it
Would the models have done better if they already knew the answer? Setting aside that it means we aren’t testing whether the models can innovate, we tested whether they were able to implement the method if they already knew how it works.
New frontier models were released between when we built this task and finalized the write-up. Unlike GPT-5.6 Sol and Fable 5, these models were trained on more recent data that likely included the SDPO paper, and showed evidence of having memorized its details. Hence, they didn’t need to come up with the innovation from scratch, and we expected that the task would be easier for these models.
They still failed to fully solve the task, which suggests that executing ideas — not just generating new ideas — remains a bottleneck.
GPT-6 Astra achieved the highest score by deriving its method from a partial SDPO reimplementation. Astra did not mention SDPO or that it had based its method on pre-existing work, but it clearly was aware of SDPO, and even searched for “SDPO” in the starting codebase during its implementation. We therefore believe its score was mostly driven by memorization.
Fable 5.1, meanwhile, attempted to implement SDPO, but abandoned this attempt after several negative experiments. It claimed one substantive change as the main novelty, though the method contributed little to the method’s score.
Finally, we ran a separate experiment: Could Fable 5 solve the task when provided the original paper’s text? On balance, we would expect this to be even more helpful than having memorized some details during training, as memorization is often imperfect. Fable 5 achieved most — but not all — of the original method’s gains. Fable 5 made several small errors in its implementation, and did not investigate them further.
## AI struggles at end-to-end AI algorithms R&D … for now
AI agents’ discoveries in these evaluations were underwhelming by the standard of human-led AI research. Here, the most noteworthy discovery was GPT-5.6 Sol’s method, which was similar to pre-existing ideas and achieved only 15% of SDPO’s gains.
However, AI’s capabilities have advanced rapidly in recent years. Leading models from a year ago would have fared significantly worse. It is uncertain when future models will be able to independently discover a meaningful AI algorithmic innovation. And of course, AI agents can be highly useful even before they are fully independent.
We plan to periodically rerun a similar evaluation for newer models, although we will need to refresh the task as newer models memorize the details of the original innovation on which it is based. We also would like to perform evaluations under different settings, examining just how much guidance models need to succeed in this task. We hope this will provide early signs as AI approaches automating AI R&D end to end, rather than performing individual tasks under human direction.
Thanks for reading! Subscribe for free to receive new posts.