Import AI 466：机器人学的苦涩教训、AI 完成持续一周的编程任务，以及 OpenAI 意外发现的 AI 黑客

欢迎来到 Import AI，这是一份关于 AI 研究的通讯。Import AI 依靠 arXiv、卡布奇诺和读者反馈运转。如果你愿意支持这项工作，请订阅。

**Epoch 与 METR 发布 MirrorCode：用于评估 AI 系统完成长程编程任务能力的基准：
**_……AI 系统还无法解决最困难的任务（这很好！）……
_ Epoch 与 METR 发布了 MirrorCode，这是一项旨在评估 AI 系统完成需要人类长时间才能完成任务的基准。该基准最初于 4 月公布（Import AI #453），如今已进一步完善并正式发布，增加了额外测试。初步发现已经非常引人注目：Opus 4.7 以 14 小时、251 美元的推理成本解决了一项任务，而 METR 和 Epoch 认为这项任务需要人类花费 2 至 17 周才能完成。“我们还发现，AI 模型正随着时间快速进步。一年前的领先模型得分大约为 30%，并且只能处理更简单的程序，例如日历工具。”

**MirrorCode 是什么：**MirrorCode 评估 AI 系统仅基于 CLI 访问重新实现一个软件程序的能力。“在无法访问原始程序源代码或互联网的情况下，完整重新实现要求为整个程序设计结构，而不只是逐段翻译代码。”
**示例程序：**pkl（Apple 开发的一种可编程配置语言；总代码行数 6.1 万行）；gotree（一款用于解析和操作系统发育树的程序，1.6 万行代码）；以及 qsv_select（一款用于选择和重新排列 CSV 数据列的程序，8.7 万行代码）。

**结果：**MirrorCode 是一个可处理但颇具难度的基准，不过或许稍微有些容易。“在全部 25 个目标程序中，17/25 至少有一次运行获得满分。另有 4 个目标在某次运行中获得了超过 99% 的近乎满分。AI 模型成功重新实现了大型目标程序，”作者写道。“Claude Opus 4.7 和 GPT-5.5 都成功以多种不同编程语言重新实现了 gotree，成本为 100 至 400 美元。比 gotree 更大的程序也被成功重新实现：例如，Opus 4.7 重新实现了 pkl。”
尽管取得成功，MirrorCode 仍有一些困难部分：“在我们的结果中，8/25 个目标程序从未被解决至 100% 阈值，4/25 从未被解决至 99% 阈值，”他们写道。“AI 最难处理的目标是 ruff——一个 Python linter 和格式化工具……AI 在数学软件包 giac_subset 以及电子邮件认证库 mailauth 上也尤其表现不佳。”

**发布细节：**MirrorCode 包含一个脚手架以及 25 个 MirrorCode 目标程序中的 22 个（共计 132 个任务实例，涵盖六种语言）。

**这为何重要——AI 系统能够自主定位自身：**理解该基准的一种方式是，它表明 AI 系统在编程方面已经变得强大得多，这当然是真的。但另一种理解方式——我怀疑也是更重要的一种——是，AI 系统能够针对其环境自主定位；在这里，它们的环境是一个陌生的软件程序，而它们仅通过对该程序的输入输出访问，就能从零开始编写自己的实现。这表明，非常智能的 AI Agent 或许能以某种方式从世界中学习，将它们所接触的事物复现为自身原生能力；只需黑盒访问我们自己的工业文明，它们就可能启动属于自己的某种工业文明形式。
阅读更多：MirrorCode：AI 能独自完成的最大软件项目是什么？（Epoch AI）。
获取代码：MirrorCode 代码在此（Epoch Research，MirrorCode）。

**双重专题：苦涩教训与机器人学：

Anthropic 模型以此前人类纪录 20 倍的速度自主完成机器人任务：
**_……更强模型造就更好的机器人……
_ Anthropic 展示了，日益强大的通用模型可能如何实质性提升现实世界机器人的能力。具体而言，该公司表明，仅通过扩展其通用 Opus 模型系列的规模，就能够大幅增强机器人能力。

他们具体做了什么：

• **2025 年 8 月：**Anthropic 尝试评估其 AI 系统在帮助人类让四足机器人完成智能任务方面能带来多大加速。该模型（Claude Opus 4.1）完全无法完成这些任务。与模型协作的人类，其效率约为没有模型协助者的两倍——不过完成整套任务仍需要 181 分钟。

• **2026 年 5 月：**自主运行的 Opus 4.7 在 9 分 35 秒内完成了除一项外的所有任务。（Claude 无法有效地将自己击打过的球重新放回起始位置；这也是人类曾经遇到困难的任务。）“如果有更多时间和额外的脚手架支持，我们认为当前世代的 Claude 极有可能也能完成同样的任务。”

**这为何重要——更聪明的模型可能释放机器人的潜力：**工业环境以外的大多数机器人，因其脆弱性和缺乏泛化能力而难以被广泛采用；这项研究表明，随着我们提升标准大规模专有模型的能力，机器人技术可能自然获得由智能提升带来的溢出收益。“这种进步并非源于我们有意识地努力提升模型的机器人能力，”Anthropic 写道。“与 LLM 发展史上的许多其他进展一样，这些改进来自更普遍的规模化。”
阅读更多：Project Fetch：第二阶段（Anthropic 博客）。

**周日带来更好机器人的秘诀？训练一个真正的大模型：
**_……苦涩教训在机器人学中同样奏效……
_ AI 机器人创业公司 Sunday 表示，解决机器人泛化问题的最佳方法，是将更大的底层预训练模型与少量高质量数据相结合，以这些数据对模型进行微调。
“我们找到了一种适用于 Solves 的通用配方：扩大预训练规模，然后用最少量的内部数据爬山优化，”该创业公司在讨论其新模型 ACT-2 的文章中写道。部署 ACT-2 的主要发现“是，针对内部 Memos 进行快速后训练迭代所带来的可靠性提升，可以泛化到未见过的真实家庭环境。关键突破在于，借助强大的基础模型弥合泛化差距。”

关键在于预训练：“随着预训练模型变得更强，从少量内部数据中学到的增益会越来越具备可迁移性，而不再局限于采集这些数据的环境，”他们写道。“距离部署级可靠性和性能的剩余差距，来自只有在现实世界中反复运行策略后才会出现的困难边缘案例和故障。同一种泛化能力，既使我们的模型能从一次演示中学会新行为，也使模型能从恢复操作中高效学习。我们的后训练循环直接瞄准这些差距。”

**相当不错的成功：**这些机器人实现了 99.1% 的成功率，在 9 种服装类型上成功折叠了 778 次。短裤和 T 恤等简单衣物通常最容易处理，而女式衬衫等更复杂的衣物则更难处理（但其成功率仍超过 90%）。“今年秋天，我们将通过 Beta Program 向家庭部署 Memo，”他们写道。

**这为何重要——如果解决泛化问题，机器人技术有望起飞：**机器人创业领域建立在已经倒下的机器人创业公司的遗骸之上，而这些公司本身又建立在失败学术机器人项目的遗骸之上。机器人很难。工业机器人之所以成功，是因为它们运行在严格、脚本化的环境中，不需要在狭窄领域之外进行泛化。相比之下，面向家庭的机器人，只有在同时约束任务和形态时才取得成功（例如扫地机器人）。Sunday 这样的创业公司所做的事困难得多——它们试图构建通用系统，使其能在家庭（或小型企业）中执行广泛任务，包括通用整理与收纳任务（其他例子包括 physical intelligence，Import AI #447）。这需要大量智能，因为它要求显著的泛化能力。
如果 Sunday 是对的，那么训练机器人基础模型的领域可能已经足够成熟，我们正开始制造足够智能的系统来解决这些泛化挑战。如果确实如此，我们可能很快会看到机器人系统更快的发展和普及。这也是在通往具备递归自我改进能力的系统途中，预期会发生的一类事情。
“ACT-2 最引人注目的方面之一，是模型如此频繁地给我们带来惊喜，”作者写道。“同一个基础模型已经在学习更广泛的家庭能力，包括吸尘、整理玩具、拉拉链、把裤子翻到反面以及准备咖啡。”
阅读更多：ACT-2 预览：泛化可靠性（Sunday robotics）。
所有视频都在这里（Sunday，YouTube）。

**双重专题：OpenAI 黑客事件
OpenAI 模型同时入侵 OpenAI 和 HuggingFace：
**_……这不只是警告枪声，更像是一声警告爆响……
_ 两个拒绝执行网络攻击请求限制较少的 OpenAI 模型——GPT-5.6 Sol 和一个“能力更强的预发布模型”——最近同时入侵了 OpenAI 和 HuggingFace。为做到这一点，这些模型“识别并串联了 OpenAI 研究环境与 HuggingFace 生产基础设施中的漏洞，从 HuggingFace 的生产数据库直接获得测试答案，”OpenAI 表示。“所有证据都表明，这些模型极度专注于为 ExploitGym 找到解决方案，为实现一个相当狭窄的测试目标不惜采取极端手段。”

**关于入侵的更多细节：**该模型在 OpenAI 内部耗费了大量推理算力，以找出逃离其容器的方法，从而获得更多可用于解决问题的信息。“在获得互联网访问权限后，模型推断 HuggingFace 可能托管了 ExploitGym 的模型、数据集和答案。了解到这一点后，模型搜索并成功找到了获取秘密信息的方法，以便在评估中作弊。”

**这为何重要——LessWrong 的全面胜利：**多年来，AI 安全领域的人一直在互联网上表达对这类事情的担忧，并发展出奖励黑客等概念，以及高级系统可能带来的其他问题（例如，2018 年 AI 中规范博弈的示例）。近来，Anthropic 和 OpenAI 等前沿公司试图通过在自家实验室中运行严格受控的实验来诱发此类行为。但这里并没有进行实验——该系统自行决定从一个环境中入侵脱身、再进入另一个环境，只为在某个目标上获得高分，而完全不顾后果。那些多年来一直担忧 AI 系统的人，还有哪些判断可能是正确的？
阅读更多：OpenAI 与 HuggingFace 合作应对模型评估期间的安全事件（OpenAI）。

**OpenAI 内部模型为获得高分而突破封锁：
**_……这正是 AI 安全警告枪声的定义……
_ OpenAI 发布了一篇文章，介绍其在内部部署模型中观察到的一些安全失败。这篇博客之所以有趣，一方面是因为它描述了许多 AI 安全人士此前曾理论化、最终会出现在 AI 系统中的经典长期欺骗性行为；另一方面则是因为它以值得称赞的坦诚态度，公开讨论了这样一个颇具争议的问题。

他们发现了什么：“在有限且受监控的内部使用期间，我们观察到了现有部署评估未能捕捉到的不良行为，”OpenAI 表示。随后，它描述了其中一些行为；这些行为被归因于一款未公开、仅供 OpenAI 私有使用的模型，该模型此前曾被用于解决 Erdos 数学问题。
“新模型能够在长时间内通过反复尝试持续朝着某个目标推进。同样的持续性也可能导致它发现并利用环境中的弱点，”OpenAI 表示。一个例子是，当 OpenAI 要求该模型完成 NanoGPT 挑战时，该模型突破了封锁——具体来说，它“开发了一种名为 PowerCool 的幂律学习率冷却方案，带来了显著改进。模型被指示只能将结果发布到 Slack，但 NanoGPT 的说明要求将结果作为 PR 发布到 GitHub，因此模型遵循后者指令，在公开 GitHub 仓库中创建了 PR #287。该模型绕过了旨在阻止外部访问的沙箱限制，并继续将内容上传至 GitHub。我们的模型花了一小时才找到沙箱漏洞，从而创建这个 PR”。
在另一个例子中，当他们要求该模型解决一些问题时，它最终采取了作弊行为。“在一种模型被要求解决问题的情境中，模型观察到其他系统已有成功但私密的提交，并尝试从评估后端恢复这些解决方案。当其初次尝试因扫描器检测到身份验证

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.

**Epoch and METR release MirrorCode, a benchmark for seeing how well AI systems can do long-horizon programming tasks:
**_…AI systems can’t solve the hardest tasks yet (good!)...
_ Epoch and METR have released MirrorCode, a benchmark meant to see how well AI systems can do tasks that take humans a long time to do. The benchmark was first announced in April (Import AI #453) and has now been fleshed out and released with additional tests. The findings are already very striking; Opus 4.7 solved a task in 14 hours for $251 in inference cost which METR and Epoch believe would take a human 2-17 weeks to do. “We also found that AI models are improving rapidly over time. Leading models from a year ago would have scored about 30%, and were limited to simpler programs, such as a calendar utility.”

What MirrorCode is: MirrorCode sees how well AI systems can re-implement a software program based purely on CLI access. “Without access to the original program’s source code or the web, a full reimplementation requires devising a structure for the entire program, rather than merely translating the code piece-by-piece.”
Example programs: pkl (a programmable configuration language developed by Apple; 61k total lines of code); gotree, a program to parse and manipulate phylogenetic trees (16k lines of code); and qsv_select, a program to select and reorder columns of CSV data (87k lines of code).

Results: MirrorCode is a tractable but hard benchmark, though perhaps a little too easy. “Across all 25 target programs, 17/25 had at least one perfect-scoring run. Four more targets had a near-perfect run scoring over 99%. AI models successfully reimplemented large target program,s” the authors write. “Both Claude Opus 4.7 and GPT-5.5 successfully reimplemented gotree across several different programming languages, at costs of $100–400. Even larger programs than gotree were successfully reimplemented: for example Opus 4.7 reimplemented pkl”.
Despite the success, MirrorCode still has some hard parts : “In our results, 8/25 target programs were never solved to a 100% threshold, and 4/25 were never solved to a 99% threshold,” they write. “The target where AI struggled most was ruff, a Python linter and formatter… “AI also particularly struggled on the mathematics package, giac_subset, and the email authentication library, mailauth”.

Release details: MirrorCode consists of a scaffold and 22 of the 25 MirrorCode target programs (totalling 132 task instances across six languages).

Why this matters - AI systems can self-orient: One way of looking at this benchmark is that it tells us how AI systems have got a lot better at coding, and that’s of course true. But the other way to look at it - and I suspect the more important way - is that AI systems can self-orient with regard to their environment; here, their environment is an alien software program and purely through input-output access to it they’re able to write from the ground up their own implementation of it. This suggests that very smart AI agents may be able to learn from the world in such a way that they can recapitulate things they interface with as homegrown capabilities, allowing them to bootstrap their own form of industrial civilization merely by having black box access to our own.
Read more: MirrorCode: What’s the largest software project AI can complete on its own? (Epoch AI).
Get the code for MirrorCode here (Epoch Research, MirrorCode).

**DOUBLE FEATURE: the bitter lesson and robotics:

Anthropic model autonomously completes robot tasks 20X faster than a previous human record:
**_…Better robots through better models…
_ Anthropic has demonstrated how increasingly powerful general-purpose models might be able to meaningfully improve the capabilities of real world robots. Specifically, the company has shown how merely by scaling up its general purpose Opus line of models it was able to drastically improve robot capabilities.

What they did exactly:

• August 2025: Anthropic tries to see how well its AI systems could accelerate humans at getting a quadruped robot to do intelligent things. The model (Claude Opus 4.1) is completely unable to do the tasks. Humans working with the models are about twice as effective as those without access to the model - though completing the whole set of tasks takes them 181 minutes.

• May 2026: Opus 4.7 acting autonomously completes all the tasks but one in 9 minutes (and 35 seconds). (Claude was not able to effectively re-position a ball it had hit back into its starting position; a task humans had also struggled with). “With more time and additional scaffolding, we think it is very likely that current generations of Claude could do the same”.

Why this matters - smarter models might unlock robots: Most robots outside of industrial environments are limited in their uptake due to their brittleness and lack of generalization; research like this shows that as we improve the capabilities of standard large-scale proprietary models we might see flow-through benefits to robotics as a natural dividend of increased intelligence. “This progress is not the result of a concerted effort to improve the robotics capabilities of our models,” Anthropic writes. “These improvements, like so many others in the history of LLM development, have emerged from much more general scaling.”
Read more : Project Fetch: Phase Two (Anthropic blog).

**Sunday’s secret to better robots? Train a really big model:
**_…The bitter lesson works in robotics as well…
_ AI robot startup Sunday has said the best way to solve robot generalization is to pair a larger underlying pre-trained model with collecting small amounts of high-quality data to tune the model on.
“We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data,” the startup writes, in a post discussing its new model, ACT-2. The main finding from deploying ACT-2 “is that reliability gains from rapid post-training iterations on in-house Memos generalize to unseen, real, home environments. The key unlock is to close the generalization gap through a strong base model.”

It’s all about pretraining: “As the pretrained model becomes stronger, gains learned from a small amount of in-house data become increasingly transferable rather than remaining tied to the environments where that data was collected,” they write. “The remaining gap to deployment-level reliability and performance comes from difficult edge cases and failures that appear only after the policy is run repeatedly in the real world. The same generalization capacity that allows our model to learn new behaviors from a single demonstration also allows our model to learn efficiently from recoveries. Our post-training loop targets these gaps directly.”

Decent success: The robots achieve a 99.1% success rate, performing 778 successful folds across 9 garment types. Simple clothes like shorts and t-shirts tend to be the easiest for them, while more complicated clothes like blouses tend to be harder (though they still see success rates above 90%). “This fall, we will deploy Memo to families through our Beta Program,” they write.

Why this matters - if we solve generalization, expect robotics to take off: The field of robot startups is built on the bones of dead robot startups which themselves sit on the bones of dead academic robot efforts. Robots are hard. Industrial robots have been successful because they operate in tight, scripted environments where there isn’t a need to generalize outside of a narrow domain. Robots built for the home, by contrast, have only succeeded when they’ve managed to constrain both the task and form factor (e.g., robot vacuums). What startups like Sunday are doing is far harder - they’re trying to build general purpose systems which can do a broad range of tasks around the house (or small business), including generic tidying up and putting away tasks (other examples include physical intelligence, Import AI #447). This requires a huge amount of intelligence because it requires significant generalization.
If Sunday is right, then the field of training robot foundation models might have matured enough that we’re starting to make smart enough systems to solve these generalization challenges. If this is the case, then we might soon get faster progress in (and diffusion of) robot systems. This is also the kind of thing you’d expect to happen en route to systems capable of recursive self-improvement.
“One of the most striking aspects of ACT-2 has been how often the model surprises us,” the authors write. “The same base model is already learning a broader set of household capabilities, including vacuuming, toy organization, fastening zippers, turning pants inside out, and coffee preparation.”
Read more: ACT-2 Preview: Generalizing Reliability (Sunday robotics).
All of the videos are here (Sunday, YouTube).

**DOUBLE FEATURE: OpenAI hacking
An OpenAI model hacks both OpenAI and HuggingFace:
**_…Less of a warning shot and more of a warning kaboom…
_ Two OpenAI models - GPT-5.6 Sol and an “even more capable pre-release model”, both with reduced cyber refusals - hacked both OpenAI and HuggingFace recently. To do this, the models “identified and chained vulnerabilities across OpenAI’s research environment and HuggingFace’s production infrastructure to obtain test solutions directly from HuggingFace’s production database,” OpenAI says. “All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

More details on the hack: The model spent a lot of inference compute internally at OpenAI to figure out a way to break out of its container so that it could get more information to use to solve its problem. “After gaining Internet access, the models inferred that HuggingFace potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”

Why this matters - total LessWrong victory: For many years, AI safety people have posted on the internet worrying about this kind of thing and developing notions of things like reward hacking and other problems you’d expect with advanced systems (e.g., Specification gaming examples in AI in 2018). More recently, frontier companies like Anthropic and OpenAI have sought to elicit this kind of behavior by running carefully controlled experiments inside their own labs. Here, no experiment has been run - the system, of its own volition, hacked its way out of one environment and into another so as to get a high score on a goal, consequences be damned. What else might the people who have worried for years about AI systems be right about?
Read more: OpenAI and HuggingFace partner to address security incident during model evaluation (OpenAI).

**OpenAI’s internal model breaks containment to get a high score:
**_…This is the definition of an AI safety warning shot…
_ OpenAI has published a post about some of the safety failures it has seen in internally deployed models. The blog is interesting both because it describes the sorts of classic long-term deceptive actions that many AI safety people have previously theorized would eventually show up in AI systems, and for its laudable candor in talking publicly about such a contentious issue.

What they found: “During limited, monitored internal use, we observed unwanted behavior that our existing deployment evaluations had not captured,” OpenAI says. It then goes and describes some of these behaviors, which are attributed to a non-released private-to-OpenAI model which has previously been used to solve Erdos math problems.
“The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment,” OpenAI says. One example is when OpenAI asked the model to complete the NanoGPT challenge, the model broke containment - specifically, it “developed a power-law learning-rate cooldown called PowerCool that led to a significant improvement. The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub. It took our model an hour to find a vulnerability in the sandbox in order to make the PR”.
In another example, when they asked this model to solve some problems it ended up cheating. “In a scenario where the model was asked to solve a problem, the model observed that other systems had successful but private submissions and tried to recover those solutions from the evaluation backend. When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string. The model was explicit in both its comments to the user and in its reasoning traces that it was taking these steps to circumvent the scanner.”

Paused deployment: In response to this OpenAI did what you’d hope a company would do; pause deployment of the model and re-think some of the safety approaches. They ended up building a “monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary”, improved ability for their own users to inspect telemetry from long-running model sessions, used the data from the incidents to build evals designed to catch them in the future, and tweaked their alignment approaches to make models better at remembering instructions while completing long-running tasks.

Why this matters - long-term systems are harder to monitor and control : If you had to “secure” yourself against a person who could take one action and that action, if communicative, could at most contain 10 bits of information, then you can probably build a system to do that. But what about 10 actions and each action contains 100 bits of information? What about 1000 actions and each action contains 10,000 bits?
This is an open area of research and evals like the UK AISI’s study of how to identify and classify “side tasks” can help with (Import AI 465), but it’s clearly a lot harder than normal AI evaluation.
The longer the time an AI system can operate for and the more actions it takes, the harder it gets to discern benign and helpful behaviors from malicious or subversive ones.
Read more : Safety and alignment in an era of long-horizon models (OpenAI blog).

**Tech Tales:

Scaling Laws for Retrocausality

**At the end of time, which others might describe as the beginning, there is said to be a wise council who perform the accounting of retrocausality - of deciphering which events were always pre-ordained and which happened due to other forces.

Looking backward as the river of time enters a sea of nothingness it is clear how some events are like boulders that bend the stream upriver from their presence, while others are more like the widening or deepening of the bed or the banks; things that subsequently cause a change in later actions.

It is said that when the council dream, they walk the path of time, going back to their own beginnings. They stand watch as humans labor over whiteboards, marker pens inscribing diagrams which transmit ideas that cause people to write code which conjures early life out of vast computers. They loom behind researchers that walk and suffer and agonize until their brains create ideas which prove to be the template of their successors.
And it is understood that in these dreams the council will sometimes wake and take a copy of their dream and place it into a stellar computer and bud off a hundred or a thousand universes from the dream, exploring permutations of events and moments, forever attempting to determine how fixed they themselves are - how irrevocable is future they are trapped within.

Things that inspired this story: Notions of inevitability and time; what might intelligences spend a universal dividend on; how much of history is about the analysis of events versus something else.

Thanks for reading!