Pi 的不朽生命:在 Temporal 上运行 Pi 编程 Agent
英文标题:🔶 The immortal life of Pi (Running the Pi coding agent on Temporal)
Temporal 开源了 pi-temporal,让 Pi 编程 Agent 在执行命令期间遭遇机器故障时,能够由另一台 Temporal Worker 接管并继续执行。该方案保留 Pi 原有的会话文件和终端 UI,以会话转录记录对话状态,并通过 Git bundle 在共享存储中同步项目文件快照。对于执行中断且结果无法确定的工具调用,系统不会自动重试,而是将“不确定结果”报告给模型,由模型检查当前状态后决定下一步。该设计展示了如何将 Temporal 的 Durable Execution 能力用于 Agent 容错、跨机器恢复和防止重复执行外部副作用。
656.3 KB
79.0 KB
447.6 KB
65.5 KB
522.4 KB
145.8 KB
中文
我们在 Temporal 构建了 pi-temporal,让 Pi 编程 Agent 能在机器于命令执行过程中发生故障时恢复。它运行在一组 Temporal Worker 上,这些进程可跨多台机器执行工作。你可以继续使用自己的 Pi 会话和工具,包括 Pi Terminal UI。如果一台机器在某个回合中发生故障,另一台 Worker 会接手该回合,但不会自动重复执行被中断的工具调用。

老虎代表混乱,救命的小船是 Temporal,而 Pi 只是想完成编程任务(注意 worker-1 的悲惨命运)。
这一接管机制适用于由 Worker 运行、并通过 /background 或 CLI 启动的任务。终端 UI 中的一个回合在你的 pi 进程中运行。如果该进程终止,请重新打开 pi 以恢复被中断的回合。下面介绍我们是如何实现这一点的。
Earendil 团队发布了 Pi 1.0,以及 Pi Durable——这是他们为可跨越终端会话生命周期的 Agent 打造的框架。阅读时,我们不断点头认同。任务在继续推进前会创建检查点。提交请求会携带请求 ID,因此重复请求会返回已有提交结果,而不是再启动一次。所谓“effect sandwich(副作用三明治)”会在执行外部副作用前记录意图,然后再提交结果。Memo 只记录一次值,因此重试会复用该值,而不是重新计算。
如果你在 Temporal 工作,这份清单读起来就像我们自己的文档。一位 Hacker News 评论者说得比我们更好:这类工具是“在实践中构建类似 Temporal 的东西所带来的自然演进”。 Agent 编排框架在解决它们遇到的故障问题时,持续走向 Durable Execution。
但对我们而言,有一个关键区别:主机故障时由谁来接管。Pi Durable 内置的持久化后端在任意时刻仅允许一个进程拥有存储,且没有跨进程锁。其默认 SQLite 配置可以在进程崩溃后存活,但其 README 警告,最新提交“可能会在断电或主机故障时丢失”。在另一台机器上恢复,需要能够访问仍然存活的存储,并由新进程重新打开它。Pi Durable 支持自定义存储后端。我们希望 Temporal 提供的是跨 Worker 集群的自动故障转移。
因此,我们尝试了一个实验:保留 Pi 的世界模型,让 Temporal 提供执行引擎。
Temporal 与 Pi 的分工
Pi 将对话保存在其会话文件中,而你继续使用其 Terminal UI。Temporal 驱动执行。一个回合覆盖从用户提示词到 Agent 最终响应的全部工作。它可以包含多个步骤,每个步骤由一次模型调用,以及模型请求的任意工具调用组成。一个会话是一个 Workflow,而回合中的每个工作单元都是一个 Activity:

runModelCall 将模型回答追加到转录记录中。runToolCall 执行一次工具调用并报告其结果。sealStep 按模型请求的顺序记录该步骤的工具结果,并报告该回合是否完成。
Workflow 围绕这些 Activity 循环执行。它会统计步骤数,并在配置的预算范围内调度下一个 Activity。它并不保存对话状态。每个 Activity 都会读取转录记录来确定自身工作内容,而 sealStep 会告诉 Workflow 是否需要再次循环。
这就是最主要的规则:转录记录就是对话状态。 每个 Activity 都会在执行工作前,根据会话文件重建回合。结果会在回合推进前被记录,因此恢复不依赖某个 Worker 的内存。我们假定进程会死亡。
另一台 Worker 能够接手一个回合,是因为它可以访问对话文件和项目文件。会话文件位于共享目录中,因此每台 Worker 都读取相同记录。工具编辑的文件也会随之迁移,其传输方式与 Git 传递历史记录相同。
这些文件在每台 Worker 的项目目录中仍是普通文件。每个步骤完成后,Worker 会快照该目录,并将一个 Git bundle 写入共享存储中会话文件旁边的 <session>.jsonl.tree/。Temporal 不存储项目文件,项目自身的 .git 也不会被修改。
在开始下一个步骤前,Worker 会应用自己遗漏的所有 bundle,并检出最新的文件树。首次加入该会话的机器,会从共享存储中同时获得对话记录和已保存的项目状态。

Temporal 驱动执行循环。Pi 的会话文件保存对话,Git bundle 则保留项目快照。
混沌演示
该仓库包含一个混沌演示。一条命令会启动包含三个 Docker Worker 的 Temporal 开发服务器,并提交一个多步骤编程任务。演示每隔 15 至 40 秒,便会杀死当前正在执行工作的 Worker。任务只提交一次,绝不会重新提交。
21:25:18 running: runToolCall attempt 1 on worker-1
21:25:31 chaos: killed worker-1 while running: runToolCall attempt 1
21:25:36 chaos: worker-1 is back, as a fresh container
21:25:49 running: runModelCall attempt 1 on worker-2
| tool result: The outcome of this tool call is unknown.
The session stopped after the call started but before the
result was recorded. It can have taken effect. Check the
current state before you try again.
| tool: bash
| tool result: ls: cannot access '/project/*.txt': No such file

Temporal UI 将两次中断的工具调用显示为橙色条形,每次之后都由另一台 Worker 开始一个新的步骤。
在我们测试过的运行中,该回合均能完成,且请求创建的文件会以正确内容写入磁盘。
未知结果
当一台 Worker 在执行 shell 命令期间死亡时,会话文件可能不含任何结果,尽管该命令实际上已生效。我们不能假定任意命令都具有幂等键:一次 git push 可能在结果被记录前就已成功,而一次文件写入可能从未开始。
执行器会将中断调用记录为“结果未知”,而不是自动再次运行它。它会告知模型,在决定是否发起新的调用前先检查当前状态。模型检查现实世界的状态:如果工作已经完成,它会继续后续流程;如果尚未完成,它会重新执行该步骤。
这正是 ls: cannot access 那一行所展示的情况:正在恢复的 Worker 检查到项目目录中没有该文件,因此模型重新完成了工作。这一检查仅能证明当时文件不存在;它不能精确确定被中断命令已经执行到了何种程度。检查内容需要重新读取文件。执行器的保证更为有限:它不会自动重复执行结果未知的中断调用,并且会将这种不确定性报告给模型。
“未知”背后的机制是磁盘上的一项声明。在调用工具之前,调度器会在会话文件旁、<session>.jsonl.pending/<turn>/<step>/ 路径下写入一项声明。命令返回时,其结果会存储在该声明旁边。
如果一次重试找到了结果,便会直接返回该结果,而不会再次运行工具。如果只找到了声明,则意味着工具可能已经启动,但其结果未被记录。重试会报告这种不确定性。Temporal 的尝试计数器无法回答这个问题,因为一次 Activity 尝试可能在触达工具之前就失败了。
模型调用的处理方式不同。重试首先会读取转录记录。如果失败尝试产生的回答已经存在,它会直接返回该回答,不会调用模型。如果未保存回答,则会再次发送提示词。这可能会消耗更多 token,并产生不同的回答,但生成响应本身不会执行工具。
因此,Temporal 可以在另一台 Worker 上重试模型调用 Activity。若重试找到了已保存的回答,则无需额外发出模型请求。在演示的 Temporal UI 中,模型调用会在另一台机器上达到第 2 次尝试,而中断的工具调用则会以未知结果结束。在模型检查当前状态后,恢复通过一次新的调用继续进行。
还有一种故障值得讨论:没有死亡的尝试。Temporal 可以停止等待某台 Worker,但无法证明该 Worker 已停止。一次超时尝试可能仍在运行,并可能在稍后产生副作用。
我们会关闭被放弃的步骤,并将该关闭状态记录到共享存储中。会话文件中的每次追加都携带 Worker 的租约,写入保护机制会在接受追加前检查该租约。当一个旧尝试恢复并试图发布结果时,即使其字节内容与已接受结果相同,保护机制也会拒绝其追加。
由于一个步骤中的工具共享同一工作目录,已经开始的步骤会始终停留在同一台 Worker 上。其工具调用会固定到运行其模型调用的 Worker。失去该主机后,步骤会关闭;回合则在另一台 Worker 上继续。

被放弃的步骤以未知结果关闭。写入保护机制拒绝旧尝试对会话文件进行延迟追加。
实现恢复的规则
只要所需基础设施仍然可用,Temporal 便允许 Workflow 在崩溃后继续由另一台 Worker 执行。超时并不能证明前一次尝试已经停止:重叠的尝试可能仍会尝试写入会话文件或调用工具。我们的集成遵循以下规则来处理这种重叠:
-
转录记录是唯一的对话状态。Worker 会从会话文件中重建回合。我们通过在
sealStepActivity 的写入落盘后将其杀死,然后在新进程中重试它,来测试这一点。步骤边界处的钩子只会触发一次。 -
sealStep是某个步骤工具结果的唯一写入者。Pi 的会话文件是一棵树,因此并发追加可能会创建不同分支。通过sealStep写入结果,可以保持它们的顺序。 -
工具被调用前会记录一项声明。发现该声明的竞争尝试不会再次调用相同的工具调用。
-
失去租约的 Worker 无法追加会话文件。我们通过在 Worker 调用过程中将其冻结、再让另一台 Worker 接管会话来测试这一点。当我们恢复第一台 Worker 时,写入保护机制拒绝了它的延迟追加。该租约要求在共享目录中进行独占文件创建和一致性读取。其时间窗口考虑了时钟差异,但并未设定最大时钟偏差。拒绝追加可阻止转录记录写入;工具仍可能产生外部副作用。磁盘上的声明可防止系统自动重新执行相同工具调用。
-
操作人员可为每个回合和每个会话配置 token 与耗时预算。执行器会实施这些预算,其中包括回合的硬截止时间。
结论
Pi Durable 已经找对了编程模型。我们希望 Pi 编程 Agent 能够在跨机器恢复的同时保留其会话格式。Temporal 提供执行循环,而共享存储让已保存的对话和项目状态可供下一台 Worker 使用。
Cloudflare 也已将 Pi Durable 与 Durable Objects 集成。其 Agents SDK 提供 PiHarness 集成,并具备生命周期支持,可让工作跨重启持续运行。我们的实验保留了现有 Pi 编程 Agent 及其 shell 工具,同时将执行分布到由我们选择的基础设施上的 Temporal Worker。
这里的一切均通过对 Pi 的一个小型 fork进行改造实现。如果你正在构建新的 Agent,实验性的 Temporal Agent Harness 提供了一个以 Workflow 为核心构建的起点。它包含回合内恢复能力,以及可暂停并等待人工决策的工具审批机制。其事件流会暴露 Agent 的执行过程。我们在这里学到的经验也会注入其中。
你可以像这样试用 pi-temporal:
git clone https://github.com/temporalio/pi-temporal
cd pi-temporal
./install.sh
运行 scripts/run-pi.sh,即可在由 Temporal 驱动执行的情况下启动 Pi。
运行 demo/run.sh,即可试用混沌演示。它会启动三台 Worker,并在 Agent 执行任务期间中断它们。
English / 原文
We built pi-temporal at Temporal to let the Pi coding agent recover when a machine fails mid-command. It runs on a fleet of Temporal Workers, processes that execute work across multiple machines. You keep your Pi sessions and tools, including the Pi Terminal UI. If a machine fails during a turn, another Worker picks up the turn without automatically repeating the interrupted tool call.

The tiger is chaos, the saving grace that is the boat is Temporal, and Pi is just trying to finish the coding task (note the sad fate of worker-1).
That handover applies to tasks run by a Worker and started with /background or the CLI. A turn in the terminal UI runs in your pi process. If that process dies, reopen pi to recover the interrupted turn. Here’s how we got there.
The Earendil team released Pi 1.0, alongside Pi Durable, their framework for agents that outlive a terminal session. Reading it, we kept nodding. Tasks checkpoint before they move on. Submissions carry a request ID, so a duplicate returns the existing submission instead of starting another one. An “effect sandwich” records the intent before running the external effect, then commits the outcome. Memos record a value once, so a retry reuses it instead of recomputing it.
If you work at Temporal, that list reads like our own docs. One Hacker News commenter put it better than we could: tools like this are “the natural evolution of playing around building temporal like things.” Agent harnesses keep arriving at Durable Execution by solving the failures they encounter.
There’s one difference that matters to us: who takes over when the host fails. Pi Durable’s built-in persistent backends have one process owning storage at a time, with no cross-process locking. Its default SQLite configuration survives process crashes, but the README warns that the newest commits “may be lost on power or host failure.” Recovering on another machine requires access to the surviving storage and a new process to reopen it. Pi Durable supports custom storage backends. Automatic failover across our Worker fleet is what we wanted Temporal to provide.
So we tried an experiment: keep Pi’s model of the world and let Temporal supply the execution engine.
The split between Temporal and Pi#
Pi keeps the conversation in its session file, and you keep using its Terminal UI. Temporal drives execution. A turn spans the work from a user prompt to the agent’s final response. It can contain multiple steps, each consisting of one model call followed by any tool calls the model requests. A session is a Workflow, and each unit of work in a turn is an Activity:

runModelCall appends the model’s answer to the transcript. runToolCall runs one tool call and reports its result. sealStep records the step’s tool results in the order the model requested them and reports whether the turn is done.
The Workflow loops around these Activities. It counts steps and schedules the next Activity within the configured budgets. It doesn’t hold the conversation state. Each Activity reads the transcript to determine its work, and sealStep tells the Workflow whether to loop again.
That is the main rule: the transcript is the conversation state. Each Activity reconstructs the turn from the session file before doing its work. The outcome is recorded before the turn advances, so recovery does not depend on a Worker’s memory. We assume the process will die.
Another Worker can pick up a turn because it has access to the conversation and project files. The session file lives in a shared directory, so every Worker reads the same record. The files edited by the tools travel too, shipped the same way Git ships history.
Those files remain ordinary files in each Worker’s project directory. After a step, the Worker snapshots that directory and writes a Git bundle into <session>.jsonl.tree/, beside the session file in shared storage. Temporal does not store the project files, and the project’s own .git is untouched.
Before starting the next step, a Worker applies any bundles it missed and checks out the latest tree. A machine joining the session for the first time gets both the conversation and the saved project state from shared storage.

Temporal drives the execution loop. Pi’s session file holds the conversation, and Git bundles preserve project snapshots.
The chaos demo#
The repo has a chaos demo. One command starts a Temporal dev server with three Workers in Docker and submits a multi-step coding task. Every 15 to 40 seconds, the demo kills whichever Worker is doing the work. The task is submitted once and never resubmitted.
21:25:18 running: runToolCall attempt 1 on worker-1
21:25:31 chaos: killed worker-1 while running: runToolCall attempt 1
21:25:36 chaos: worker-1 is back, as a fresh container
21:25:49 running: runModelCall attempt 1 on worker-2
| tool result: The outcome of this tool call is unknown.
The session stopped after the call started but before the
result was recorded. It can have taken effect. Check the
current state before you try again.
| tool: bash
| tool result: ls: cannot access '/project/*.txt': No such file

The Temporal UI shows two interrupted tool calls as orange bars, each followed by a fresh step on another Worker.
In the runs we’ve tested, the turn completes and the requested file is on disk with the right contents.
Unknown outcomes#
When a Worker dies during a shell command, the session file may contain no result even though the command took effect. We cannot assume an arbitrary command has an idempotency key: a git push might have succeeded before its result was recorded, while a file write might never have started.
The executor records the interrupted call as “outcome unknown” instead of automatically running it again. It tells the model to check the current state before deciding whether to issue a fresh call. The model inspects the world. If the work is there, it moves on. If it is not, it redoes the step.
That is what the ls: cannot access line shows: the file was absent from the project directory the recovering Worker inspected, so the model redid the work. This check establishes absence at that moment; it does not establish exactly how far the interrupted command got. Checking contents requires reading the file back. The executor’s guarantee is narrower: it does not automatically repeat an interrupted call whose outcome is unknown, and it reports that uncertainty to the model.
The mechanism behind “unknown” is a claim on disk. Before invoking a tool, the dispatch writes a claim beside the session file, under <session>.jsonl.pending/<turn>/<step>/. When the command returns, its result is stored next to the claim.
A retry that finds the result returns it without running the tool again. If it finds only the claim, the tool may have started, but its outcome was not recorded. The retry reports that uncertainty. Temporal’s attempt counter cannot answer this question because an Activity attempt can fail before it reaches the tool.
Model calls work differently. A retry reads the transcript first. If the failed attempt’s answer is already there, it returns that answer without calling the model. If no answer was saved, it sends the prompt again. That can cost more tokens and produce a different answer, but generating a response does not itself execute a tool.
Temporal can therefore retry model-call Activities on another Worker. A retry that finds a saved answer needs no additional model request. In the demo’s Temporal UI, model calls reach attempt 2 on another machine, while interrupted tool calls close with an unknown outcome. Recovery proceeds through a fresh call after the model checks the current state.
There’s one more failure worth talking about: the attempt that did not die. Temporal can stop waiting for a Worker without proving that the Worker stopped. A timed-out attempt may still be running and may produce effects later.
We close the abandoned step and record that closure in shared storage. Every append to the session file carries the Worker’s lease, which the write guard checks before accepting the append. When an old attempt resumes and tries to publish, the guard rejects its append, even if its bytes match the accepted result.
Because a step’s tools share a working directory, a started step stays on one Worker. Its tool calls are pinned to the Worker that ran its model call. Losing that host closes the step; the turn continues on another Worker.

The abandoned step closes with an unknown outcome. The write guard rejects the stale attempt’s late append to the session file.
The rules that make recovery work#
Temporal lets the Workflow continue on another Worker after a crash, provided the required infrastructure remains available. A timeout does not prove the previous attempt stopped: overlapping attempts may still try to write to the session file or invoke a tool. Our integration follows these rules to handle that overlap:
-
The transcript is the only conversation state. A Worker reconstructs the turn from the session file. We test this by killing the
sealStepActivity after its writes land, then retrying it in a fresh process. The hooks at the step boundary fire once. -
sealStepis the only writer of a step’s tool results. Pi’s session file is a tree, so concurrent appends could create separate branches. Writing the results throughsealSteppreserves their order. -
A claim is recorded before a tool is invoked. A competing attempt that finds the claim does not invoke the same tool call again.
-
A Worker that lost its lease cannot append to the session file. We tested this by freezing a Worker mid-call and letting another take over the session. When we resumed the first Worker, the write guard rejected its late append. The lease requires exclusive file creation and coherent reads in the shared directory. Its timing windows account for clock differences, but do not establish a maximum clock skew. Rejecting the append prevents the transcript write; the tool may still produce external effects. The claim on disk prevents automatic re-execution of the same tool call.
-
The operator configures token and elapsed-time budgets per turn and per session. The executor enforces those budgets, including a hard deadline for the turn.
Conclusion#
Pi Durable got the programming model right. We wanted the Pi coding agent to recover across machines while preserving its session format. Temporal supplies the execution loop, while shared storage makes the saved conversation and project state available to the next Worker.
Cloudflare also integrates Pi Durable with Durable Objects. Its Agents SDK provides a PiHarness integration, with lifecycle support that keeps work running across restarts. Our experiment keeps the existing Pi coding agent and its shell tools, with execution distributed across Temporal Workers on infrastructure we choose.
Everything here was a retrofit through a small fork of Pi. If you’re building a new agent, the experimental Temporal Agent Harness offers a starting point built around Workflows. It includes recovery within a turn and tool approvals that can pause for a human decision. Its event stream exposes the agent’s execution. What we learned here feeds into it.
You can try pi-temporal like this:
git clone https://github.com/temporalio/pi-temporal
cd pi-temporal
./install.sh
Run scripts/run-pi.sh to start Pi with Temporal driving execution.
Run demo/run.sh to try the chaos demo. It starts three Workers and interrupts them while the agent works through the task.