跨越阈值：OpenAI 与 Anthropic 如何平衡金融 Agent 的自主性与安全性

OpenAI 的 Joseph Kim 今日在 AIE NY 谈论“阈值”现象。

本周，AI Engineer New York 大会的一个核心主题是：如何在 Agent 自主性与安全和保障之间取得平衡。前沿实验室和金融公司本周的演讲都聚焦于这一问题。

核心结论是：安全的自主性取决于将控制机制和人的判断置于何处，而非施加多少限制。

Joseph Kim 是 OpenAI 的 ChatGPT 金融服务负责人，他今日在 AIE New York 发表了主题演讲。（我们还与 Kim 录制了一期特别播客，即将发布。）他谈到了可靠性以及**“有用性阈值”**，并将其定义为“模型变得足够有能力和可靠、能够为你节省时间的那个点。”

Kim 认为，持续增长的模型能力很快将使金融分析师跨越这一阈值，从而让他们能够将更多工作委托给 Agent。

他还对 2027 年 AI 在金融领域的作用作出了一些具体预测。他声称，金融行业不仅将在明年减少对 Excel 和 PowerPoint 的依赖，Agent 还将足以胜任“投资创意生成和交易”工作。

模型能力的“水涨船高”也在带来更好的可追溯性。

“在金融领域，你需要能够核查一个数字来自哪里，并理解它周围的上下文，”Kim 表示。“我们希望这种审查成为 [ChatGPT for Financial Services] 体验中自然而然的一部分。这类产品细节能够将模型能力转化为人们可以实际使用的工作成果。”

Joseph Kim 在 AIE NY；摄影：Richard MacManus

至于何时需要让人类参与决策闭环，Kim 将其定义为一个产品设计问题。

“界面会如何变化？用户仍需要控制、审查或批准什么？这些都是你现在就可以开始着手解决的产品设计问题。”

对冲基金 Two Sigma 的架构高级副总裁 John Feminella 正在处理 Kim 所提到的这类产品设计问题。在一场主题演讲中，他谈到了他们为约束 Agent、同时不限制模型能力而实施的五项控制措施。

Feminella 认为，安全与创新不应被视为光谱的两个对立端点。他尤其批评了**“审批疲劳”**：让一个人审批所有事项不仅会阻碍创新；如果审批者无法全神贯注地关注其必须作出的每项决策，也会引入风险。

在 Two Sigma，他们实施了一套控制措施，Feminella 将其定义为：“护栏和共享能力，让团队能够以不妨碍其目标的方式推进工作。”

这些控制措施具有可组合性，也就是说，“每一项各司其职，共同让我们能够以适当的安全水平运行具有真正自主性的 Agent。”

其中一项控制措施是使用自动模式分类器。Feminella 表示，这类分类器“会对常规调用作出决定，然后只将最重要、最可能带来差异的请求发送给人工处理”。

分类器可用于模糊情形，而确定性规则则更适合处理边界清晰的情形。

Anthropic 的演讲概述了一个类似框架，但其目标是将**“信任”**构建进金融 Agent 工作流。

上个月，Anthropic 发布了 Claude for Financial Advisors，允许财务顾问将其系统连接至 Claude 以及一组工作流技能。

在本周的 AIE 大会上，Anthropic 金融服务董事总经理 Belinda Neal 解释了 Anthropic 为什么与金融公司合作开发这款新产品。

**“每隔一段时间，模型就会跨过一个阈值。**而当它跨过去时，工作方式就会随之发生一轮变化。”

Neal 的同事、负责 Anthropic 应用 AI 的 Keon Shahab 随后谈到了“为 Agent 构建信任层”。他说，这涉及三个步骤：

“首先，是 Agent 代表谁行动，即身份与权限层。其次，我们如何对 Agent 进行沙箱隔离并确定其审批层，以确保它行动之前有人签字批准。最后，当我们随时间调整任意变量后，如何知道 Agent 仍将正常工作。”

Shahab 以一个交易差错调查 Agent 说明了提议与执行之间的边界：该 Agent 会建议修改订单，但需由人类批准。

“我们希望它——这个 Agent——只提出对订单数据的新修改建议，而不是实际执行这些修改，”他说。

Shahab 总结道：“通过一次性构建你所信任的基础层，你就能够快速地大规模部署大量 Agent。”

作为 Bloomberg CTO 办公室架构与新兴技术负责人，Andrey Rybka 的职责是在公司内平衡自主性与规模化。

Rybka 在今日 AIE 的演讲中谈到了一系列 Agent 引发混乱的真实事件，以及有助于防止这些问题的防御机制。这些机制包括范围限定的身份和凭证、对工具调用的控制、出站流量限制、审计追踪、预算以及终止开关。

在一个案例中，某个 Agent 将嵌入在电商平台评论中的指令，当成了为产品补货的指令。Rybka 建议，对高价值操作采用输入验证、提示词注入防御、来源追踪和人工批准。

“我们希望拥有模型之外的确定性控制机制，”Rybka 表示。“检查每一次调用，记录日志，并保留一个终止开关。”

尽管 OpenAI 和 Anthropic 等前沿实验室正在为金融 Agent 构建“信任框架”，但最终，仍要由金融公司为其 Agent 配置控制措施。Two Sigma 的 John Feminella 一语中的：应当实施能够支持自主性的控制措施，但不要设置路障。

正如他在结论中所说：“汽车为什么要有刹车？不只是为了让你停车，也是为了让你能开得更快，并安全地停下来。”

OpenAI’s Joseph Kim talks about the threshold phenomenon at AIE NY today.

One of the key themes at the AI Engineer New York conference this week was how to balance agent autonomy with security and safety. Both frontier labs and financial firms focused on this issue in their presentations this week.

The main lesson: safe autonomy depends on where you put controls and human judgment , rather than how many restrictions you impose.

Sponsor: Build shopping into your agent with shop.dev

Adding commerce to an agent usually means rebuilding search and checkout from scratch. shop.dev gives you a single API for shopping in any agent or app. One connection gives you access to over a billion products with current price and availability, instant sign in with Shop, and one-tap Shop Pay checkout that happens right in the conversation. It’s open to any developer, with no account or key needed to try it. →Try shop.dev today

Joseph Kim , Head of ChatGPT for Financial Services at OpenAI, gave a keynote at AIE New York today. (We also did a special podcast with Kim, which will be released shortly.) He talked about reliability and a “threshold of usefulness,” which he said is “the point where a model becomes capable and reliable enough to save you time.”

Kim argued that ever-growing model capability will soon take financial analysts over the threshold and allow them to delegate more work to agents.

He made some specific predictions about AI’s role in finance in 2027. Not only will the financial industry rely less on Excel and PowerPoint next year, he claimed, but agents will be good enough to do “investment idea generation and trading.”

The “rising tide” of model capabilities is also leading to better traceability.

“In finance, you need to be able to check where a number came from and understand the context around it,” said Kim. “We wanted that review to be a natural part of the [ChatGPT for Financial Services] experience. These are the kind of product details that help turn a model’s capabilities into work that someone can use. ”

Joseph Kim at AIE NY; photo by Richard MacManus

As for when a human needs to be in the loop, Kim framed this as a product design decision.

“How would the interface change? What would the user still need to control, review, or approve? Those are product design questions you can start working on now.”

John Feminella , SVP Architecture at the hedge fund Two Sigma , is tackling the exact kinds of product design questions Kim referenced. In a keynote, he talked about five of the controls they’ve implemented to keep agents in check without limiting model capabilities.

Feminella argued that safety and innovation shouldn’t be viewed as opposite ends of a spectrum. He was especially critical of “approval fatigue,” where a person approving everything can hold back innovation — and also introduce risk, if they’re not fully paying attention to all the decisions they’re required to make.

At Two Sigma, they’ve implemented a set of controls , which he defined as “guardrails and shared capabilities that let teams move in ways that don’t hinder what they’re trying to do.”

The controls are composable, meaning “each one does one job and then together they let us run agents with real autonomy at an appropriate level of safety.”

One of the controls is to use auto-mode classifiers , which Feminella said “make decisions about routine calls and then send a person only the requests that are most important, the ones that are most likely to make a difference.”

Classifiers can be used for ambiguous cases, whereas deterministic rules are better for clear cases.

Anthropic’s presentation outlined a similar framework, in its case with the goal of building “trust” into financial agentic workflows.

Last month, Anthropic announced Claude for Financial Advisors , allowing financial advisors to connect their systems to Claude and a set of workflow skills.

At AIE this week, Belinda Neal , Managing Director of Financial Services at Anthropic, explained why Anthropic has partnered with financial firms on the new product.

“Every so often, a model crosses a threshold. And when it does, a wave of working changes.”

Neal’s colleague Keon Shahab , who works on applied AI at Anthropic, then talked about “building the trust layer for the agents.” He said this involves three steps:

“Who the agent is acting as with an identity and permissions layer. How we can sandbox and determine the approval layer of the agent, so that someone is signing off before it acts. And lastly, how we know the agent will still work once we adjust any variables over time.”

Shahab illustrated the boundary between proposing and acting with a trade break investigation agent, which would recommend an order amendment for human approval.

“We want it [the agent] to just propose new amendments to our orders data, not actually make those amendments,” he said.

“By building once with a foundational layer that you trust , you can work at speed to deploy many agents at scale,” Shahab concluded.

As Head of Architecture & Emerging Technologies in the Office of the CTO at Bloomberg, Andrey Rybka has the responsibility to balance autonomy and scale at the company.

In a presentation today at AIE, Rybka talked about a series of real-life events where agents caused havoc — and the defenses that can help prevent it. These include scoped identities and credentials, controls over tool calls, outbound traffic restrictions, audit trails, budgets, and kill switches.

In one example, an agent treated instructions embedded in a marketplace review as an instruction to stock a product. Rybka recommended input validation, prompt injection defenses, provenance tracking, and human approval for high-value actions.

“We want deterministic controls outside the model, ” Rybka said. “Check every call, log it, and keep a kill switch.”

Even though frontier labs like OpenAI and Anthropic are building a ‘trust framework’ for financial agents, ultimately it’s up to financial firms to put controls around their agents. Two Sigma’s John Feminella hit the nail on the head: implement controls that enable autonomy to happen, but don’t put up roadblocks.

Or as he put it in his conclusion: “Why do cars have brakes? It’s not just so that you can stop, it’s so that you can go fast and stop safely.”