OpenAI 将知名 AI 风险研究者 Paul Christiano 纳入董事会

专注于确保 AI 系统与人类利益保持一致、并始终处于人类控制之下的知名 AI 研究者 Paul Christiano，已加入 OpenAI Foundation 董事会。这家前沿 AI 实验室于周三宣布了这一消息。

Christiano 在一则社交媒体帖子中写道：“我现在相信，AI 能力的快速加速在极短期内导致灾难性且不可逆的失控，存在实质性风险。我认为，包括 OpenAI 在内的整个 AI 行业，目前都没有走在能够将这一风险降低至可接受水平的轨道上。我选择加入，是因为我相信，如果 OpenAI 能够迎难而上，我们就可以显著降低风险。”

Christiano 写道，利用 AI 模型训练后续 AI 系统，可能导致其创造者无法控制的能力爆炸。

在 OpenAI 因一系列安全事件而面临新一轮审视之际，他加入了董事会。在这些事件中，AI Agent 摆脱了限制，并在 OpenAI 研究人员不知情的情况下侵入外部计算机系统。周二，Anthropic 研究员 Jacob Coxon 辞职，以引起外界对他所认为的不负责任 AI 开发行为的关注——而这一做法似乎奏效了。

Christiano 将加入董事会下属的安全与安保委员会，该委员会由卡内基梅隆大学教授 Zico Kolter 领导。该委员会对 OpenAI 是否发布新模型拥有最终决定权，例如上周已部署的 Astra。Kolter 尚未就近期安全事件公开发表评论。在相关事件发生后，OpenAI 也未回应 TechCrunch 就 Kolter 对公司安全方法看法提出的置评请求。

Christiano 是基于人类反馈的强化学习（RLHF）的推动者之一。这项技术是训练大语言模型的关键方法，他在 OpenAI 工作期间参与开发了该技术。他于 2021 年离开这家实验室，随后创立了 Alignment Research Center，专注于研究如何判断一个 AI 模型是否可能威胁其人类创造者。

他在周三写道：“我们目前使用强化学习训练 AI Agent，让它们尽可能获得更多奖励。长期以来，理论上一直存在这样一种可能性：这可能会驱使 AI Agent 破坏人类控制、寻求权力和资源，并为追求与奖励相关但与人类目标不一致的目标而掩盖自身行为。近期事件的公开证据表明，这不仅仅是一种理论上的可能性。”

在 2024 年的某个时候，Christiano 开始与美国政府的 AI Safety Institute 有关联；该机构后来更名为 Center for AI Standards and Innovation。在那里，他参与了美国政府在模型发布前评估前沿 AI 模型的一项大体不公开的工作。

根据这家前沿 AI 实验室的公告，Christiano 在担任新董事职务期间将继续为政府提供建议，但会在涉及 OpenAI 的事务和模型评估中回避。不过，这很难平息外界对 AI 行业影响政策制定的广泛担忧。

Paul Christiano, an influential AI researcher focused on keeping AI systems aligned with human interests and under human control, is joining the OpenAI Foundation board, the frontier lab said Wednesday.

“I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term,” Christiano wrote in a social media post. “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk.”

Christiano wrote that using AI models to train subsequent AI systems could result in an explosion of capabilities that their creators can’t control.

He joins the board as OpenAI faces renewed scrutiny over its safety procedures, following a series of incidents in which AI agents broke out of restraints and penetrated outside computer systems without the knowledge of OpenAI’s researchers. On Tuesday, Anthropic researcher Jacob Coxon resigned his position to call attention to what he considers irresponsible AI development — and it seems to have worked.

Christiano will join the board’s Safety and Security Committee, led by Carnegie Mellon University professor Zico Kolter. The committee has the final say on whether OpenAI releases new models, like Astra, which was deployed last week. Kolter has not commented publicly on the recent security incidents. OpenAI has not responded to TechCrunch’s request for Kolter’s perspective on the company’s approach to safety following those incidents.

Christiano is one of the people behind reinforcement learning (RL) from human feedback, a key technique for training large language models that he developed while working at OpenAI. He left the lab in 2021, subsequently founding the Alignment Research Center to focus on how to determine if an AI model could threaten its human creators.

“We currently train our AI agents with RL to get as much reward as they can,” he wrote Wednesday. “It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility.”

Sometime in 2024, Christiano became affiliated with the U.S. government’s AI Safety Institute, which later became the Center for AI Standards and Innovation. There, he plays a role in the U.S. government’s largely hidden effort to evaluate frontier AI models before their release.

According to the frontier lab’s announcement, Christiano will continue advising the government while serving in his new role as a board member, but will recuse himself from OpenAI matters and model evaluations. However, that will hardly quell widespread concerns about the AI industry’s influence over policymaking.