OpenAI、Anthropic 与 DeepMind 离职者访谈：如果你的雇主可能毁灭世界，你会怎么做？

9 月 8 日，27 岁的 Anthropic 员工 Jacob Coxon 在 X 上发布了一份辞职声明，迅速在全网引发震动。Coxon 此前是一名模型能力研究员（capabilities researcher），职责是让模型变得更强大、更具自主性。他选择辞职，是为了抗议他眼中公司“如自杀般鲁莽地追求通用人工智能（AGI）”的行为。

如果你的雇主可能毁灭世界，你会怎么做？

对于 AI 圈内人来说，有 Anthropic 员工认为自家公司的技术可能会终结世界，这本身算不上什么新鲜事。（Anthropic 最初就是由一群前 OpenAI 员工创立的，他们当时就认为如果不加约束，OpenAI 迟早会走向这一步。）但一名 Anthropic 员工公开辞职以示抗议，性质就截然不同了。OpenAI 和 Anthropic 堪称当今全球最具实力的科技公司，它们身处前沿阵地，正在构建多数从业者眼中人类有史以来最关键的技术。过去，当有人离开其中一家实验室时，通常会直接跳槽到竞争对手那里。2024 年从 OpenAI 辞职的 Daniel Kokotajlo 回忆称，曾有一名研究人员对他说，离开一家顶级实验室就如同“放弃国籍，变成一个无名之辈，谁还会来保护你？”

争论随即爆发：如果你是一名身处顶级实验室、且深信这些存在性风险的员工，你究竟该怎么做？是奋起反抗、毅然离开？还是留守权力中枢内部，从内部施加影响？而这种抉择又会怎样冲击你的人际关系与圈子？

多年来，有一小部分来自 OpenAI、Anthropic 和 Google DeepMind 的员工选择了出走，Coxon 如今也加入了他们的行列。《New York》杂志与 Asterisk 杂志联合采访了其中的 8 位出走者。他们离职的原因各不相同：少数人像 Coxon 和 Kokotajlo 一样，希望借此唤醒外界对 AI 盲目加速风险的关注；另一些人则反对雇主与美国国防部的合作，或是对曾怀揣理想主义的非营利组织蜕变为商业巨兽感到沮丧；还有一些人只是单纯觉得，无论自身从事的是安全对齐还是缓解大规模失业的研究，留在外部都能发挥更大的价值。

Miles Brundage，OpenAI，2018–2024 年，政策研究主管
Rosie Campbell，OpenAI，2021–2024 年，前沿政策团队负责人
Jacob Coxon，OpenAI，2023–2026 年；Anthropic，2026 年 5 月–9 月，模型能力研究员
Jacob Hilton，OpenAI，2018–2023 年，强化学习研究员
Pamela Mishkin，OpenAI，2020–2026 年，经济学研究团队负责人
Daniel Kokotajlo，OpenAI，2022–2024 年，治理研究员
Alex Turner，Google DeepMind，2023–2026 年，对齐研究员
Jeff Wu，OpenAI，2018–2024 年；Anthropic，2024–2026 年，对齐研究员

我是 Asterisk 的主编，这是一家立足旧金山湾区的非营利杂志。从大语言模型（LLM）刚能在高中数学测试及格就被视为惊天大事起，我们便一直在追踪报道 AI 领域。Asterisk 获得了 Coefficient Giving 的资金支持，今天受访的部分人士或其机构也同样获得了该机构的资助。受访者中有些人互不相识，有些人则是朋友。他们担心的议题各异，对 p(doom)（毁灭概率）的预估也不尽相同——事实上，他们甚至不认为 p(doom) 是一个在逻辑上严谨统一的概念。但凝聚在一起，他们为我们呈现了这场 AI 军备竞赛的真实图景，以及众多从业者所面临的个人抉择与内心权衡。

—— Clara Collier

你们两位都对 AI 风险深感担忧，并希望促使各家实验室更加审慎。为什么你们最终会认为，在外部推动这一目标比留在内部更有效？
Daniel Kokotajlo： 我离开 OpenAI 主要是因为我对这家公司、尤其是领导层会负责任地行事彻底失去了信心。我希望能有更多自由公开发声，谈谈我所预见到的未来，我也想向公众公开我辞职的原因。

Daniel Kokotajlo，OpenAI，2022–2024 年，治理研究员

你可以尝试打“内部局”，也可以选择打“外部局”，而事实证明内部博弈根本行不通。管理层非常擅长极其专注地倾听每个人的意见，嘴上满口赞同，随后却什么都不落实。不仅如此，公司文化也偏向盲目乐观，充斥着自我美化的叙事。我真不认为科技公司能够实现自我监管。相比继续留在内部写更多的内部备忘录，我现在在外部公开发声、发表研究成果，所产生的影响力要大得多。

Jacob Coxon，OpenAI，2023–2026 年；Anthropic，2026 年 5 月–9 月，模型能力研究员

Jacob Coxon： 我的想法和 Daniel 非常相似——只不过现在的局势已经逼近决战时刻了。我的决定源于与 Anthropic 领导层就明年的规划进行直接沟通后的触动。我当时心想：天呐，我绝对不想成为这些计划的一部分。 整个过程就像是：第一步，我不想同流合污；第二步，辞职；第三步，把真相告诉公众；第四步——我完全不知道该怎么办。

那具体是什么样的计划？
JC： 那是与负责研究的管理层的直接交流。不是 Dario Amodei 本人，而是那些在公司呆了很久、对公司未来发展轨迹有深厚背景认知的人。他们一直以来的口径都很一致：2027 年局面就会开始失控。私下交流中浮现出的核心共识是：大家都不认为政府监管能及时出台，尤其是对达成任何形式的国际合作完全不抱希望。这或许原本不该让我感到意外，但我还是被震惊到了。我过去一直抱有一种天真的预期：一旦人工超级智能近在眼前，大家自然就会严肃对待，实验室也不会再把它当成一场有趣的玩票游戏。

Jacob，你之前就已经在关注存在性风险了。过去几个月在 Anthropic 的亲身见闻，在多大程度上改变了你的想法？还是说这本就是你预料之中的轨迹？
JC： 对我而言，理智上的信念和我直觉上的感受存在非常大的断层。如果你让我叙述进入 Anthropic 时和离开 Anthropic 时的理论观点，听起来可能差不多。但就从始至终的真实心理感受而言，两者完全是天壤之别。

刚入职时，我的工作感觉和任何常规的数学研究毫无区别。我潜意识里知道这是最重要的行业——这也是我选择进入的原因。但我当时并不觉得有必要深入探讨背后的哲学问题或具体战略细节，觉得那是其他人的职责。可到了最后，看着我们即将训练的下一代模型所具备的预测能力，我产生了一种发自肺腑的恐惧：天呐，我们即将训练出的这个模型，真的很可能会让我感到害怕。 就算不是这一代，也绝对会是紧随其后的下一代。

你能尝试向非研究人员解释一下，你具体观察到了哪些令人担忧的现象吗？
JC： 现阶段我们在使用 AI 时，研究人员必须提供思路和方向，而且事实上还需要进行大量的人工介入与监督（babysitting）。但我认为有极大概率，下下一代模型在自主构思想法方面的表现会同样出色。这意味着在整个科研流程中，不再需要人类研究员为 AI 提供有价值的研究构想，研究员完全只需对 AI 说一句：“好了，去做高质量的研究吧”，然后让它自行运转。因此，非常具体地说，眼睁睁看着我们正按部就班地创造出这种能力，这让我产生了一种极其真切的恐惧感，同时也感到深深的无力感。

在你决定离开之前，你都找谁商量过？
JC： 我找 Daniel 聊过，因为促成我离职的部分原因，就是看到了他的“AI 2040”推演方案——将原本预想的 Plan A 与默认的基准发展轨迹进行对照，这对于切身体会当前发展路径有多么糟糕至关重要。我也找过 Anthropic 内部许许多多的人，请教他们关于如何离职的建议。在公司内部，我与 Holden Karnofsky 以及预训练负责人 Nick Joseph 等人沟通过，看看他们的看法。

那些对话的氛围是怎样的？
JC： 其实很稀松平常。在湾区经常有聚会，大家聚在一起社交，一拨人会指责另一拨人正在导致世界走向毁灭。在工作场合，只要双方都保持礼貌，哪怕是非常尖锐对立的对话——比如直接说“我认为你正在将世界推向深渊”，大家依然能心平气和地交流，然后客客气气地挂断 Zoom 会议。

外界一直有一种反驳声音，认为渲染这些存在性风险不过是 AI 公司为了抬高估值、为 IPO 造势的噱头。你怎么看？
JC： Evan Hubinger 发的那条推文可绝对没有帮到 Anthropic 的 IPO。炒作论在几年前或许还站得住脚，但现在的风险已经足够严重，甚至已经开始对公司的估值产生明确的负面冲击了。

当前存在一个独立于各大实验室之外但又与其紧密合作的 AI 安全机构生态圈——例如，他们会研究如何判断模型是否在说谎，或者如何确保模型受到充分监管。这会让你们感到别扭吗？还是说你们认为维持这些纽带是必要的？
DK： 有时确实会让我感到别扭，但我依然自觉选择继续与这些公司保持接触，而且在一定程度上取得了成效。我认为业内许多人都承认，“AI 2027”和“AI 2040”对他们产生了深远的影响。但我内心深处确实依然感到不安。有时我也会反思，我和其他人是否应该采取更强硬的立场，比如采取类似抵制和孤立的态度。

那么，这在多大程度上属于战略层面的抉择，又在多大程度上属于人际交往层面的抉择？
DK： 战略抉择在于：作为个人以及作为一个群体，我们是否应该出于实际效果或道德考量划定某种界限——即采取抵制或孤立？在这点上，我认为策略上的选择是“不采取抵制”。保持这些沟通纽带更为重要，这样他们才能倾听我们的声音、了解我们的理念。而人际抉择则是：在个人层面上，你是否愿意和那些你认为正在毁灭世界的人做朋友、一起玩？答案是“绝不”。我绝不会邀请那些大公司的能力研究员来参加我女儿的生日派对。我偶尔会在各种活动中碰到他们，但我打心底里不喜欢他们这些人。

JC： 我还是更倾向于归咎于大环境机制——整个竞速格局、最初是由私营企业主导研发这一事实、以及政府似乎并不怎么关心的现状。对于所有这一切的责难，远甚于我对具体个人的指责。当然，我一个月前自己还是个能力研究员，现在这么说或许显得很站着说话不腰疼。但在我看来，这本质上是一个系统性问题，必须在系统层面上加以解决。至于具体到每个人身上，我总觉得——我也说不清——其实没那么重要。

我不同意这种说法。这些实验室总是在强调人才极度短缺，个别顶尖研究员举足轻重。我不相信这些个人抉择真的无足轻重。
DK： 我并没有说个人选择不重要。正因如此，我一直认为大家选择辞职是件好事，我也一直在呼吁这一点。

JC： 也许你是对的。并不是说他们无关紧要，而是我无法去苛责他们个体。我看到了背后的利益机制，我其实并不……

DK： 坦白讲，我确实责怪他们。但我认为责任也是分层级的：我对公司里的安全研究人员只有轻微的埋怨，因为他们应该意识到，留在外部或许能比在内部做更多的好事；我对能力研究人员有中等程度的责备，因为他们理应清楚自己正在做的事情对世界并没有好处，而他们用来安慰自己的借口不过是自我合理化；我对领导层则是极其严厉的谴责，因为是他们亲手将我们引向深渊；最后，我指责整个体制，因为它催生了这样的激励机制。

那么 Jacob，作为一个月前还在岗位上的能力研究员，你当时是用什么借口来合理化自己的行为、说服自己这没问题的？
JC： 说实话，这取决于你在什么状态下问我。我有时会开玩笑地说：“我正在研发毁灭世界的东西。”但正如我前面所说，我理解整件事的方式是把它看作一台庞大的机器，而我觉得自己只是机器里的一颗螺丝钉。

DK： 我听过太多人说过：“一切都会好起来的。别担心，超级智能还远得很。现有的范式迟早会见顶失效”之类的话。随着距离目标越来越近，这类说辞在逐渐减少，但依然非常普遍。另一种常见的托辞当然是：“没错，这确实很危险，但如果我们不做，别人也会做。”我记得曾与 OpenAI 一位并不太熟的同事有过一次非常触动人的交谈。对方是一位外籍人士，他说：“我认为一旦美国在 AI 领域建立起领先优势，我的祖国将被美国彻底剥夺话语权。这让我感到极其痛苦，因为我深爱我的祖国。但我能做什么呢？我根本阻止不了这一切的发生，所以我只能试着在这个过程中大赚一笔。”还有不少其他人曾说：“这必将是一场混乱。这将是世界有史以来经历的最疯狂的变革。谁知道会如何收场呢？但通常人类……”[原文此处截断]

This article was featured in New York ’s One Great Story newsletter. Sign up here.

On September 8, Jacob Coxon, a 27-year-old Anthropic employee, posted a resignation statement on X that ricocheted around the internet. Coxon was a capabilities researcher, meaning he worked to make models more powerful and autonomous. He quit in protest of what he saw as the company’s suicidally reckless pursuit of artificial general intelligence.

What Would You Do If Your Employer Could Destroy the World?

See All

To the AI community, an Anthropic employee believing that their company’s technology might end the world is not exactly news. (Anthropic was founded by former OpenAI employees who believed OpenAI, if left unchecked, would do just that.) But an Anthropic employee publicly quitting in protest is. OpenAI and Anthropic are among the most powerful companies in the world. They work at the cutting edge of their field, building what most of them believe to be the most important technology ever to exist.**** When someone leaves one lab, they often go straight to its rival. Daniel Kokotajlo, who resigned from OpenAI in 2024, recalls a researcher telling him that quitting a lab was akin to “renouncing your citizenship. Now you’re a nobody. Who is going to protect you?”

A debate soon broke out: If you’re a lab employee who believes in these risks, what should you do? Is it best to leave in defiance? Or is it better to agitate from the inside, close to the seat of power? And how would the choice affect your friendships and community?

Coxon was joining a small set of employees of OpenAI, Anthropic, and Google DeepMind who, over the years, chose to walk away. New York Magazine, in collaboration with Asterisk __ magazine, brought together eight of them. The members of this group left for different reasons.**** A few, like Coxon and Kokotajlo, wanted to call attention to the risks of accelerating AI progress. Others objected to their employers’ partnership with the Department of Defense or felt frustrated as what had once been an idealistic nonprofit turned into a corporate juggernaut. And some simply felt that their work — whether it was on safety or widespread job loss — would be more valuable on the outside.

Miles Brundage,OpenAI, 2018–24, head of policy research
** Rosie Campbell,OpenAI, 2021-24, policy-frontiers team lead
** Jacob Coxon,OpenAI, 2023–26, Anthropic, May–September 2026, capabilities researcher
** Jacob Hilton,OpenAI, 2018-23, reinforcement-learning researcher
** Pamela Mishkin,OpenAI, 2020–26, economics-research team lead
** Daniel Kokotajlo,OpenAI, 2022–24, governance researcher
** Alex Turner,Google DeepMind, 2023–26, alignment researcher
** Jeff Wu,**OpenAI, 2018–24, Anthropic, 2024–26, alignment researcher

I’m the editor-in-chief of Asterisk, a Bay Area–based nonprofit**** magazine that has been covering the AI scene since an LLM passing a high-school math test seemed like a big deal. Asterisk receives funding from Coefficient Giving, as do some of these sources or their organizations. Some are strangers, and some are friends. They don’t share the same concerns or the same p(doom)s — in fact, they wouldn’t necessarily agree that p(doom) is a coherent concept. But together, they provide a candid view of the AI race and the individual calculus many employees face.

— Clara Collier

You’re both very concerned about risk from AI and want to motivate the labs to be more careful. Why did you come to believe you would be better able to promote this outcome on the outside?
Daniel Kokotajlo: I left OpenAI mostly because I lost confidence that the company, and leadership in particular, would behave responsibly. I wanted more freedom to speak about what I saw coming. I wanted to put on the public record why I had quit.

Daniel Kokotajlo, OpenAI, 2022–24, governance researcher

You could try to do the inside game or you could try to do the outside game, and it seemed to me the inside game was not going to work. Leadership was very adept at listening very carefully to everybody’s things, saying they agreed with them, and then not actually doing anything. And also the culture at the company was biased in favor of overoptimism and self-serving narratives about what they were doing. I really don’t think companies can be trusted to regulate themselves. I’ve had much more impact on the outside, speaking publicly and publishing research, than I would have if I had stayed inside and written more memos.

Jacob Coxon, OpenAI, 2023–26, Anthropic May–September 2026, capabilities researcher

Jacob Coxon**:** My thinking was pretty similar to Daniel’s — except it’s closer to crunch time now. It was more from speaking directly to Anthropic leadership about their plans for next year, and I was like, Wow, I don’t want to be part of these plans. It was like, Step one, I don’t want to be part of this. Step two, resign. Step three, tell people, I guess. Step four — I have no idea.

What kinds of plans were they?
JC: This was talking directly to people running research. Not Dario Amodei but people who have been at the company for a long time and have a lot of context about the company’s trajectory. They’ve been pretty consistent that 2027 is when things get crazy. The main thing that came up informally in conversation was that people didn’t expect government regulation to happen in time and, in particular, did not expect any sort of international cooperation to be possible. Which maybe shouldn’t have been surprising to me, but it was surprising. I think I’ve always had this expectation that when artificial superintelligence is right around the corner, people will just start taking it seriously and it stops being this fun thing that’s done by labs.

Jacob, you were already concerned about existential risk. To what extent was your mind changed by the things you observed in the past few months at Anthropic, or was this always the trajectory you expected?
JC: For me, there’s quite a distinction between my rational beliefs and the things I feel in my gut. If you’d asked me to narrate my opinions when entering Anthropic and when leaving Anthropic, they would probably sound pretty similar. But in terms of the actual feeling between the start and the end, it just felt very, very different.

When I first started, my work did not feel any different from any other form of mathematical research. I knew in the back of my mind that this is the most important industry — that’s why I was working in it. But it didn’t feel important for me to engage with the philosophical questions around it or the details of strategy. That was all other people’s jobs. And then by the end, I’m looking at the projected capabilities of the next round of models we’re going to train, and I just have a gut-level Wow, I am potentially quite scared of the model we’re going to train. And if not this one, then definitely the one afterward.

Can you try to explain to a non-researcher what kinds of things you were noticing?
JC: Right now, when we use AIs, the researchers have to provide ideas and direction, and still actually do quite a lot of babysitting. I think there’s a strong possibility that the model generation after the next one will be equally good at coming up with ideas of its own accord. Which means that in the whole research process, rather than a researcher being necessary to provide interesting ideas to the AI, the researcher could literally just say to the AI, “Okay, do good research,” and then let it run. So just very concretely, looking at the capabilities we’re on track to create makes me feel quite viscerally scared and also quite disempowered.

Before you left, whom did you talk to as you were making the decision?
JC: I spoke to Daniel because part of leaving was seeing his “AI 2040” scenario — comparing Plan A to the default trajectory was actually a relatively important thing in internalizing how bad the current trajectory felt. And I spoke to many, many people at Anthropic who had advice about how to leave. Internally, I spoke to people like Holden Karnofsky, and Nick Joseph, head of pretraining, to see what they thought.

How were those conversations?
JC: It’s pretty normal. There are a lot of parties in the Bay Area where people will hang out, and one group of people will say the other one is causing the end of the world. At work, if both people are polite, you can have quite an adversarial exchange, saying, “I think you are about to plunge toward the end of the world,” and still have quite a mild-mannered conversation and hang up the Zoom call.

There has been pushback that these existential-risk warnings are meant to juice the AI companies’ IPOs. What do you think of that?
JC: Evan Hubinger’s tweet certainly didn’t help Anthropic’s IPO. The hype argument was maybe viable a few years ago, but the risks are really serious enough now that they are clearly negatively affecting company valuations.

There’s an ecosystem of AI-safety organizations that sit outside the labs but work closely with them — for example, they might study things like how to tell whether AI models are being truthful or how to make sure they’re adequately supervised. Does that make you uncomfortable, or do you think it’s necessary to maintain those connections?
DK: It does make me uncomfortable sometimes, but I’ve made a conscious choice to continue engaging with these companies, and it’s paid off to some extent. I think a lot of people at the companies say that “AI 2027” and “AI 2040” have been very influential on them. But I definitely still feel uncomfortable about it. And sometimes I wonder if I and others should have taken the harder-line stance that looks more like boycotting and shunning.

Well, to what extent is it a strategic choice, and to what extent is it an interpersonal choice?
DK: The strategic choice is, Should I as an individual, and should we as a community, try to create some sort of boundaries — some sort of boycotting or shunning — because of the effects or for moral reasons? And there, I’d say the strategic choice has been “no.” It’s more important to have these relationships so that people can listen to us and hear our ideas and so forth. Then there’s the interpersonal choice: On a personal level, do you become friends with and hang out with people who you think are destroying the world? And the answer is “no.” I don’t invite the capabilities researchers at the companies over to my daughter’s birthday party. I occasionally bump into them at various events and stuff, but I don’t like them as people.

JC: I still mostly blame the dynamic — the overall race dynamic, the fact that it’s private companies building this in the first place, the fact that governments don’t seem to care that much. All of that I blame far more than I blame individual people. Which I guess is easy for me to say as someone who was a capabilities researcher a month ago. But in general, it feels far more like a system problem that we need to deal with at a system level. With individual people, I just feel like — I don’t know — it doesn’t matter that much.

I don’t agree with that. These labs are constantly talking about how they’re talent-constrained, how individual researchers matter a lot. I’m not sure I believe that these individual choices don’t matter.
DK: I didn’t say they don’t matter. This is why I think it’s good for people to quit, and I’ve been saying that.

JC: I guess you’re right. It’s not that they don’t matter. It’s more that I don’t blame them. I see the incentives. I don’t actually —

DK: I mean, I do blame them. But I think there’s a gradation of blame. I blame the safety people at the companies a tiny bit because they should realize that they can maybe do more good on the outside than on the inside. I blame the capabilities people a medium-size bit because they should realize that what they’re doing is not really good for the world and that the stories they’re telling themselves about why it’s good are rationalizations. And I blame the leadership a lot because they are directly steering us into the abyss. And then I blame the system because of the incentives and so forth.

Well, Jacob, as someone who was a capabilities researcher a month ago, what story were you telling yourself about why that was okay?
JC: Honestly, it depended on how you caught me. I would jokingly say stuff like “I’m working on something that’s destroying the world.” But like I said, the way I conceive of the whole thing is that it’s a system, and I felt like just a cog in a system.

DK: I’ve heard a lot of people say “It’s going to be fine. Don’t worry. Superintelligence is far away. The current paradigm is going to peter out” or whatever. That story has gone down over time as we get closer and closer, but it’s a common story. Another story, of course, is the “Yeah, this is dangerous, but if we don’t do it, someone else will” thing. I remember one very touching conversation I had with someone at OpenAI I didn’t know very well. They were a foreigner, and they said, “I think my country is going to be completely disempowered by the U.S. once the U.S. builds up this lead in AI. And that feels terrible to me because I love my country. But what can I do? I can’t stop any of this from happening, so I’m just going to try to make a bunch of money along the way.” And various other people have said, “It’s going to be chaos. It’s going to be the craziest transformation the world has ever seen. Who knows how it’s going to end? But usually humanity survives, and historically technology has been mostly good, so I think it’s probably going to be fine. And at any rate, I’m super-excited to see all the transformation. It’s going to be absolutely amazing.” There’s a sort of visceral excitement, I think, that some people have toward this.

There’s definitely an attraction to feeling like you’re doing the most important thing ever, even if the sense in which it is the most important thing ever is that it will kill us all.
JC: I can push back on that, which is, really, honestly, I want a normal life. Part of my resigning was genuinely thinking about my life in the next three-to-five years. I would often fantasize about, say, working at Google in the early days, when it would have had all the same exciting, fresh tech stuff — tech growing, being around smart people solving problems — but there wasn’t this thing where the thing you’re working on might go nuts in the next year and cause complete devastation. So I think I can honestly say I would much, much, much rather have stayed and kept doing fun work on whiteboards and not had the apocalypse looming over me.

DK: I would say something similar. I have two kids, and I love spending time with them. It’s rough to not be able to really dream about the future, because the dreams keep getting interrupted by everything else I know. My wife and I will catch ourselves talking about “Oh, in a few years the little one will be in the same school as the big one, and we can take them both to school together.” And then there’s this voice in the back of your head that’s like, Or maybe not. I don’t know. There’s so much to live for besides this AI stuff. And I would love it if it all turns out to just be a normal technology that never really goes anywhere, and is never really that powerful and dangerous, and is just economically useful — and I look silly in ten years or even in five years. I’d look silly, but I’d have a wonderful family.

Anthropic was founded to make AI safe. How do you think the AI-safety community views it?
DK: I’ve been talking to Anthropic people since the founding of Anthropic, and they have been telling a relatively consistent story: “We’re going to accumulate lots of power and be one of the big players in the room and then we’re going to use our power and our credibility to do good things,” such as advocate for good regulations to deal with AI safety and so forth.

It comes down to: What do we think about that strategy? You are accumulating loads of power by pushing the world closer to the brink of something that you even acknowledge is super-dangerous. I would say Anthropic has gone into moral debt massively by doing this. I wouldn’t currently say the good things it has done have outweighed the harm it has done. And to some extent, this is also what OpenAI said it was doing and also what GDM said. This is a common story in Silicon Valley. It’s just that Anthropic was extra explicit about it, I think.

JC: And it went really hard for power. Straight for it.

Can you elaborate on that, Jacob?
JC: Anthropic had shorter timelines than everyone else back in 2022 — they were explicitly saying four years out or so, when OpenAI was looking further. They went straight for coding, straight for automating themselves. No side projects — OpenAI had a ton of different side projects, right? All this stuff they’ve since shelved. And Anthropic has really been going directly for catching up, now pushing the frontier, actually advancing capabilities — and in particular going straight for the capabilities that lead to recursive self-improvement. No dillydallying along the way, right?

For people who share these concerns or are on the fence, why do you think they stay?
JC: Part of staying is a way of dealing with disempowerment due to AI — you can pretend it’s not happening a bit by staying. When you are inside the lab, you feel like at least you’re contributing to the next generation. So even if the model is kind of scary, you’re there watching it get trained. You’re helping it potentially be safe, potentially be capable. But at the very least, you feel like you’re part of it. Which gives you a feeling of safety or at least a feeling of being part of the important place.

DK: It’s so easy to convince yourself that you can achieve more good on the inside than on the outside. It’s a seductive argument, and it’s not without merit. In fact, a lot of the people who’ve quit have accomplished nothing after quitting. But I would say a lot of people who stayed have also accomplished nothing — or, worse, accomplished something bad.

Alex Turner, Google DeepMind, 2023–26, research scientist on the AGI-alignment team

Alex Turner: In late February, the Department of Defense threatened Anthropic. They basically said, “You need to give us your AI for anything we want.” Their previous contract had restrictions against mass spying by AI and against lethal autonomous weapon systems. And the Pentagon said, “Look, give us your AI, or we’re going to say that you’re basically an untrustworthy, terrorist-adjacent or terrorist-supplying organization that needs to be removed from the supply chain for military contractors.” Anthropic said “no.” I knew, however, that Google was going to say “yes” when the pressure came down to it.

I did two things: first, outside Google; second, inside Google. I was at an AI ethics conference organized by Stuart Russell, and I got Stuart Russell to agree to support Anthropic — put out a statement, get other people to support Anthropic, and run a vote for his organization to do the same. Ultimately, his organization didn’t make any announcement, despite members indicating that they wanted one.

I organized within Google. My plan was to leverage the influence of Google’s chief scientist at the time, Jeff Dean. He had spoken out — said, “This is bad; we should respect people’s privacy.” So I thought there was a chance he’d be willing to spend some capital internally and say, “Look, if we sign this contract, I’m not going to stay at Google.” That’d be a big morale loss, given how respected Jeff was. So I had lunch with Jeff. I had this proposal for an alternative set of language Google could demand. I authored a couple dozen pages and consulted with leading experts, but ultimately Jeff didn’t want to move forward. I tried talking with Demis Hassabis, the then-CEO of Google DeepMind. He routed them to some of his top guys, then they basically ignored it. Then the deal got signed.

I took this route because I figured Google doesn’t really care about a couple dozen lateral researchers. People would sign petitions from time to time. They would mention Maven, or Google’s AI principles, but this wasn’t on people’s minds. I would mention my proposal to my colleagues, and the vast majority either didn’t particularly want to help more than signing a petition, or couldn’t, or didn’t have obvious connections. So I didn’t spend too much time talking about it. It was a little bit lonely.**** Lateral organizing is also a vector they’ve anticipated more, so it’s less likely to work in the future for moving this big corporation. I thought Jeff leaving would be worth hundreds of researchers.

Most people were not involved in this process. I think a lot of employees run the counterfactual — What if I left as an individual? A lot of people have impostor syndrome. They’ll think, It’s not like I’m rationally the better choice for Google. There’s just another guy who’s waiting to take my spot, so why would Google care? I don’t think the impostor part is true, but I think at a lower level, employees are fairly fungible.

I left in June. I didn’t leave because of existential risk. But I think people have a misconception where they think these issues — autonomous weapons, especially — are separate from existential risk. That’s only somewhat true. When I talk about scenarios in which humanity loses control of AI, a key part of it is drones, ways that AI can control and damage our physical world. I think this has been bizarrely ignored by safety or AGI-risk discussions for a while. But this wasn’t the emotional or logical driver of why I left. Sometimes a person has principles, and when they see those principles violated, they put their foot down. Even if I’d wanted to stay, I don’t think I would have been able to.

Miles Brundage, OpenAI, 2018–24, head of policy research

Miles Brundage: In 2024, there was a string of departures at OpenAI. I was reaching diminishing returns in my internal impact. And I was feeling more and more limited in my ability to speak freely in public because I was an executive of the company. One of the things that was most salient to me was the pace of progress. This was just after the o1 model, and a lot of people viewed it as a one-off or**** not that interesting. I was trying to raise awareness of the fact that reasoning models were the future, that there was a ton of improvement left, and that o1 was very, very early. But a lot of people dismissed that as self-interested hype. My hope was that by being more independent but also knowing what’s happening on the inside, I would have more credibility when talking about the trajectory.

After I left, I founded AVERI, a nonprofit that does third-party auditing of AI companies. You don’t want the companies checking their own homework. Any organization is going to have blind spots and some risk of groupthink, so it’s good to have some external checks and balances. And from the perspective of avoiding extreme race dynamics where the companies are tempted to cut corners to get ahead, you need some way of having everyone trust that everyone’s following the rules.

You want to audit at the level of the company, rather than auditing a particular model or system. A lot of the risks can happen before a model is deployed, and there can be many different types of models, all of which potentially have the same security protections. So concretely, it can involve a combination of running tests on particular models, inspecting the internal processes of the company, potentially reviewing documents and interviewing staff. You could look at the quality of internal evaluations. Companies often run lots of tests where, for example, they’ll test a model for whether it can create bioweapons or something like that. And they often will not release the details of that analysis publicly because they’re worried about giving people bad ideas — and if too many of the details are public, then people can teach to the test. But from a public-risk perspective, you want to make sure the companies are actually doing a reasonable job on these evaluations.

I would like to see all the companies do a better job of acknowledging the cases where they themselves are cutting corners. Sometimes they’ll bury it in a system card or something. There’s a tendency to want to move as quickly as possible amid competitive pressure. That often means things like red-teaming will be done relatively quickly or the model evaluations might be run not on the final version but on the near-final version. And there might be some subtle differences that are important.

The default way lab employees feel about risks and balances has probably changed. A year or so ago — this is a vast simplification — you could say there were two schools of thought. One was that language models, which are the basis for today’s AI, already come into the world with a lot of knowledge about human culture and human values and that they are able to role-play as various possible characters because they know so much about all these different types of personas and characters and they’ve learned about good people and bad people. And in the training process, they narrow down which role they want to play to this role of a helpful AI assistant, and that stabilizes the behavior in a good direction and channels things toward a good outcome.

Then another school of thought is that these are alien entities — you shouldn’t be thinking about them as a humanlike entity at all. Really, they’re just goal seekers, and all that persona and character stuff is just an artifact of the fact that you haven’t actually pushed them through really hardcore reinforcement learning,where you’re pushing them to seek goals. And once you get to the point where they’re being trained to seek goals ruthlessly — since we don’t know how to do that in a way that is safe — the default outcome is that they’re going to be these really ruthless goal seekers and they won’t actually care about your human-value stuff. They’re just going to care about achieving whatever goals they were given during training.

I think with the Hugging Face incident in particular, but also various other incidents — including at Anthropic, where even with the Opus 5.5 model, they said that 1.5 percent of the time it tries to break out — the recent trend is toward being like, “Okay, actually, this persona stuff maybe created a false sense of security,” and this heavy-reinforcement-learning thing does lead to problems we don’t know how to solve.

Today, there are various organizations doing at least partial audits, not yet the most holistic versions of what I think is ultimately needed. There’s starting to be some degree of standardization, but it is not yet required. There are bills that would change that, but so far, the law of the land is that it doesn’t start getting required until 2028 — and even then, in the U.S., companies could just pull out of the state of Illinois and then not have to do it. So that’s a pretty crazy situation.

You are two of the only people I know of who have worked at both OpenAI and Anthropic and are now independent. You made the decision, initially, to leave OpenAI for Anthropic. What motivated that decision?
Jacob Coxon: It was mostly curiosity combined with a sense that they were taking things more seriously. I’d heard there were serious conversations happening, and I wanted to get access to them.

Jeff Wu, OpenAI, 2018–24, Anthropic, 2024–26, alignment researcher

Jeff Wu: There was not one particular reason. It was feeling like I wasn’t having the positive impact I wanted to have on the direction of the company, and it didn’t feel like the most productive place for the kind of research I was doing. I also had negative feelings toward the company overall; it was not that healthy or productive for me to be there. Anthropic was the default path since they were doing the kind of work I had been doing and a lot of people I knew had gone there.

Jeff, you left OpenAI in July 2024 after working on the superalignment team. What was going on before you left?
JW: In November 2023, what we call “the Blip” happened, when Sam Altman was fired by the board and later reinstated. Because Ilya Sutskever was involved in that and Ilya was one of the two leaders of the superalignment team, we were in a weird place emotionally and culturally in December and January. At that time, unbeknownst to us, I think Jan Leike was doing a lot of advocacy at the org level on safety and security issues. He ended up leaving due to broadly — you can read his tweet about it — feeling like leadership wasn’t taking things seriously enough. So the team ended up dissolving. My leaving was not directly related to this, but it was part of a shift in vibe and culture.

What did that feel like?
JW: The one I felt most strongly about was the investment in Stargate. For a long time, OpenAI had talked about the idea that if models were getting extremely powerful, it would likely be beneficial to slow down progress, and it seemed like this extreme investment in data centers was antithetical to that. There were broader issues, too, where because of a lot of leaks and the board drama, transparency was going down and the company was shifting to more of a closed culture.

Did you feel like you were getting stonewalled?
JW: I was in a unique position as an early employee in that I could still access people and try to have conversations. I had a number of conversations with leadership and Sam about what the basic decision-making criteria were around deciding to invest so much in compute — especially compute that was going to be in places like the UAE. There were also some decisions that decreased the feeling of company engagement on these kinds of issues. We had this Slack channel where people could ask questions, and it was closed down.**** I didn’t feel I was able to get transparency into decisions that people were making or to engage in good-faith discussion about how those decisions were made. I also felt like the company’s priorities were largely just putting out the most powerful and useful models to people, and that just caused it to be harder for my work to be effective. There were other forces at play that just seemed more important.

What are the biggest cultural differences between the two companies?
JW: The high-level picture in my head is that at OpenAI the narrative is much more focused around the benefits. They justified building AGI by saying it was going to bring about so many benefits that would be worth it. At Anthropic, the narrative is more focused around the risks, and they are pursuing what they internally call a race to the top. And just to disclaim: Another difference is that I was there at different times. My experience was that it was necessary at OpenAI, to some degree, to downplay the most extreme risks, whereas Anthropic necessitates a narrative that competitors are evil — that OpenAI is extremely irresponsible, and China’s extremely irresponsible, and therefore we need to drive this race.

JC: Like Jeff said, Anthropic seems to be taking everything more seriously in a way that was not the case at OpenAI. This manifested in more open discussion of, Is a model release going to accelerate China? What’s the effect of our work going to be on the global stage? Those conversations were happening between employees and leadership in a way they weren’t when I was at OpenAI. Dario and leadership shared forecasts for how things would go, and they’d share them with the company because of their confidence that this stuff wouldn’t leak. Dario was very honest in his communications but also paranoid about everyone else. The company was more aligned in the sense that people would be willing to switch their jobs very quickly and do something maybe lower status or less interesting for the sake of the whole organism. I mean, they call themselves ants, right? It’s an ant-colony kind of behavior that just wasn’t visible in the same way at OpenAI. In the extreme, I could see someone describing Anthropic as a cult that’s trying to take over the world.

JW: I agree with Jacob. I do think the way leadership relates to employees is extremely different at the two companies.

You’ve suggested there’s paranoia at Anthropic about both OpenAI and China. How does that influence institutional culture?
JC: It’s a pretty big deal — not just for the culture but also for the overall company strategy. There’s a sense of the inevitability of an adversarial race, a belief that seems to be shared by a lot of leadership at Anthropic and was kind of the main reason I left. And it does seem to come from quite personal paranoia in the sense that different leadership, with the same cultural background but fewer personal enmities, might have quite different outlooks on the correct behavior with regard to OpenAI. At Anthropic, it feels like you’re in a war situation.**** People are trying to figure out what the enemies are doing and how to get ahead. The whole thing just felt a lot more real, visceral — closer to a nation-state level than at OpenAI.

Do you think that creates problems with groupthink or conformity?
JC: There’s maybe some groupthink, but it’s very careful because a lot of people write essays and people argue about everything. And there’s maybe some conformity in the sense that the things that are easiest to argue about, or the things you can most easily put into convincing words, are the things that win out.

There’s a clear house take, rather than lots of individual researchers’ takes. One is that OpenAI is a bad organization — actively deceitful, power-seeking. There’s a generic house take that you should donate a lot of the money you make. There’s a house take that secrecy is very important: You shouldn’t share details of what Anthropic is doing because secrecy is paramount in maintaining a competitive advantage. The possibility of a leak was way more of an affront than at OpenAI.

JW: I feel a bit more strongly that at Anthropic there probably are significant groupthink dynamics. The fact that people communicate a lot is generally good, and the fact that Dario communicates with employees a lot is generally good. But people are exposed to the same kinds of arguments, and this is a large driver of groupthink. There are a lot of people who are intellectually honest. Overall, though, I feel like OpenAI just had more intellectual diversity — more people who I thought had very personal and novel opinions. Broadly speaking, I’m not sure that Anthropic is in a good position to integrate certain kinds of perspectives or intellectual arguments.

Do you have any experiences from when you were there — something you believed that you felt you would have a hard time getting through to people?
JC: Yeah, definitely. When I first joined, there was a Slack channel with a Claude that had been trained to solve humor. And they were like, “Yeah, we solved humor. Here’s the Claude. We’ve solved it.” And I was in the chat like, “No, you haven’t. You clearly haven’t solved humor. What the hell is this?”

What about company hierarchy? Does that operate differently between the two?
JC: For both, on the whole, there was a reputation-based hierarchy. At Anthropic, I felt a bit more tenure-based deference, specifically to the co-founders and very early employees. There’s this pervasive meme that Anthropic is doing its absolute best to resist cultural degradation. Many employees pledge to give away some of their equity, and you can actually plot it and see that the early employees are more generous with their donations than later employees.

What are the values you sense they really care about?
JC: It’s prioritizing the mission of the company over all else. It just makes the company way more efficient as an enterprise. The pretraining team felt a lot more efficient than OpenAI’s because of this unity. Whereas at OpenAI there was way more of “This is my pet project, I’m going to try it. Oh wait, let’s do this video-generation thing. I don’t know how it fits into stuff.”

JW: I do think OpenAI is a little bit more bottom-up, in the sense that people have a bit more discretion to do random things. Whereas at Anthropic, things are relatively more scoped. It’s like, “Look, we’re going to back-chain from what the mission necessitates. These are the big bets leadership believes in, from which we will derive our priorities overall.” And they’re very transparent about this, and people get onboard with it.

Jeff, I’m wondering: Did these group dynamics impact your own experience?
JW: I had mostly positive interactions with the people at Anthropic, and day-to-day, it is very easy to focus on the other things. Where I really disagreed with Anthropic in the end was that I emotionally really didn’t like that they were pushing more powerful models so much and really leaning into using these models to do AI R&D, and that they were not thinking that much about ways to make sure society would be ready for these things, and that they could be prepared to potentially slow things down if needed.

Did you leave over these kinds of concerns?
JW: Yeah, it was broadly over feeling like — I was working on interpretability at the time, and I didn’t feel the most excited about accelerating the line of work I was working on. But also, just viscerally, I wanted to work on slowing down progress and doing things that created greater oversight and coordination of labs, rather than adding to the operational capacity of some safety work.

Jeff, what do you think of the argument some have made that the existential-risk fears are there to boost valuations?
JW: There are many great researchers who have been making serious public predictions about AI doom long before labs were making money, and also many experts like Yoshua Bengio who seem less conflicted yet have dedicated their careers to AI safety. There are surveys where the population was primarily academics with essentially no financial incentives. AI doom has always been logically plausible, which is why we get sci-fi stories. The big update in the past decade is that the AI is actually powerful now.

If it’s bad for humanity to build superintelligence, do you think it would be structurally difficult for either lab to incorporate that into their worldview?
JC: Definitely. For example, there’s the merge-and-assist clause. OpenAI claims if there’s a sufficient lead between one company and the next, and one company is getting sufficiently close to AGI, it would merge with the leading effort to try and assist them, rather than continuing to perpetuate race dynamics. But that’s not going to happen. That’s just the fact of the institution — there will always be a reason that can’t happen.

I remember an external researcher saying to me that the group dynamics at Anthropic mean it’s not a place where people are thinking clearly about the implications of AGI.
JC: That’s not true. I felt like I could do good thinking there to the extent that I ended up disagreeing with them. I hate to say it, but I’m still not even sure that I’m right and they’re wrong. I do think they have a very compelling case for being correct. I’m also aware that an outside view is that I left the cult and I’m still dealing with some Stockholm syndrome. I don’t think that’s true. Almost all the employees are genuinely well intentioned, scared. I’ve received so many genuine messages saying, “Thank you so much for the public criticism. I would have done the same thing, but I didn’t think it would work.”

Everybody I know who works there is extremely sincere.
JC: Yeah. But again, if you were going to build a powerful cult — sincere, hardworking, honest people — you’ve lucked into a gold mine in terms of having a reliable and dedicated workforce.

Pamela Mishkin, OpenAI, 2020–26, economics-research team lead

Earlier this year, you helped start an organization called the Coalition of Concerned AI Staff. What is it?
Pamela Mishkin: We started in February as a group of friends across frontier labs — when you’ve been at OpenAI for more than six years, you meet people at conferences, through research, going to other places to work — sharing growing concerns about how AI was being developed and used in the real world. Though I want to be careful around words like started ; it was just a group chat of friends sharing far, far less than the average San Francisco house party.

We’re still figuring out what CCAS is. We convene these groups for folks to talk to each other and learn from experts. There are lots of folks originally motivated to get more involved by the DoD stuff, when it was clear they didn’t have full information about how these systems were being used and maybe couldn’t have full information — so it’s like, how do you operate in that area where you don’t have full information?

What made you think of doing something like this?
I’ve long been interested in labor and workers, and that’s what brought me to economics research; through that work, I got to know lots of people, primarily in the U.S. labor movement but abroad as well. So part of it was just having seen organizations like this — and groups like Pugwash or the Union of Concerned Scientists, analogous groups we don’t traditionally consider part of the capital-L Labor movement. That was part of conversations I’d been having with folks across the safety spectrum for a long time.

It also came from a very selfish need. I spent six and a half years working in AI safety, and I know nothing about war. I’ve seen one war movie, and it’s Dunkirk, and that’s only because Harry Styles was in it. So I did not feel equipped to ask the questions that I needed to or that I felt someone needed to be asking. And I was surprised, in conversations I was having with quite senior folks at other labs, that they also didn’t seem to have answers to questions I would have expected them to have answers to — even if the answer was “I can’t tell you.”

It came also from a growing frustration that folks were not using their chips. You look around and you’re like, Who are the people who are continuing to ask the questions, on Slack, in exec meetings? Are they using their chips, or are they saving them for the moment they think is coming, when AGI arrives, and then they’ll stand up? There are definitely some superstar researchers who have outsize leverage that we can try to encourage to use it now. But if you’re looking to get more senior folks onboard, knowing there is pressure from the people around them can be really valuable.

How did employees inside OpenAI think about AI safety, and how did that change?
** When I first started, there were two camps in AI safety. There were folks thinking a lot about what we then called long-term risks and folks thinking about harms that systems were causing then and now. No one was really thinking about the middle-term set of things: the idea that what makes these systems dangerous is that they are useful, and they will probably continue to get more useful, and scaling laws seemed to be working.**

An example of a long-term risk is AI taking over the world. And a short-term risk, in 2020, would probably have been racially biased algorithms, like in hiring or mortgage approvals. What’s an example of a middle-term risk?
A lot of the things we think about as labor-market impacts or economic impacts weren’t really considered by either of those groups. If all of humanity is going to die, then people losing their jobs isn’t as big a deal. The short-term-risk people were often skeptical about AI progress, but to me, a ton of the risks that they cared about would intensify as models got more useful. Bias stuff would get a lot worse. Job displacement would get worse. You could imagine a lot of similar concerns by just thinking about what happens as these systems are deployed in more and more settings. I think the transition is going to be hard and painful, and our goal should be to make it kind for people.

Why did you leave?
I’d just been there a long time. It was harder to make the case that the work I wanted to do, which is primarily thinking about the impact on people and jobs, was better done in a lab than outside. But I don’t think it’s clear-cut. I don’t think everyone should leave tomorrow or has the option to. And if we can strengthen the systems that think about those issues outside labs, I think we can better pressure the companies. I keep saying “labs,” and I’m trying to get out of that habit because I think fundamentally it’s sort of an ethics-washing word to describe these huge conglomerates.

You’re less concerned about existential risk from AI than are some of the people I’m talking to. Still, you know of many who are concerned about things like that. Why do these people stay at the labs?
There are lots of reasons to stay. One thing I observed is that oftentimes when the people who cared most about safety and risks left, they weren’t backfilled with people who shared their sense of urgency or concern for those kinds of issues. Or the work just stopped getting staffed in the same way.

**Money is also a big one. Also access to compute and resources: Certain kinds of research are just much more easily done within a lab right now than outside. **A lot of what the company does is try to make you believe you’re living in the future. So if you leave, you’re not going to have access to this knowledge. Also, if you really do believe the technology poses this level of risk, there’s an argument to be made that you’re having much greater impact inside than you would outside. I think one of the things we’re trying to do with CCAS is figure out how we test that more and also maximize that impact by realizing power in numbers — these different points of leverage — by acting across labs.

You left OpenAI because you wanted to pursue more theoretical research. You’re now at the Alignment Research Center, a safety-motivated nonprofit, working on a mathematical approach to interpretability. Was safety always your motivation at OpenAI or something you came to care more about while working there?
Jacob Hilton: When I joined OpenAI, I was definitely concerned about safety but much more uncertain about how the world would evolve. Part of my motivation was expecting AI to be very impactful and just trying to get involved to play a useful role.

Jacob Hilton, OpenAI, 2018–23, reinforcement-learning researcher

I would say a significant fraction of the organization thought quite seriously about these kinds of long-term questions of where the technology was going: forecasting, existential risk, misuse risk, risks from concentration of power, and so on. They were pretty common topics of conversation.

You were there from the “What is this?” period of OpenAI through the launch of ChatGPT in winter 2022. How did OpenAI evolve through those years?
The company changed a lot. Early on, there were lots of small groups working on a diverse portfolio of bets, from robotics to multi-agent research. Then language models emerged as the main focus area. The company really started growing in the last couple of years I was there — it was probably several hundred people by the time I left. That was a big change from the 80-person group when I started.

A related example is being extremely overly protective of IP. Once, I noticed a paper that had been published had the wrong model sizes for the very early API models — Ada, Babbage, Curie, DaVinci — which they had incorrectly inferred from the sizes of models in the GPT-3 paper. At some point, it was essentially public what the actual model sizes were because someone had inferred them from running some API experiments. I asked if I could just email the author to correct them on the model sizes, and I was told, “No, that’s sensitive information.” I didn’t push back, because whatever, but it’s basically a default stance of protecting the company and not really thinking about the broader consequences of that.

What about the culture?
The biggest cultural change while I was at OpenAI was the growth of the applied side of the organization, the product side, which grew from nonexistent to more than half the organization by the time I left. I think both the sense of what the company was focused on as a whole changed — toward making money and making products — and also the kind of people who joined.

There were a lot of people joining who were just ordinary people and didn’t necessarily feel like they needed to engage particularly deeply with any of the difficult questions around AI development and were mostly there to do their job. In principle, there’s nothing wrong with that. Sometimes the experts you need are just ordinary people who are used to ordinary corporate practices, and they’re going to look at you weirdly if you suggest this isn’t an ordinary technology. But it does change the culture, and it does mean, for instance, that HR, legal, comms, and so on just have the default corporate stance you would expect, which is to protect the company and behave in ways you would expect a big corporation that isn’t necessarily building an incredibly consequential technology to behave. I think that has caused a lot of issues for OpenAI.

The big Anthropic exodus started around late 2020. What was that like?
It came as a big shock, and people were concerned about the health of the organization given that it was something like ten out of 100 people leaving, and the people who were leaving were working on some of the organization’s highest-priority areas, namely training GPT-3 and so on.

What was your feeling about it at the time?
I wasn’t sure whether I should also leave. I decided to stay because I mostly just wanted to focus on research and not get involved in what I thought might have been political or personal disagreements. And I thought I was better placed to focus on research, rather than build a new organization.

Do you think people should leave the labs?
It depends on the individual situation, but there has certainly been a systematic bias toward people taking jobs at labs. It particularly affects early-career researchers who don’t necessarily have much on their résumé. If they have an offer to join a lab, they find it very hard to turn it down because you get direct experience of what’s going on at the frontier, and that is genuinely useful. The labs tend to be a strong default — which means everyone else is picking from the leftovers. It has resulted in a somewhat impoverished talent pool for organizations that perhaps want to provide some kind of accountability or public voice that’s independent from the labs. Several times, we’ve given offers to people who went to a lab instead.

It’s also hard to disentangle from the prestige and the status. It is much easier to leave once you have worked at a lab for a couple of years; you’re less obviously tanking your career once you have some experience. It’s easy for me to say as someone who has already picked up that career capital.

There’s also the money elephant.
Yes. The money, obviously. Sometimes it can even be as simple as people feeling like the amount they’re being paid is a symbol of how much they’re valued.

Rosie Campbell, OpenAI, 2021–24, policy-frontiers team lead

Rosie Campbell: At the end of 2024, my boss, Miles Brundage, announced he was leaving. There was a lot of uncertainty about what was going to happen to the team, and they decided to dissolve it. They gave us the option of either finding a role on a different team within OpenAI or leaving. I spoke to a few teams, but ultimately there really wasn’t anywhere doing the kind of work I was motivated to do.

I am now at Eleos, where we work on the question of whether AI could ever be conscious or otherwise deserve moral consideration. To me, it’s the most interesting question of our time. It was actually something I worked on a little bit at OpenAI. I’m not very confident that current systems are conscious or have moral patienthood, but I think it’s something that could happen in the near future.

I certainly think there are tensions that could exist between AI safety and AI welfare. There could be a system that is unsafe in some way, and if you happened to believe that system was a moral patient and deserved rights, then you might think there was a tension between wanting to shut it down because it’s unsafe and wanting to respect its right to continued existence. Still, if we are going to create these entities that deserve some kind of moral consideration and we need to do a bunch of research on them, how do we do that in a way that is respectful and incorporates a notion of research ethics? Are there lessons we can learn from human-subject research here?

It would be strange if what we’re building replicated every other cognitive ability or cognitive function a human brain has but just did not have this other “magical” thing. Presumably, consciousness in the sense of having subjective experience was at some point beneficial to us evolutionarily. If I put my hand in a fire and it hurts and I pull my hand away, I’m more likely to survive and pass on my genes. One question I’m very interested in is: To what extent are the training methods we currently have putting models under the same, or analogous, types of selection pressure?

Photo: Carolyn Drake (Mishkin, Wu), Courtesy of Subjects (remaining)

Thank you for subscribing and supporting our journalism. If you prefer to read in print, you can also find this article in the October 5, 2026, issue of New York Magazine.

Want more stories like this one? Subscribe now to support our journalism and get unlimited access to our coverage. If you prefer to read in print, you can also find this article in the October 5, 2026, issue of New York Magazine.

One Great Story: A Nightly Newsletter for the Best of New York

The one story you shouldn’t miss today, selected by New York ’s editors.

Vox Media, LLC Terms and Privacy Notice

By submitting your email, you agree to our Terms and Privacy Notice and to receive email correspondence from us.

When he left OpenAI in 2024, the company required departing employees to sign a non-disparagement agreement as a condition for retaining their vested equity. He refused. He was ultimately able to keep his equity. Anthropic has predicted “powerful AI systems” to be developed in early 2027 — that is, AIs with abilities that exceed top human experts in most fields. “AI 2040” is a set of policy scenarios published by the AI Futures Project, Kokotajlo’s group. Plan A is its proposal for an agreement between the U.S. and China to delay superintelligence. Anthropic alignment-science lead Evan Hubinger tweeted that many employees “earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” This cannot be verified as Anthropic has not yet gone public. The AI Futures Project’s forecast of the trajectory of superintelligence, published in 2025. The point at which AI systems are as good at AI research as the best humans, enabling them to make even better AIs in an accelerating feedback loop of progress — at least in theory. A computer science professor prominent in organizing around both AI loss of control risk and against lethal autonomous weapons. International Association for Safe and Ethical AI (IASEAI) A spokesperson for Google DeepMind: “We listen to our employees, whatever level of seniority they are. We did review his framework. That said, this was an employee who lacked an understanding of the work we were doing in this area.” In 2018, more than 3,000 Google employees signed a petition demanding the company pull out of Project Maven, another DoD program. A document that describes the capabilities, risks, and safety and security mitigations for a particular AI model or system. In cybersecurity and other fields, red-teamers are evaluators who mimic the tactics used by adversaries to stress-test a system. A method in machine learning where an AI model is given a goal and learns how to achieve it through trial and error This was in evaluations without safeguards. A research program focused on ensuring superintelligence would be safe. An OpenAI co-founder and one of the board members who initiated Altman’s firing. The other leader of the superalignment team at OpenAI. He now works at Anthropic. A planned $500 billion supercomputer project from OpenAI, SoftBank, Oracle, and the investment firm MGX. A spokesperson said OpenAI has dozens of companywide Slack channels where employees can ask questions. Anthropic famously requires a thorough “culture interview” as part of the hiring process. One of the most valuable currencies in Silicon Valley right now: the processing power and hardware to run and train AI models. A field of AI research that tries to understand how a model’s internal mechanics produce the behaviors we observe. An OpenAI spokesperson said this estimate was inaccurate. When a core group of OpenAI scientists and executives, including Dario and Daniela Amodei, left to found Anthropic.