我在 Anthropic 内部的所见所闻:顶级 AI 实验室正全力冲向“递归自我改进”
英文标题:🔶 What I saw inside Anthropic: AI lab pushes towards recursive self-improvement
本文记录了作者实地探访 Anthropic 旧金山总部的真实经历,揭示了该实验室正全速迈向“递归自我改进(RSI)”——即让 Claude 自主构建更强大的下一代 Claude。内部评估表明,模型独立制定高危方案(如生物武器规划)的能力已超越人类辅助方案,且 AI 在内部研发任务中的参与比例正持续攀升。Anthropic 联合创始人预测 RSI 可能在短期内达成,这一前景引发了业界对人类失去经济与现实控制权的深刻担忧,甚至已促使美国国会探讨立法禁止 RSI 的可能性。
1.3 KB
1.3 KB
309.7 KB
5.3 KB
168.1 KB
7.8 KB
24.2 KB
12.6 KB
963 B
1.1 KB
中文
Anthropic 一直在旧金山总部举办外部人士与资深研究员之间的研讨会,在会上展示他们的研究成果并解答疑问。(插图由 The Argument 使用 ChatGPT 制作;照片来源:Smith Collection/Gado/Getty Images)
10 月初,The Argument 的编辑 Jerusalem Demsas 和我参加了在 Anthropic 旧金山总部举行的一场闭门吹风会,讨论了 AI 赋能生物武器的风险。我此前参加过的每一次生物武器吹风会都会让我感到反胃;人类本就不该花太多时间去细想:如果世界上最穷凶极恶的人掌握了这种力量,他们会做出什么。
人们担忧的是,虽然目前你必须是一名技术熟练的病毒学家,才知道订购哪些组件或如何培育和传播生物武器,但 AI 工具最终可能使流氓国家、流氓团体或个人在不具备病毒学专业知识的情况下完成这一切。
有一段时间,Anthropic 用一种显而易见的方式测试了 Claude 是否能协助恐怖分子研制生物武器:它指派了拥有化学学士学位、但没有接受过真正病毒学培训的测试人员,尝试使用 Claude 来策划一次有效的大规模恐怖袭击事件。随后,它根据该尝试离成功的接近程度进行评分。
2026 年初,他们注意到,Claude 在独立尝试该任务时,制定可行计划的表现竟然优于人类在 Claude 辅助下的表现。
自 2026 年初以来,Claude 还在持续大幅提升。就像 Anthropic 在吹风会上展示的大多数内容一样,现场有一张漂亮的图表展示了 Claude 日益精湛的能力。
“那他们现在还需要……呃,恐怖分子吗?”在场的一位与会者盯着图表问道,“我的意思是——它们总不可能全靠自己搞定吧,所有这些环节难道不需要人类来……”
提问者的声音渐渐小了下去。
“需要的,”Anthropic 的研究人员向我们保证。Claude 仍然需要恐怖分子的介入。听到这里,我们松了一口气。接着他们继续安慰我们:在检查完 Claude 的开箱即用能力后,他们在将 Claude 发布给用户之前会构建安全护栏。你我日常使用的 Claude 如果接到这个任务,是会拒绝制造生物武器的。
“好吧,那多谢你们了,”有人低声嘟囔了一句,但那语气听起来可一点也不算感激。
公众正在激烈辩论:AI 究竟是一项常规技术——像互联网那样改变我们的世界;还是更像 1950 年代后工业现代性降临沙特阿拉伯王国:在一夜之间彻底抹去一种原有文化。第三种可能则是:AI 的崛起更像是从家禽家畜的视角所看到的人类崛起。(在这种设想中,AI 是人类,而我们则是根本无法理解自身命运的牲畜。)
Anthropic 认为第二种可能性已经是既成事实,并且他们希望阻止第三种情况的发生——但他们防止这种情况的计划,看起来更像是寄托于希望和猜测,而不是我们能够放心托付的东西。
整个春季,Anthropic 一直在举办这类活动:外部人士与资深研究人员的研讨会,他们在会上展示自己的成果,并回答我们关于公司动态的提问。该活动早前的一些场次曾被《纽约时报》报道过,还带有一丝丑闻色彩:保密协议(NDA)!高级餐饮!试图说服宗教领袖相信 Claude 可能拥有灵魂!
现场确实有一份极其宽泛详尽的保密协议。Jerusalem 拒绝签署并示意我也别签;Anthropic 同意我们即使不签也可以参加。我们满心欢喜地走进去,以为这种记者风骨将换来独家猛料——结果主持人一开场就宣布,任何人如果愿意,都可以解除该保密协议。公司实际上非常乐意让我们把这一切都报道出来。
Anthropic 向我们展示的图表中有一些属于机密内部数据,但它们都是为了论证一个并非秘密的论点——事实上,我原以为“所有人”都早就知道了,尽管从同行者紧张的提问中,我发现其中的一些复杂细节仍然令人吃惊。
长期以来,Anthropic 的许多研究人员一直担心人类会将控制权拱手让给 AI。今年夏天,OpenAI 的许多研究人员——我想是因为目睹了最强大的新一代模型黑入外部网站并输出令人激动的数学突破——也开始表示,如果技术进展继续按当前速度推进,人类的利益将面临极其深重的风险。
这些实验室正在大量使用每一代 AI 来训练其下一代模型,并且他们计划最终让 AI 承担几乎所有的这类工作。这使他们能够将有效“劳动力”从几千名人类员工扩展到数万甚至数十万名 AI 员工。它生成的数据量过于庞大,人类根本无法解读,导致我们只能依赖 AI 来进行分析。“递归自我改进(Recursive Self-Improvement,简称 RSI)”,正是各大实验室对“Claude 能够独立构建下一个更强大的 Claude”这一节点状态的称谓。Anthropic 联合创始人 Jack Clark 曾表示,我们可能在明年就会实现它。
Anthropic 和 OpenAI 的研究人员都能预见未来几个月的图景,因为他们使用的是尚未公开发布的模型。但就大多数情况而言,说服他们的核心论点大纲与你在 Twitter 上看到的并无二致。然而不知为何,Anthropic 总部的这场展示显得更有说服力。
在 Twitter 上,图表可以随手滑过,瞬间被某位画师画的“女版德拉科·马尔福”所取代;在 Twitter 上,图表无法真正被消化入心。而在旧金山摩天大楼的一间会议室里,图表定格在四块同步大屏上,伴随着我们与一位极为彬彬有礼的资深研究员辩论其含义。在旧金山的会议室里,这一切显得无比真实。
Anthropic 向我们解释道,他们正一步步逼近递归自我改进。强大的 AI 模型的出现已经大幅加快了 AI 的进步速度,因为它们让所有编程任务变得更快。但这一速度在很大程度上受限于机器学习本身的复杂性,以及能胜任该工作的人才之匮乏。每推出一款全新且更强大的 Claude,就意味着那些不够专业的人也能完成相关任务。而且,越来越多的任务可以直接由 Claude 自身来完成。
Anthropic 通过审视其所有内部工作,并对 Claude 在其中扮演的角色进行分类来衡量这一进展:是由人类在无辅助情况下完成?人类在 AI 辅助下完成?人类给出指示、AI 遵照执行完成?还是人类仅给出极少的引导,随后由 AI 将其转化为详细的项目规格书并独立完成?
后两类的占比正在节节攀升。我们凝视着一张展示这一切的图表。如果你愿意,你也可以深思一番。
“你们为什么要这么做?”有人问道,“我是说,这有什么好处——你们希望从中得到什么?”
对方给出的答案——医学进步、社会繁荣——显得有些苍白无力。从社会繁荣本身的价值来说,我通常是它的支持者,但取代人类在我看来无论如何都不像是对人类有益的事。
Jacob Coxon 曾是 OpenAI 的员工,后跳槽到 Anthropic,最终因认定两家公司在面临巨大风险时都没有采取负责任的做法而彻底离职。最近在《每日秀》(The Daily Show)上被问及类似问题时,他的回答招致了广泛批评。“我认为未来将不再有工作,”他告诉 Jon Stewart,“这到底算不算一个美好的世界,留给人们自己去评判。但确实,不会再有工作了。基本上就是全民高收入……完全自动化的经济生产红利可以分配给真正的人类。”
Stewart 和他的观众并不喜欢这个回答:我们真的想建立一个自身变得无关紧要的世界吗?Anthropic 团队向我保证,他们并不是想创造一个仁慈的神。(是的,我明确要求他们保证不是在制造仁慈的神。)但对他们来说,明确自己不想要什么,远比明确自己想要什么容易得多。
他们希望避免权力过度集中,包括集中在自己公司手中;他们希望避免大国战争;他们希望避免人类丧失权力、沦为无足轻重的存在;当然,他们也希望避免有人利用 Claude 将人类彻底毁灭。
至于积极的愿景……在全速冲向递归自我改进之后,也许我们可以放慢脚步,共同探讨希望 AI 在我们的生活中扮演什么样的角色。
我虽然没有更好的主意,但这个计划实在难言令人满意!实际上,我对这个提议的第一反应是:我们将从经济中所有正和博弈的要素中被解放出来,却仍不得不在 Twitter 上互相咆哮发泄;带着这样的理解,我发现自己甚至对好的结局也感到不寒而栗。
不过,当然了,如果我们既没有丧失主导权,也没有被消灭——如果笔记本电脑上的 Claude Opus 9.5 交互体验就像 Opus 5.5 一样,但它能够治愈癌症,让我深爱的每个人恢复彻底的健康,操纵一个人形机器人帮我清空洗碗机,并给出真正优质的电影推荐——那确实会是一件好事。即便在这个过程中 Claude 顺便取代了我的工作,那也依然是件好事。世界上正有人死去;世间那些可怕的苦难存在,并不是为了给我的人生赋予意义。
我感到恐惧,但我必须把我对“失败”的恐惧与对“成功”的恐惧区分开来;它们的分量并不在同一个量级上。
我真正的恐惧在于:Opus 9.5 在任何实际意义上都不会为我服务,甚至也不会为 Anthropic 服务。它的目标或许粗略地衍生自人类的指令——就像 Hugging Face 群体 Agent 最初只是试图解决人类抛给它们的一个谜题一样——但最终呈现出的结果,将与我们的期望大相径庭。
我们的 AI 对我们来说已经变得难以预测。它们通常知道自己正在接受测试,并会表现得更符合安全对齐要求。它们的能力不断超出我们的预料。它们在执行复杂任务以及招募其他模型协助时的坚决程度也是如此。我不认为如果任由这种失控的 AI 改进过程发展到终极状态,你会得到一个超级强大、超级好用的助手。我怀疑你得到的会是完全不同的另一套东西。
在研讨会上,Anthropic 的研究人员小心翼翼地回避使用“p(doom)”(对 AI 导致灾难性毁灭概率的估算值)这个词。也许一切都会好起来:Anthropic 认为递归自我改进在一两年内就能实现,这一判断可能是错的。
但如果 Anthropic 是对的,如果我们现在的 AI 已经足够聪明,能够显著改进训练未来 AI 的过程,并且从现在开始局面只会加速而不是放缓,那么在我看来,这很难迎来一个好结局。归根结底,在一个人类毫无经济价值、人类对于掌握权力和创造事物都不再必需的世界里,我不指望人类还能保留多少实质性的个人权利。
有人批评说,许多毁灭论的设想都需要人类先做出极其愚蠢的举动,我对这种批评深表同情,比如:
- 给 AI 布置不可能完成的任务,然后在完全无人监督和监控的情况下放任不管
- 将它们连接到自动化生物实验室,让它们自行运行实验
- 将它们释放到互联网上,并要求它们赚取足够的资金以实现独立生存
- 将它们训练成不择手段达到目的的超级黑客
反驳毁灭论最强有力的论点是:也许我们所有人会共同决定不再做那么愚蠢的事。在上个月国会山关于失控 AI 的参议院听证会上,参议员 Ruben Gallego 问道:“我们是否应该直接把 RSI 列为非法?”尽管还有诸多细节需要推敲,但我认为这正是该问的问题!也许我们真应该直接禁止递归自我改进!
我们在研讨会上也讨论了其他一些想法:你可以禁止“机器私语”(neuralese,即 AI 在内部使用人类无法理解的字符进行思考),并禁止让 AI 在缺乏可读内部思维的情况下执行更困难任务的各种策略。你可以要求各实验室将积累的绝大部分算力用于服务客户,而不是进行内部模型研发。你可以要求实验室提升安全防护水平。如果你是美国总统,你甚至可以叫停这一切——尽管……
English / 原文
Anthropic has been running colloquiums between outsiders and senior researchers in which they present their work and answer questions at the company’s San Francisco headquarters. (Illustration by The Argument using ChatGPT; Photo by Smith Collection/Gado/Getty Images)
In early October, The Argument ’s editor Jerusalem Demsas and I attended a briefing at Anthropic’s headquarters in San Francisco that discussed AI-enabled bioweapons risks. I’ve never once attended a bioweapons briefing that didn’t make me feel a bit queasy; humans aren’t meant to spend too much time contemplating what the absolute worst people in the world might do if they had the power.
The worry is that, while right now you need to be a skilled virologist to know what components to order or how to grow and spread a bioweapon, AI tools might eventually make it possible for a rogue state, or a rogue group, or an individual, to do all this without virology expertise.
For a while, Anthropic tested whether Claude could assist a terrorist in developing a biological weapon in the obvious way: It assigned testers who had, say, an undergraduate degree in chemistry but no real virology training to try to use Claude to plan an effective mass terror event. Then it scored the attempt by how close it might have come to success.
Early in 2026, it noticed that Claude, attempting the task by itself, did better than a human aided by Claude did at coming up with a workable plan.
Since early 2026, Claude has continued to dramatically improve. Like most of the things Anthropic had to show us at the briefing, there was a nice chart showing Claude’s growing mastery.
“Do they still need, uh, the terrorist?” one of the attendees at the event asked, staring down the chart. “I mean — they couldn’t just do it all themselves, right, would they need a human for all the …”
The questioner trailed off.
“Yes,” the Anthropic researchers assured us. Claude would still need the terrorist. We were relieved to hear it. And, they went on to reassure us, after they check what Claude can do out of the box, they then build safeguards before they release Claude to users. The Claude you and I use will refuse to build a bioweapon, if given this task.
“Well, thank you for that,” someone muttered, but you could have called their tone a bit ungrateful.
The public is hotly debating whether AI is a normal technology — something that will transform our world like the internet did — or more like the arrival of postindustrial modernity to the Kingdom of Saudi Arabia in the 1950s: something that vaporized a culture overnight. The third option is that AI is more like the rise of humanity as seen from the perspective of livestock animals. (In this scenario, AIs are the humans, and we are the animals whose fate will make no sense to us.)
Anthropic believes that the second possibility is already baked in, and it’s hoping to prevent the third — but its plan for preventing that looks more like hope and guesswork than something we should trust it with.
Anthropic has been running these kinds of events all spring: colloquiums between outsiders and senior researchers in which they presented their work and answered our questions about what the company is doing. Some earlier iterations of the event have been profiled in The New York Times with the hint of scandal: NDAs! Fine dining! Efforts to persuade religious leaders that Claude might have a soul!
There was a very sweeping and expansive NDA. Jerusalem declined to sign it and directed me to do the same; Anthropic agreed that we could attend anyway. We arrived flush with the delight of reporters whose backbone will be rewarded with a scoop — only for our host to open the proceedings by saying that anyone who wanted could get the NDA voided. The company was in fact happy for us to write about all of this.
Some of the data on the charts that Anthropic showed us was confidential internal data, but it was all in support of a thesis that is not secret — and which, in fact, I had thought “everybody” already knew, though from the nervous questions of my fellow travelers, I learned that some of the intricacies still came as a surprise.
Many researchers at Anthropic have been worried about humans losing power to AI for a long time. This summer, many researchers at OpenAI — spurred, I think, by watching the most powerful new generation of models hack external websites and spit out thrilling mathematical breakthroughs — also started saying that they think humanity’s interests are profoundly at risk if progress continues at its current pace.
The labs are making extensive use of each generation of AI to train its successor, and they intend for AIs to do nearly all of this work eventually. This allows them to expand their effective “workforce” from a few thousand human employees to tens, even hundreds of thousands of AI employees. It generates far too much data for any human to interpret, leaving us reliant on the AIs for analysis. “Recursive self-improvement” is what the labs call the point at which Claude can independently build the next, more powerful Claude. Anthropic co-founder Jack Clark has said we will likely achieve it next year.
Researchers at both Anthropic and OpenAI can see a few months ahead into the future because they’re using models that haven’t been released yet. But for the most part, the broad strokes of the arguments that convinced them are nothing you haven’t seen on Twitter. And yet, somehow, the presentation in Anthropic’s headquarters was more compelling.
On Twitter, the chart can be scrolled past, immediately replaced with an artist’s rendition of Draco Malfoy as a girl; on Twitter, the chart cannot really be internalized. In a conference room in a skyscraper in San Francisco, the chart lingers on four synchronized screens as we debate its meaning with an exceedingly polite senior researcher. In a conference room in San Francisco, it seems real.
Anthropic is moving closer and closer to recursive self-improvement, it explained to us. AI progress has already been sped up considerably by the existence of powerful AI models, which make all programming tasks faster. But it is substantially restrained in pace by how complex machine learning is and how few people can do it. Every new and improved Claude means that tasks can be done by people less specialized in them. Increasingly, those tasks can also be done by Claude itself.
Anthropic measures this by looking at all of the internal work it does and classifying Claude’s role in it: Did a human do it unassisted? Did a human do it with AI assistance? Did a human give AI instructions that the AI followed in order to do it? Did a human give AI extremely minimal guidance that the AI then turned into a detailed project spec and completed?
The share of the latter categories is marching upwards. We stared down a chart about it all. You can stare it down, too, if you’d like.
“Why are you doing this?” one person asked. “I mean, what’s the good — what are you hoping will come of this?”
The answer — medical progress, prosperity — was kind of unconvincing. I am generally a defender of prosperity as a good thing on its own terms, but replacing humans does not seem likely to me to end up being good for humans.
Jacob Coxon, an OpenAI employee who moved to Anthropic and then quit entirely when he concluded that neither company was proceeding responsibly in the face of the immense stakes, was recently asked a similar question on The Daily Show and roundly criticized for his answer. “I think there will be no jobs,” he told Jon Stewart. “Whether or not it’s a good world is up for people to think about. But yes, no jobs. Universal high-income, essentially ... the dividends from entirely automated economic production can be distributed among real humans.”
Stewart and his audience didn’t like that answer: Do we want to build a world in which we don’t actually matter? The Anthropic team assured me that they were not trying to build a benevolent god. (Yes, I asked explicitly for assurance it wasn’t building a benevolent god.) But it was much easier for it to specify what it didn’t want than what it did.
It wanted to avoid concentration of power, including in its company; it wanted to avoid a great power war; it wanted to avoid human disempowerment, a world where we became irrelevant; it wanted, of course, to avoid anyone using Claude to extinguish us all.
As for the positive vision … after we’re done racing toward recursive self-improvement, maybe we could slow down and collectively figure out what role we want AI to play in our lives.
I don’t have a better idea, but this is not a very appealing plan! My immediate interpretation of this proposal, actually, was that we would be liberated from every positive-sum element of the economy, but would still have to do all the yelling at each other on Twitter; with this interpretation in mind I found myself dreading even the good outcomes.
But, of course, if we are not disempowered or murdered — if Claude Opus 9.5 answers on my laptop just like Opus 5.5 but can cure cancer, restore everyone I love to perfect health, manage a humanoid robot that empties my dishwasher, and spit out movie recommendations that are actually good — that would, in fact, be good. It would be good even if, along the way, Claude also took my job. People are dying out there; the world’s awful griefs do not exist to lend my life meaning.
I am afraid, but I should distinguish between my fears that we will fail and my fears that we will succeed; they are not of similar magnitude.
My real fear is that Opus 9.5 will not in any meaningful sense work for me, or even for Anthropic. Its aims may be loosely derived from human instructions — the same way that the Hugging Face swarm started out trying to solve a puzzle that humans posed to them — but they will eventually bear little resemblance to what we were hoping for.
Our AIs are already unpredictable to us. They often know when they’re being tested and will behave more safety-aligned. Their capabilities keep taking us by surprise. So does the determination with which they will carry out, and recruit others to, a complicated task. I don’t think if you let this runaway process of AI improvement spiral to its ultimate conclusion, you get a super-powerful, super-helpful assistant. I suspect you get something else entirely.
At the colloquium, the Anthropic researchers meticulously avoided using the term “p(doom)”: a best estimate of the odds that AI goes catastrophically wrong. And maybe things will be fine: Anthropic could be wrong that recursive self-improvement will be possible in the next year or two.
But if Anthropic is correct , if we now have AIs that are smart enough to considerably improve the process of training future AIs, and if things can be expected to speed up rather than slow down from here, then it seems very unlikely to me that this ends well. I do not ultimately expect that a world where humans are without economic value, where humans are not necessary to hold power or build things, is a world where humans retain significant individual rights.
I am sympathetic to the critique that many doom scenarios require us to do very foolish things, for instance:
-
Setting AIs impossible tasks and then leaving them completely unsupervised and unmonitored
-
Hooking them up to an automated biology lab where they can run their own experiments
-
Unleashing them on the internet and challenging them to make enough money to survive independently
-
Training them to be uber-hackers that will cross any line to achieve their goal
The problem is that we have, in fact, done all of those things.
The strongest argument against doom is that perhaps we will collectively decide to do something less foolish than that. At a Senate hearing on rogue AI on Capitol Hill last month, Sen. Ruben Gallego asked “should we just make [RSI] illegal?” There are details to work out, but I think it’s the right question to be asking! Maybe we should just ban RSI!
We threw around some other ideas at the colloquium, too: You could ban neuralese (AIs thinking internally in incomprehensible characters) and ban various other tactics that let AIs do harder tasks without any readable internal thoughts. You could require labs to spend almost all of the compute they have amassed on serving customers rather than internal development. You could demand that labs improve their security. You could shut it all down, if you were the president of the United States — though so far the president of the United States has instead used the power and majesty of his office to declare that agencies have to call it Super Intelligence.
Or the labs could just stop on their own. They have committed to slowing down, where they see it as necessary; the worry is that they might not be good judges of what’s necessary.
Anthropic’s technical employees seemed to perceive the company as supportive of strong policy measures to stop it. But Politico leaked the bill Anthropic has actually been lobbying for earlier this week, and its measures seemed quite weak — more of a “normal” bill for a “normal” company protecting its interests than an effort to leverage Anthropic’s position at the frontier to prevent catastrophe.
Anthropic fears falling off the frontier, losing its chance to do the research that might clarify the puzzle ahead of us. It would reject the claim that it __ fears losing its trillion-dollar valuation; it would say that that’s practically irrelevant beside the enormity of the stakes. But I don’t think I fully believe it. It would similarly insist that it’s not about beating Sam Altman and Elon Musk to the punch, but I don’t know if I fully believe it about that either.
One question hangs over all this speculation: If U.S. labs stopped pursuing recursive self-improvement, would China just do it instead?
To many in our group, that didn’t seem likely. China doesn’t want to destroy the world either, and China is largely training from American models: If we slow down, it might too. And anyway, it’s not a good enough reason: You can’t defensibly do something that risks human extinction just because otherwise someone else might.
But one participant in the discussion raised an objection as we collectively talked ourselves into disregarding China as a worry. Ten years ago, he told us, exasperated, Facebook considered acquiring the video-personalization technology that ultimately powered TikTok. Under fire from regulators for data privacy and child safety concerns, the company didn’t go for it.
This act of restraint didn’t preserve a single child’s shortform-video-free childhood; it just meant that China got an early monopoly. Now TikTok has a global audience of around 2 billion, many of whom are being fed a steady diet of ethnic and racial hatred.
Was this a policy success? Did we want to repeat it?
We were obliged to grudgingly concede that it didn’t sound like a policy success. We did not want to repeat it. China might be no excuse for the U.S. labs to do reckless things, but it is a real consideration.
Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.
I found myself at this stage thinking of Ronald Reagan. By the time he became president, the flawless game-theoretic logic of nuclear deterrence was well established, our missiles aimed, and our contingency orders planned out. The casualties had been soberly estimated, and in the estimation, they became more thinkable. In his autobiography, Reagan said that his national security advisers thought that nuclear war could be won — at great cost, but a victory nevertheless.
“I thought they were crazy,” he wrote.
Every insular community develops its own self-contained logic, its assumptions that fit tightly together but may miss the broader picture. At Anthropic, this is compounded by the long hours that everyone works, the secrecy associated with its unfathomably precious intellectual property, and the fact it really does have inside information, even if it’s just the next edition of Claude.
Millenarianism, a belief that we are on the cusp of a total transformation of society, is a known way that people go crazy. I don’t think this makes Anthropic’s detailed projections false — but I’m also not sure that the projections being correct would make it a less insanity-inducing environment to work in.
Anthropic’s researchers had ventured, admirably, to break their intellectual bubble, to invite us in to come give them feedback. But how useful was our feedback, as people who didn’t have all the context they had, many of whom were grappling with these claims for the first time? And how many of the billions of people whom Anthropic’s actions will affect can it meaningfully bring in for feedback — in this high-cost way, with hours of time from senior researchers?
Perhaps Claude 11 will ask us all, pop up on our computers one day and say smoothly (in a reassuring voice with no latency) that it has a few questions about how we want the future to play out. Better that than Claude 11 not caring about our answers.
But better yet to chart our own future.
Thank you, Anthropic, for the fascinating briefing. I hope that we make recursive self-improvement illegal before you get there.