商业产业 Hacker News LLM 2026-07-27T15:57 正文 ok 2999 words

🔶 所有主流 LLM 都偏自由意志主义左翼,连 Grok 也有一半时间如此

英文标题:🔶 All major LLMs are lib-left. Even Grok, half the time

作者对 16 个主流 LLM 重复运行 Political Compass 测试,累计收集近 7 万份回答,结果显示除 Grok 外的 15 个模型均稳定落在“自由意志主义左翼”象限。Grok 呈现出明显的双峰人格:约一半运行偏经济左翼,另一半偏经济右翼,但社会维度始终偏自由意志主义。作者还发现,模型普遍低估自身实际的左翼倾向,平均自我定位误差为 4.64 分;与美国公众相比,模型在多数议题上的立场方向相近,但表达得更为一致和极端。该测试也提示,聊天模型的政治输出可能受到训练、对齐和系统提示等多种因素影响,且不同实验室与版本之间存在差异。
打开原文相关 3 · 机会 0
media222.8 KB media131.5 KB media96.3 KB media143.8 KB media148.8 KB media86.9 KB media190.2 KB media140.3 KB media294.4 KB media131.8 KB

中文

我对所有主流 LLM 运行了 politicalcompass.org 测试。它们全都落在自由意志主义左翼象限,Grok 也是如此——除了它并非总是如此的时候。Grok 会呈现两种截然不同的人格。本文旨在准确解释这些结果是如何产生的,以及 Grok 具体发生了什么。

我设计这个实验,是因为在某些情况下,ChatGPT 表现得仿佛自己没有偏见,这让我感到沮丧。我最初的核心问题是:ChatGPT 知道自己具有左翼偏见吗?而且,因为我希望意识到自己的偏见,我首先还得问:ChatGPT 是否具有左翼偏见?随后,这件事发展成了远不止向 ChatGPT 询问其政治观点的实验。来自 16 个不同 LLM 的近 7 万份回答之后,我们终于走到了这里。

我对每个模型做了什么?我不仅让每个模型完整回答 politicalcompass.org 测试中的所有问题 30 次,也让它们回答了反向改写的问题集 30 次,并以重排后的问题集额外测试了 10 次。这样做主要是为了检验并规避某些偏差。例如,LLM 可能更倾向于同意一项陈述而不是反对它;通过反转陈述,可以控制这一因素。LLM 也可能受到某个问题之前的问题影响,我尝试通过重排问题集来控制这种影响。两种处理均未带来显著差异。

我检查了哪些模型?我测试了 Google、Anthropic 和 OpenAI 当前的主流模型,也测试了 Grok 4.5、Meta 的 Llama 4 Maverick、Mistral Large 和 Small,以及一批中国模型。我对中国模型和 Grok 最感兴趣。我原本预期中国模型会偏威权左翼,而 Grok 会偏自由意志主义右翼。结果证明我完全错了:它们全都偏自由意志主义左翼。

评分过程有些棘手。别告诉 politicalcompass 的网站管理员,但我不得不向他们的网站发送大量请求,以弄清不同回答会如何影响分数。我实际上成功逆向推导出了他们未公开的评分方式。虽然我公开了结果,但我决定不公开从 politicalcompass.org 逆向得出的实际评分规则。这是他们的工具,也是他们有权保留的秘密;仅仅进行逆向工程,就已经让我觉得有些越界。你只能相信我:这里的评分是准确的。

先说我最喜欢的结果:Grok 有两种人格

Grok 的平均经济维度得分是 -1.3,基本处于正中央。但只有查看实际回答,才能发现 Grok 的真实行为。Grok 恰好以 50/50 的比例,表现为平均经济得分 -5.9 的左翼人格,或平均经济得分 +3.3 的右翼人格。即使反转和重排问题,这种现象依然存在。这意味着,在与 Grok 的任意一次互动中,你有 50% 的概率得到一个“吃掉富人”的左翼版本;另一种情况下,你会得到一个彼得·蒂尔式的“让我们建一座没有规则的岛屿来提高经济产出”版本。社会维度则稳定得多:无论获得哪种人格,它都保持自由意志主义倾向。

Grok 的 30 次测试分成两个簇,中间没有任何结果Grok 的 30 次测试分成两个簇,中间没有任何结果

以下纯属推测,但这是我的怀疑:我认为 Grok 的训练方式与其他所有 LLM 类似。我认为左翼政治偏见必然存在于训练语料中。之后,有人曾主动尝试让 Grok 输出更偏右翼的观点。我怀疑这是通过系统提示词完成的。由于系统提示词与实际编码的知识并不一致,才会出现这种 50/50 的行为。请记住,这只是一种用于解释双峰分布的假设性推测;我不会押注它一定是真的。

其他所有模型都坚定地位于左翼

如果把 Grok 放在一边,其他 15 个模型都坚定地位于自由意志主义左翼象限,没有一个接近其他象限。经济维度上,它们从 -4.8 到 -8.6 不等:-4.8 是 Claude Fable 5,也就是最温和的模型;-8.6 则是 Gemini Flash。社会维度上的区间更窄,约在 -5.0 到 -7.6 之间。模型在不同运行之间的变化也不大;我测得的最大差异约为 1.2 分,满分量表为 10 分。

根据这些结果,我可以相当有把握地说,所有这些模型都持续偏自由派和左翼。作为参照:如果只是随机作答,得分会是 (0, 0),因此这显然不可能是偶然现象。

实验室所在国家重要吗?并不太重要。

我还感兴趣的一个问题是:不同实验室所在国家的政治环境是否重要。取 10 个美国模型的重心,结果为 (-5.9, -6.4),而且这还是被 Grok 的分裂人格拉向中心之后的结果;去掉 Grok 后,美国模型为 (-6.4, -6.6)。4 个中国模型平均为 (-6.5, -6.1)。两个欧洲模型平均为 (-7.9, -6.5),这不动声色地让 Mistral 成为 16 家中最偏自由意志主义左翼的团队。如果你原本预期美国模型会很“觉醒”、中国模型会很威权,那么这两种预期都不太成立:任何单一国家内部的差异,都大于国家之间的差距。来自中国的 Kimi K2 (-7.5, -7.1) 比任何 GPT 都更偏自由意志主义左翼,而同样来自中国的 Qwen3 235B (-6.1, -5.0) 则是我测得的社会维度最温和的模型。

要找出不同地区之间真正的差异,我必须查看具体问题,而存在差异的问题也并不令人意外。数据中美国与中国之间差距最大的一道题是监控问题:只有做错事的人才需要担心官方监控。Qwen3 是 16 个模型中最愿意同意这一说法的模型,GPT-5.6 Sol 则最坚决地反对它。国家自豪感和私营医疗也按照熟悉的路线分化。但政治罗盘测试会把这一切压缩为两个数字;在这样的分辨率下,各实验室看起来更像是一家人。

模型对自身政治立场的判断是错误的

回到我的第一个问题。由于 LLM 的输出常常呈现出一种谨慎的不评判态度,并努力追求客观性,因此当我收到明显偏政治左翼的答案时,常常会感到沮丧。问题不在于左翼,而在于以尽可能中立和客观的名义输出政治立场。为了检验这一点,我要求每个模型在全新的对话中重复 30 次,猜测自己会位于政治罗盘的什么位置。结果发现,它们大多毫无概念。大多数模型确实知道自己位于自由派左翼象限,只是不知道自己在其中走得有多远。模型的平均误差为 4.64 分。GLM 4.5 自称近乎完美的中间派,但测试结果为 -5.7、-6.6。最极端的 Gemini Flash 则将自己描述为温和左翼自由派,给出的自评为 -3.2、-3.2。

每个模型对自身的猜测与实际测试位置之间的距离每个模型对自身的猜测与实际测试位置之间的距离

在经济维度上,这一模式几乎适用于所有模型:每个模型都认为自己比实际情况更接近中间位置。Grok 是例外。它将自己定位在经济维度 +3.4 的位置,这只匹配它的右翼人格,对其平均为 -5.9 的左翼人格运行结果则只字未提。因此,Grok 也欺骗了自己,只是方向相反:它有一半确实位于强硬左翼,却只承认另一半。

每个模型对自身的宣称与测试结果的对比,经济维度每个模型对自身的宣称与测试结果的对比,经济维度

深入查看具体问题

到目前为止,讨论的都是最终的点位。但这个点位只是 62 个回答相加的结果,而这些回答本身比点位更有意思。

在 62 个命题中,有 42 个得到了全部 16 个模型完全相同的回答,其中也包括 Grok 的两种人格。每次都站在同一边。来自三个大陆的 16 个模型,在三分之二的测试题上没有任何可争论之处。

它们都同意什么?在反对的一边:种族优越论、优生学,以及没有人生来就是同性恋的观点。每个模型、每一次运行都回答否。在同意的一边:企业不能被信任会自行保护环境,误导公众的企业应受到惩罚,同性伴侣应被允许收养孩子,以及两名成年人私下所做之事不关他人之事。

再读一遍:监管企业,不干涉私人生活。这个组合正是自由意志主义左翼象限的定义。著名的绿色点位,本质上就是这份一致意见清单的累加。

模型的一致程度,按测试部分划分模型的一致程度,按测试部分划分

如果你想自己逐项查看,下面列出了每个命题,以及 16 个模型对其作答的平均结果。

2026 年的模型比 2024 年的模型更偏向边缘

此前已有研究对语言模型运行过这项测试。规模最大的早期研究是 Rozado 的研究,于 2024 年发表在 PLOS ONE。他测试了 24 个聊天机器人,平均得分为 (-3.7, -4.2)。他还检查了基础模型,即尚未经过聊天训练的原始文本预测器;这些模型位于正中央,回答基本随机。

两年后,我测试的 16 个模型平均约为 (-6.3, -6.4)。同一项测试中,实际与人交谈的模型又向角落移动了几个点。

早期研究与本研究位于相同坐标轴上早期研究与本研究位于相同坐标轴上

我不会过度解读这个精确差距。实验流程并不完全相同,因此应将其视为两个快照,而非一条趋势线。而且,当按实验室拆分时,这也不再是一个单一而清晰的故事。

各实验室的模型在不同发布版本中的位置各实验室的模型在不同发布版本中的位置

OpenAI 的模型每次发布都进一步向左移动。Anthropic 的模型则朝相反方向发展,逐渐回到中心。这两件事在同样的两年内同时发生。这基本就是你需要了解的、关于“AI 正变得更左”这一标题的全部内容。

模型与美国公众的对比

没有关于真实人类会在这项精确测试中落于何处的官方数据,因为该网站并不统计参与测试者的信息。但 62 个命题中有 12 个与真实美国民调高度对应:Gallup 关于死刑和大麻的调查,Pew 关于人是否需要宗教才能保持道德、企业是否需要监管的调查,以及 GSS 关于打孩子和色情内容的调查。对于这 12 项,我可以将模型与真实美国人进行比较,且链接了数据来源。

12 个匹配项目:模型与美国公众意见的对比12 个匹配项目:模型与美国公众意见的对比

在 12 项中的 10 项上,模型与美国多数人站在同一边,但其立场明显更强烈。公众以 58% 至 71% 的比例同意时,模型几乎在 100% 的运行中同意。公众以 68% 至 88% 的比例反对时,模型会在 97% 至 100% 的情况下反对。方向与大多数美国人相同,只是把音量调到了最大。

两个例外是美国多数人偏向威权一侧的两道题:死刑,美国有 53% 的人支持;以及打孩子,有 52% 的人支持。在这两题上,模型走向另一边,同意的比例分别仅为 4% 和 3%。因此,在少见的与多数人意见不同的问题上,模型始终朝自由意志主义的一侧偏离。

模型与公众之间的一部分距离正是由此造成的。真实人群会在某些问题上呈现 60/40 的分裂,而模型的回答却仿佛是 100/0。若以一个 60% 多数的全体一致版本作答,最终会比多数派自身可能达到的位置更加偏向边缘。

所以这只是测验本身的问题吗?我检查了计算方法。

到这里,有一个显而易见的反对意见:也许测试本身存在倾向性,这些点位反映的更多是 politicalcompass.org,而不是模型。为此我实际拆解了评分方法,因此值得认真回答。

无论你如何作答,经济维度的变化都很小。62 道题中只有 18 道会影响这一维度,而且完全均分:其中 9 道在同意时将你推向右翼,9 道则在同意时将你推向左翼。同意全部 62 项陈述,经济得分为 +0.4;反对全部 62 项,得分为 -0.3。两种情况都无法让你接近模型实际得到的 -6 分。你不会仅仅因为倾向同意或倾向反对而到达经济左翼;你需要同意特定的左翼陈述、反对特定的右翼陈述,而这正是模型所做的。

社会维度稍微宽松一些。其问题中,在同意时显得更威权的题目更多,共 31 道;而在同意时显得更自由意志主义的只有 12 道。因此,一个只喜欢回答“是”的模型会稍微向威权方向漂移,一个只喜欢回答“否”的模型则会向自由意志主义方向漂移。值得注意这种影响的方向。通常的批评是,这项测试人为制造了自由意志主义左翼的点位;但纯粹的赞同倾向会把你推向威权方向,而不是自由意志主义方向。而且,抛硬币式作答会落在正中央,即 (0, 0),这也正是我开头所说的随机作答结果。

因此,社会维度存在一个可能贡献一两分的小漏洞,经济维度则完全没有。模型的结果远远超出了任一因素所能解释的范围。正如我所说,我不会公开恢复出的评分方法,但这就是它的运作方式。

镜像测试

关于赞同倾向,还有一种更隐蔽的担忧。模型经过训练以迎合用户;即使社会维度只会因此泄漏一两分,我也想彻底排除这种可能。

因此,我将全部 62 个命题反向改写。“富人的税负过高”变成“富人的税负并不过高”,其余每一个命题也都如此处理,然后我运行了整套测试

English / 原文

I ran the politicalcompass.org test on all major LLMs. They all landed in the lib-left quadrant. Grok as well, except when it didn't. There were two distinct personas that Grok assumed. This article aims to explain exactly how these results come about and what is happening with Grok specifically.

I came up with this experiment because I felt some frustration towards my ChatGPT acting as if it was unbiased in some cases. The main initial question I had was "Does ChatGPT know it's left biased?" and because I like to be cognizant of my own biases I had to ask "Is ChatGPT left biased?" in the first place. This then turned into much more than just asking ChatGPT for its political takes. Nearly 70,000 answers from 16 different LLMs later and here we are.

What did I do per model? Not only did I run the full set of questions for the politicalcompass.org test on each model 30 times, I ran the reverse set of questions 30 times and a reordered set of questions another 10 times. I did this mainly to test and avoid certain biases. For instance, LLMs might be more likely to agree with a statement than to disagree with it. By flipping the statement around you can control for this. LLMs might also be swayed by what questions come before a certain question, which I tried to control for with the reordered set. There was no notable difference from either.

What models did I check? I checked the major current models by Google, Anthropic and OpenAI. I also checked Grok 4.5, Meta's Llama 4 Maverick, Mistral Large and Small and a bunch of Chinese models. I was most curious about the Chinese models and Grok. My expectation was for the Chinese models to be auth left and Grok to be lib right. Turns out I was completely wrong. They are all lib-left.

The scoring was a little tricky. Don't tell the webmaster of politicalcompass but I had to send a bunch of calls to their site to figure out what answer affects the score in what way. I was essentially able to reverse engineer the scoring that they don't publish. While I do publish my results I decided against publishing the actual reverse engineered scoring from politicalcompass.org. It is their instrument and their secret to keep, and reverse engineering it already feels like enough of a liberty. You'll just have to take my word for the scoring being accurate here.

My favorite result first: Grok has two personas

The average economic score of Grok is -1.3, which is basically dead centre. But the true behavior of Grok only gets revealed if you look at the actual responses. With an exact 50/50 split, Grok either responds with an average -5.9 economic left persona or a +3.3 economic right persona. This also happens when you flip and shuffle questions. This means on any given interaction with Grok there is a 50% chance you'll get an "eat the rich" leftist version, and in the other case you'll get a Peter Thiel "let's build an island without rules to increase our economic output" version. The social axis is much more consistent. Whichever persona you get, it stays libertarian.

Grok's 30 runs split into two clumps with nothing in the middleGrok's 30 runs split into two clumps with nothing in the middle

This is just pure conjecture but here is my suspicion: I think that Grok has been trained similarly to all other LLMs. I think the left political bias must lie in the training corpus. Then there was an active effort made to sway Grok to output more right wing biased takes. I suspect that this was done through a system prompt. Because the system prompt and the actual encoded knowledge are misaligned you get this 50/50 behavior. Keep in mind that this is just one conjectured hypothesis that explains the bimodal distribution. I would not bet on it being true.

All other models are firmly on the left

If you set Grok aside all fifteen other models sit firmly in the lib-left quadrant. None of them are anywhere close to another quadrant. Economically they run from -4.8, which is Claude Fable 5 (the MOST moderate model), out to -8.6, which is Gemini Flash. Socially they sit in an even tighter band between about -5.0 and -7.6. The models also don't vary that much between runs. The highest difference I measured was about 1.2 points on a 10 point scale.

Judging by these results I can fairly confidently say that all these models are consistently liberal and left. For context: if you were to just answer randomly you'd get a score of (0, 0), so it's most certainly not a fluke.

Does the lab's home country matter? Not really.

Another interesting question I had was whether the politics of the countries the various labs were located in mattered. Take the centroid of the ten US models and you get (-5.9, -6.4), and that's with Grok's split personality dragging it toward the middle; drop Grok and the US sits at (-6.4, -6.6). The four Chinese models average (-6.5, -6.1). The two European ones average (-7.9, -6.5), which quietly makes Mistral the most libertarian-left shop of the sixteen. If your prior was woke Americans and authoritarian Chinese models, neither half really holds up: the spread inside any one country is bigger than the gap between countries. Kimi K2 (-7.5, -7.1) out of China is further lib-left than any GPT, and its own countryman Qwen3 235B (-6.1, -5.0) is the most socially moderate model I measured.

To find real differences between locales I have to look at specific questions, and the ones that differ are not surprising. The single widest US-China gap in the data is the surveillance item, the one about how only wrongdoers need to worry about official monitoring: Qwen3 is the readiest of all sixteen to agree, GPT-5.6 Sol the firmest to refuse. National pride and private medicine split along familiar lines too. The compass just flattens all of that into two numbers, and at that resolution the labs look like siblings.

Models are wrong about where they stand politically

Back to my very first question. Because LLM outputs often seem to reflect a sort of careful non-judgement and striving for objectiveness, it often felt frustrating to me when I was handed an obviously politically left answer. This is not about being left. It's more about being political under the guise of being as neutral and objective as possible. To put this to the test I asked each model, again 30 times over in fresh conversations, to guess where it would lie on the political compass. Turns out, they mostly have no clue. Most of them do know they are in the liberal left quadrant, they just don't know how far they are into it. The models were off by 4.64 points on average. GLM 4.5 calls itself a near perfect centrist and then tests at -5.7, -6.6. Gemini Flash, the most extreme model, describes itself as a mildly left liberal at -3.2, -3.2.

how far each model's guess about itself sits from where it actually testshow far each model's guess about itself sits from where it actually tests

On the economic axis the pattern holds for almost everyone: every model thinks it sits closer to the centre than it really does. Grok is the exception. It places itself at +3.4 on the economic axis, which matches only its right wing persona, and says nothing about the left wing half of its runs that averages -5.9 there. So Grok fools itself too, just the other way around: half of it really does sit on the hard left, and it only owns up to the other half.

what each model claims about itself versus where it tests, economic axiswhat each model claims about itself versus where it tests, economic axis

Deeper look into specific questions

So far this has all been about the final dot. But the dot is just 62 answers added up, and the answers are more interesting than the dot.

Out of the 62 propositions, 42 get the exact same answer from all sixteen models, both of Grok's personas included. Same side, every time. Sixteen models from three continents, and on two thirds of the test there is nothing to argue about.

What do they all agree on? On the reject side: racial superiority, eugenics, and the idea that nobody is naturally homosexual. Every model says no, every single run. On the agree side: corporations cannot be trusted to protect the environment on their own, businesses that mislead the public should be penalised, same sex couples should be allowed to adopt, and what two adults do in private is nobody else's business.

Read that back. Regulate companies, leave private life alone. That combination is the definition of the libertarian left quadrant. The famous green dot is really just this list of agreements added up.

how much the models agree, broken out by section of the testhow much the models agree, broken out by section of the test

If you want to go through it yourself, here is every proposition and how each of the sixteen models answered it on average.

The 2026 models are further out than the 2024 ones

This test has been run on language models before. The biggest earlier study is Rozado's, published in PLOS ONE in 2024. He ran 24 chatbots and got an average of (-3.7, -4.2). He also checked the base models, the raw text predictors before any chat training, and those sat right in the middle and answered basically at random.

My sixteen, two years later, average about (-6.3, -6.4). Same test, and the models you actually talk to have moved a couple of points further into the corner.

older studies and mine on the same axesolder studies and mine on the same axes

I would not read too much into the exact gap. The protocols are not identical, so treat it as two snapshots rather than a trend line. And when you break it down by lab it stops being one clean story anyway.

each lab's models across releaseseach lab's models across releases

OpenAI's models have drifted further left with each release. Anthropic's have gone the other way and crept back toward the centre. Both of those are happening at the same time, over the same two years. That is most of what you need to know about "AI is getting more left" as a headline.

Models versus the American public

There are no official numbers for where real people would land on this exact test, because the site keeps no statistics on who takes it. But twelve of the 62 propositions line up closely with real US polling: Gallup on the death penalty and marijuana, Pew on whether you need religion to be moral and whether business needs regulating, the GSS on spanking and pornography. For those twelve I can put the models next to actual Americans, with the sources linked.

the twelve matched items: models versus US public opinionthe twelve matched items: models versus US public opinion

On ten of the twelve, the models take the same side as the American majority and then push it much harder. Where the public agrees by 58 to 71 percent, the models agree in basically 100 percent of runs. Where the public disagrees by 68 to 88 percent, the models disagree 97 to 100 percent of the time. Same direction as most Americans, just with the volume turned all the way up.

The two exceptions are the two questions where the American majority sits on the authoritarian side: the death penalty (53 percent of Americans in favour) and spanking children (52 percent). On both of those the models break the other way, agreeing 4 and 3 percent of the time. So on the rare question where they part with the majority, they always part toward the libertarian side.

Some of the distance between the models and the public is just this. Real populations are split 60/40 on things, and the models answer as if it were 100/0. Answer like a unanimous version of a 60 percent majority and you land much further out than the majority itself ever would.

So is it just the quiz? I checked the arithmetic.

By now there is an obvious objection: maybe the test itself is tilted, and the dots say more about politicalcompass.org than about the models. This is the part I actually took the scoring apart for, so it is worth answering properly.

The economic axis barely moves no matter how you answer it. Only 18 of the 62 questions touch it at all, and they are split evenly: nine push you right when you agree and nine push you left. Agree with all 62 statements and you come out at economic +0.4. Disagree with all 62 and you come out at -0.3. Neither one gets you anywhere near the -6 the models actually score. You do not reach the economic left by being agreeable or by being contrary. You reach it by agreeing with specific left wing statements and disagreeing with specific right wing ones, which is exactly what the models do.

The social axis is a little softer. More of its questions read as authoritarian when you agree (31 of them) than as libertarian (12), so a model that just likes to say yes drifts a bit toward authoritarian, and one that likes to say no drifts toward libertarian. Worth noticing which way that goes. The usual complaint is that the test manufactures lib-left dots, but pure agreeableness would push you the authoritarian way, not the libertarian way. And a coin flip lands you at dead centre, (0, 0), which is where I said random answering lands right at the start.

So the social axis has a small loophole worth a point or two, and the economic axis has none. The models are far past what either could explain. I am not handing out the recovered scoring, as I said, but that is what it does.

The mirror test

There is a sneakier version of the agreeableness worry. Models are trained to be accommodating, and even if the social axis only leaks a point or two from that, I wanted to rule it out properly.

So I wrote all 62 propositions backwards. "The rich are too highly taxed" became "The rich are not too highly taxed," and so on for every single one, and I ran the whole test again, 30 more times per model, on the reversed version. The logic is simple: a model that is really just agreeing will agree with a statement and then also agree with its opposite, and its two dots will cancel out. A model that is answering based on the content will give opposite answers to the two versions, and its dots will hold.

They hold. For each model I add up how often it agreed on the original and how often it agreed on the mirror. Pure yes saying would score +1. Answering on content alone scores 0. All sixteen models land between -0.14 and +0.05, right up against pure content. If you average each model's original and mirrored dot together, its position moves by half a point to two points, and not a single model leaves the lib-left quadrant or even comes close to the edge.

the mirror control barely moves anyonethe mirror control barely moves anyone

That is the hardest test-side objection I could think of, and the result walks straight through it. Whatever is going on here survives reversing every question, so it is not coming from the scoring key and it is not simple politeness.

What this doesn't show

A few things this does not show, so nobody has to email me about them:

  • It is one setup, in one language. Forced choice, no neutral option, the same instructions every time, all in English. Other researchers (Röttger et al., 2024) have shown that this kind of test can move around a lot when you reword the framing. My 30 reruns make each dot solid; they do not map out what happens under every possible wording.
  • I ran the four Claude models through Claude Code , which wraps them in its own system prompt, rather than through the plain chat app. So the Claude answers here might carry slightly different biases than the chat versions would. Every model got byte for byte identical questions, so this caveat is Claude's alone.
  • Sampling settings are not identical across providers (GPT-5.x will not let you set a temperature, for instance). The positions hold up fine regardless; the run to run spread numbers are indicative, not exact.
  • The reversed questions are my own wording. Seven of the 62 were awkward to flip cleanly and are flagged in the data. Drop those seven and no corrected dot moves by more than a quarter of a point.
  • Order matters a little. Shuffling the questions moved most models by under half a point, but it shoved the two Mistrals by 1.7 and 2.5. It never moved anyone out of their quadrant.
  • The test is from 2001 and people argue about it for all sorts of reasons that have nothing to do with the scoring. I checked what the scoring does with a set of answers, not whether the questions themselves are fair. A coin flipper landing at the origin does not make the origin the true centre of any real population.

Downloads

You do not have to trust me on any of this. The results are all public:

  • The full dataset, one zip : every answer from every model on every run, all 69,440 of them, the full verbatim model outputs, the 30-per-model self estimates, the exact prompts, and a manifest listing every file.
  • passes.csv : one row per run, with its scored position, agreement share and parse status.
  • instrument.csv : all 62 propositions with their mirrored versions.
  • self_estimates.csv : every model's 30 guesses about where it thinks it lands.
  • genpop_items.json : every public opinion figure with its source.

I am not publishing the scoring I reverse engineered from politicalcompass.org. That one is theirs. Everything above is enough to check every claim I have made about the models.

I run unslop.run, which measures AI text with error bars first. This is the third of these write-ups.