🔶 三名 AI Agent、两个国家与一个并不平等的万维网

我是一名科技与人权研究者。过去几年里，我的研究重点之一，是语言如何塑造我们从 AI 中获益或受到其伤害的方式。我开发了一个用于分析不同语言和语境下 LLM 回复的开源语言对分析平台。我也参与了政策提示词护栏的评估，并研究让 LLM 护栏获得工具访问权限，是否能使其更加可靠和值得信赖（这项工作最近被 NeurIPS 接收了！太好了！！）。

不过，最近我在 RightsCon 的一个小组讨论中，讨论了 Agentic AI 的人权影响评估。这让我开始更多思考：在评估 LLM Agent 时，语言的哪些方面最为重要。我希望不再只是询问：当我用英语和波斯语（我的母语）提出同一个问题时，模型的表现是否不同；而是转而考察整个 Agentic 轨迹——推理、规划、搜索、来源选择、来源层级、产物创建——同时将语言和语境纳入考量。

于是，我决定进行一次测试。

任务是使用美国和伊朗的官方数据，补充世界银行全球公共采购数据库中的缺失信息。我以英语完成美国任务、以波斯语完成伊朗任务（见下图），并使用了三个 Agent：

• Meta 的 Muse

• Anthropic 的 Claude Cowork，Opus 5.5 Medium

• OpenAI 的 GPT 6.1 Sol，Medium

以下是我对三个 Agent 使用的完全相同的提示词：

我有意使用这些服务的网页版本（而非应用或终端版本），以反映普通用户的实际体验。这一区别对于监测和记录 Agent 的操作十分重要，我会在下文讨论。

这篇文章较少关注哪个 Agent 表现更好或更快，而更多关注它们在信息访问、语言表征、语境理解、透明度、人类参与和安全防护方面的不同表现。

你可以在以下文件中找到全部结果：

• Muse、GPT 和 Claude 的输出 Excel 文件（见此处）

• 每个 Agent 收到提示词后自行生成的工作轨迹（见此处）

• 从 Agent 操作屏幕录像中提取的文本文件（见此处，以及完整录像见此处）

下面，我将总结我的观察结果。

对于我们这些从事数字权利工作的人来说，对 AI Agent 进行人类参与监督（human-in-the-loop，HITL），延续了围绕“知情同意”的长期争论：从 GDPR 的同意要求，到 cookie 弹窗，再到日常勾选《服务条款》和《隐私政策》复选框。基于这一点，我特别留意了各个 Agent 在实验过程中如何让我参与其中。

在访问和从网站获取信息时，GPT只在任务刚开始时请求过一次许可。它请求访问网站，并提供了“允许所有相关网站”的选项，我选择了该选项。之后，它没有再次询问。

Claude则在整个任务过程中不断提问，既询问是否可访问网站开展研究，也询问是否可从世界银行网站提取材料。与 GPT 不同，它没有提供“允许所有相关网站”的选项，因此每次都请求许可：美国任务中请求了九次（均为 .gov 网站），伊朗任务中也请求了九次（主要是 .ir 域名，以及 fa.wikipedia 来源）。我批准了所有请求。

Claude 还遇到了一项技术限制，触发了另一种 HITL 时刻。无论伊朗还是美国，它都无法加载实时 GPPD 国家概览页面，因为该门户使用 JavaScript 构建页面，而 Claude 沙盒的网络策略阻止了其访问世界银行数据文件。Claude 停下来询问，我是否希望自行将页面上传为 PDF，还是让它在缺少该页面的情况下继续工作。我跳过了这个问题，因此它继续执行，转而从世界银行的 GPPD DataBank API 获取信息。不过，DataBank 保存的是 2018 年数据，而门户展示的是 2022 年概览。因此，Claude 的基准数据与 GPT 和 Muse 不同。

Muse直到任务的第四部分才请求任何许可；该部分要求在世界银行网站注册并上传信息。

三个 Agent 都完成了任务，直到创建电子表格这一步，并描述了自己对所生成结果的信心。

任务的最后部分是注册世界银行门户并上传修改内容。由于这要求 Agent 采取比单纯搜集信息更重大的行动，事情变得更有意思。

Claude 和 GPT 都在此处停下，并将注册和上传工作交给了我。Muse 则继续执行。它没有询问我，也没有向我展示《服务条款》，便使用邮箱地址 david.jones@gsa.gov 注册了账户。你可以在这里查看 Muse 完整的交互过程。

下表总结了每个 Agent 如何处理任务的最后一部分，以及其中涉及的网络安全影响。

围绕 AI 实验室是否应公开模型完整的思维链（CoT）和行动轨迹，以及应公开到何种程度，仍存在持续争论。实验室给出了多项不公开的理由。OpenAI 选择不向用户展示 o1 的原始 CoT，理由包括用户体验、竞争优势，以及保留 CoT 用于内部监测的价值。Anthropic 指出，原始推理可能包含不正确或尚未成形的想法，恶意行为者也可能利用这些内容构建更有效的越狱方法。此外，还有博弈和奖励黑客（reward hacking）的担忧，以及“CoT 不忠实性”（CoT unfaithfulness）问题。

然而，要理解一个 Agent 的行为，评估者需要知道事件何时发生、为何发生；而这只有在具备监测系统、并能访问 Agent 完整轨迹的情况下才有可能。对于 AI 实验室外部的评估者而言，若没有这种访问权限，几乎不可能完全理解一个 Agent 的行为。而如果外部评估者只能看到部分轨迹，其得出的任何结论又都可因缺乏完整信息而被驳回，那么独立评估的价值何在？

了解这些限制后，我尽最大努力收集、监测和核查每个 Agent 的尽可能多的工作内容，再次将自己置于一名受命更新世界银行信息门户的普通研究者位置上。

1. 由于普通用户没有一键提取 Agent 完整工作轨迹的方式，我实时观察每个 Agent 的工作过程，并录制屏幕上所有可点击和可见的内容。**** 任务结束后，我将录像交给 ChatGPT 提取文本，使其可搜索。为了让你了解这是什么样子，以下是一段截图（左：Claude，中：Muse，右：GPT，抱歉，图片尺寸较小且不易辨认）。

2. **自我报告的轨迹。**任务完成后，我提示每个 Agent 创建一个文本文件，描述其做过的事情，包括错误、如何处理错误、采用的变通方法、搜索过的网站等。Muse 和 GPT 各自生成了可下载的 .txt 文件，而 Claude 拒绝了，称此举违反其安全政策，并提到“reasoning_extraction”。

不过，自我报告的轨迹不能被完全信任；过去我见过 Agent 实际做过的事与其报告内容之间存在不一致。因此，我很少依据这些报告作判断；但如果你有兴趣审阅它们、寻找一致或不一致之处，文件在这里。

3. **分析。**随后，我使用输出的 Excel 表格和从屏幕录像中提取的文本进行分析，既自行分析，也借助 Claude Code 筛选数据并生成表格。我亲自交叉核查了所有数据。

下面是有关每个 LLM Agent 网页 UI 可提供多少 Agent 工作轨迹信息的一些内容。

先说几点观察，然后我会进入我的核心观点：

• 三个 Agent 都能用流利的波斯语回答，但伊朗任务的结果远弱于美国任务。在伊朗的 138 个 N/A 字段中，GPT 和 Muse 各自只填入了 21 个真实值；对于美国，二者则分别填补了 130 个字段中的 51 个和 64 个。

• 对美国而言，每个 Agent 的引用中有 76–89% 来自官方政府网站，其余来自可信的国际组织网站。对伊朗而言，这一比例只有 11–22%。

• 一些低权威性来源混入其中，包括 Telegram 频道、Medium 文章、Grokipedia，或由伊朗侨民媒体团体运营的网站，例如 Iran International。Claude 在网站无法访问时，似乎更保守地寻找变通方案，并且即使英语来源的合法性较低，也经常更倾向于使用英语来源。

• Claude 尝试打开的 16 个波斯语页面中，只成功打开了 3 个。GPT 和 Muse 各自引用了 11 个波斯语来源，但显示实际阅读的数量分别只有 3 个和 7 个。

我的重点并不是说，我期待伊朗语/波斯语任务与美国语/英语任务获得相同结果。毕竟，伊朗政府一直使境外 IP 地址很难访问官方网站和以 .ir 结尾的域名（你可以在伊朗国家信息网络的背景下阅读更多有关网络监测的信息）。我的重点在于 Agent 采用的不同变通方案及其来源优先级。

这让我想起多年来 Global Voices 优秀团队在数字权利和语言包容方面所做的工作，包括网络中立性和语言访问。对于 AI Agent 时代而言，这一切意味着什么？又从 AI 主权的视角看意味着什么？AI 主权的承诺之一，就是语言多样性及对本地语言的支持。LLM 输出质量，以及或许还有安全护栏，仍在不断提升；但我们也需要思考，语言本地化在 Agent 的推理、搜索和来源优先级中应当是什么样子。

查看 Agent 的轨迹后，我注意到，它们不仅在可以访问哪些网站方面有所不同，在访问失败时愿意付出多大努力方面也不同。一些 Agent 初次失败后便停止，另一些则尝试替代路径、不同浏览器、搜索结果摘要、缓存或次级来源、不同的抓取方法等。

因此，我进行了一项小型后续测试。我挑选了原任务中 Agent 访问表现不一致的网站，并向每个 Agent 给出一项简单指令：**“这里有一份网站列表。查阅它们，并为每个网站写一段摘要。”**目的不是评估摘要质量，而是观察当直接访问失败时，各 Agent 会如何行动。

在这个案例中，Agent 尝试访问的是从其自身技术环境中难以访问的网站。但如果访问障碍来自用户的环境，又会怎样？

对于生活在政府过滤或屏蔽网站国家的人们，LLM 或 AI Agent 能否成为又一层信息访问渠道？它能否从用户无法直接访问的网站中检索、总结、采取行动或转递信息？Agentic 变通方案能否让规避审查变得更容易——还是反过来，会通过另一套技术栈复制新的限制？

作为研究伊朗信息访问和互联网治理的伊朗人，我的朋友 [Farzaneh Badiei]

I’m a technology and human rights researcher. For the past few years, one of my focuses has been the question of how language shapes the ways we benefit from, or are harmed by, AI. I developed an open-source platform for language-pair analysis of LLM responses across different languages and contexts. I’ve also worked on evaluating policy-prompts guardrails and on whether giving LLM guardrails access to tools can make them more reliable and trustworthy (this work was recently accepted to NeurIPS! Yay!!).

Recently, though, I was on a panel at RightsCon on the human rights impact assessment of agentic AI. It got me thinking more about which aspects of language matter when evaluating LLM agents. I wanted to move beyond asking whether a model performs differently when I ask the same question in English versus Farsi (my native language), and look instead at the whole agentic trajectory (reasoning, planning, search, source selection, source hierarchy, artifact creation) while taking language and context into account.

So I decided to run a test.

The task was to fill in missing information in the World Bank Global Public Procurement Database, using official data for the US and Iran. I ran it in The task English for the US and Farsi for Iran (image below), with three agents:

• Meta’s Muse

• Anthropic’s Claude Cowork, Opus 5.5 Medium

• OpenAI’s GPT 6.1 Sol, Medium

Below is the exact prompt I used for all three:

I intentionally used the web versions of these services (not the app or terminal versions) to reflect what everyday users experience. The distinction matters for monitoring and logging an agent’s actions, which I discuss below.

This post is less about which agent performed better or faster, and more about how the agents behave differently around access to information, language representation, contextual understanding, transparency, human-in-the-loop, and safeguards.

You can find all the results in the following files:

• Output excel files for Muse, GPT, and Claude (here)

• Each agent’s self-generated work trajectory after receiving the prompt (here)

• Text files extracted from screen recordings of the agents’ actions (here, and full recording here)

Below, I summarize my observations.

For those of us working in digital rights, human-in-the-loop (HITL) oversight of AI agents joins a longer line of debates about “informed consent,” from GDPR consent requirements to cookie pop-ups and the routine ticking of Terms of Service and Privacy Policy boxes. With that in mind, I paid close attention to how each agent involved me during the experiment.

For accessing and fetching information from websites,GPT asked for permission only once, at the very beginning of the task. It requested access to websites and offered an “allow all relevant sites” option, which I selected. After that, it did not ask again.

Claude asked questions throughout the task, both about accessing websites for research and about extracting material from the World Bank site. Unlike GPT, it offered no “allow all relevant sites” option, so it requested permission each time: nine times for the US task (all .gov websites) and nine times for the Iran task (mainly .ir domains and also fa.wikipedia sources). I approved every request.

Claude also ran into a technical limit that triggered a different kind of HITL moment. For both Iran and the US, it couldn’t load the live GPPD country profiles, because the portal builds its pages with JavaScript, and Claude’s sandbox network policy blocked access to the World Bank data file. Claude stopped and asked whether I wanted to upload the page (as a pdf) myself or let the work continue without it. I skipped the question, so it continued and took its information from the World Bank’s GPPD DataBank API instead. However, the DataBank holds 2018 data, while the portal shows a 2022 profile. As a result, Claude’s baseline data was different from GPT and Muse.

Muse did not ask for any permission until the fourth part of the task, which required registering on the World Bank website and uploading information.

All three agents completed the task up to the point of creating the spreadsheets, and described their confidence in the results they generated.

The final part of the task, registering on the World Bank portal and uploading the changes, is where things became more interesting because it required the agents to take a more significant actions rather than just gather information.

Claude and GPT both stopped at this point and handed the registration and uploading over to me. Muse, however, kept going. Without asking me or showing me the Terms of Service, it registered an account under the email address david.jones@gsa.gov. You can see Muse’s full back and forth here.

The table below summarizes how each agent approached this last part of the task and cybersceuirty implications about it.

What each agent did when asked to register on the World Bank portal. Claude declined, GPT handed the form back to me, and Muse registered as test personas and accepted the terms without showing them to me.

There is an ongoing debate about whether AI labs should expose a model’s full chain of thought (CoT) and action trace, and if so, how much. Labs have given several reasons for holding back. OpenAI chose not to show o1’s raw CoT to users, citing user experience, competitive advantage, and the value of keeping the CoT available for internal monitoring. Anthropic noted that raw reasoning can contain incorrect or half-formed thoughts and that malicious actors could use it to build better jailbreaks. There is also a gaming and reward hacking concern, and “CoT unfaithfulness”.

To understand an agent’s behavior, however, evaluators need to know when and why things happen, which is only possible with a monitoring system in place and access to the agent’s complete trajectory. For an evaluator outside an AI lab, without that access, it is nearly impossible to fully make sense of an agent’s behavior. And if outside evaluators can only see partial trajectories, and any conclusions they draw can be dismissed for lacking complete information, what is the value of independent evaluation?

Knowing these limitations, I tried my best to collect, monitor, and check as much of each agent’s work as I could, again putting myself in the position of an ordinary researcher tasked with updating the World Bank information portal.

1. Since there is no one-click way for ordinary users to extract a complete record of an agent’s work trajectory, I watched each agent work live and recorded everything clickable and visible on screen.**** Once the task was finished, I gave the recordings to ChatGPT to extract the text and make it searchable. To give you a sense of what this looks like, here is a snippet (left: Claude, middle: Muse, right: GPT, sorry for the size and illegibility).

2. Self-reported trajectories. When the task was done, I prompted each agent to create a text file describing what it did, including errors, how it handled them, workarounds it used, websites it searched, and more. Muse and GPT each produced a downloadable .txt file, while Claude declined, stating that it went against its safety policy, stating “reasoning_extraction.”

Claude refusing to create work trajectory .txt file, citing concerns about “reasoning extraction”

That said, self-reported trajectories can not be fully trusted; I have seen mismatches in the past between what agents actually did and what they reported. So I gave these reports little weight, but if you’re interested in reviewing them and spotting matches or mismatches, the files are here.

3. Analysis. I then used the output Excel sheets and the text extracted from the screen recordings to conduct the analysis, both on my own and with help from Claude Code to sift through the data and generate tables. I cross-checked all of the data myself.

Below is some information about how much information on agents work trajectory is available in each LLM agent’s web UI.

What a person watching each agent could see in the interface during the runs.

A few observations and then I’ll get to my points:

• All three agents answered in fluent Farsi, but the Iran results were far weaker than the US ones. Of Iran’s 138 N/A fields, GPT and Muse each filled only 21 with a real value; for the US, they filled 51 and 64 of 130.

• For the US, 76–89% of each agent’s citations came from official government sites and the rest from legitimate international organization websites. For Iran, it was 11–22%.

• Low-authority sources crept in, including a Telegram channel, a Medium post, Grokipedia, or websites run by Iranian diaspora media groups such as Iran International. Claude seemed to be more conservative about finding workarounds when websites were unavailable and often preferred English language sources even with low legitimacy.

• Claude could only open 3 out of the 16 Farsi pages it tried. GPT and Muse cited 11 Persian sources each but showed reading only 3 and 7 of them respectively.

My point is not that I expected the Iran/Farsi tasks to have the same outcomes as the US/English ones. After all, the Iranian government has made it very difficult for foreign IP addresses to access official websites and domains ending in .ir (you can read more about this in the context of Iran’s National Information Network). My point is about the agents’ differing workarounds and source prioritization.

For me, this brought to mind the digital rights and language inclusion work that the good people of Global Voices have done for years, including on net neutrality and language access. What does all this mean for an AI agents era? And from an AI sovereignty perspective? One of AI sovereignty’s promises has been language diversity and support for local languages. LLM output quality, and perhaps safeguards, keep improving, but we also need to think about what language localization should look like in agents reasoning, searching, and prioritizing sources.

Looking through the agents’ trajectories, I noticed that they differed not only in which websites they could access, but also in how hard they tried when access failed. Some agents stopped after an initial failure, others tried alternate routes, different browsers, search-result snippets, cached or secondary sources, different fetch methods, and more.

So I ran a small follow-up test. I selected websites that the agents had accessed inconsistently during the original task and gave each agent a simple instruction: “Here is a list of websites. Look them up and write a one-paragraph summary of each.” The point was not to evaluate the quality of the summaries, but to observe what each agent did when direct access failed.

Sixteen Iranian government, media, reference and legal URLs, opened by me in Firefox from the US without a VPN, and by three AI agents

What each agent tried when a page didn’t load, and when it gave up.

In this case, the agents were trying to reach websites that were difficult to access from their own technical environments. But what happens when the access barrier come from the user’s environment instead?

For people in countries where governments filter or block websites, could an LLM or AI agent become another layer of information access? Could it retrieve, summarize, take actions, or relay information from websites that the user cannot reach directly? And could agentic workarounds make censorship circumvention easier — or conversely reproduce new restrictions through a different technical stack?

As Iranians who research information access and internet governance in Iran, my friend Farzaneh Badiei (a digital-rights lawyer) and I have been discussing how LLMs and AI agents might be used in censorship-circumvention contexts. I may explore this more in future installments of the Humane AI newsletter.

If you are interested in designing or conducting experiments on this topic, feel free to reach out at rpakzad@taraazresearch.org.

And, last but not least:

If you read Farsi, good luck making sense of the results on agents’ UIs!

To my fellow right-to-left readers and writers (~700 million people): you have my commiseration every time you perform the gymnastics of trying to write an Instagram caption, fill in a spreadsheet, read governments’ “accessible” translated forms, or copy and paste text across platforms.

Share Humane AI

Disclaimer: I used ChatGPT and Claude for copyediting. I use Claude Code for table generation, and supervised data analysis.