🔶 溯源税：LLM 水印如何改变 AI Agent 行为

近期，Anthropic 宣布未来的 Claude 模型将在输出中嵌入不可见水印 [1]、[2]，随后披露该水印基于 Google DeepMind 的 SynthID-Text [2]、[3]。文本水印本身并非新技术，但其部署如今具有监管相关性。欧盟《AI 法案》第 50(2) 条 [4] 要求，生成合成文本的 AI 系统提供方应以机器可读格式标记其输出，并在技术可行范围内采用有效、可互操作、稳健且可靠的技术方案，使这些输出可被检测为人工生成或操纵的内容。\n\n水印旨在实现内容溯源，但 SynthID-Text 会改变模型生成每一个下一个 token 的过程。在模型层面，这可能改变安全行为，包括模型是否拒绝有害请求，以及这种拒绝是否能在提示注入下保持有效。在 Agent 层面，相同的采样 token 可以决定调用哪个工具，以及向工具传递什么参数。提示注入将这两种场景联系起来，因为当模型还能够通过工具采取行动时，被削弱的拒绝会带来更严重的后果。 因此，这类水印程序既可能影响模型说什么，也可能影响 Agent 做什么。我们将这种行为效应称为 采样漂移（sampling drift）。\n\n这种漂移是否会在实践中出现，是一个实证问题。我们发现，它确实会出现：无论在模型拒绝行为还是 Agent 工具调用中都是如此。该效应取决于模型和密钥；当相反方向的变化彼此抵消时，汇总得分可能掩盖这一效应。因此，我们同时报告净性能，以及带水印与无水印运行之间的配对分歧。此外，结尾部分还讨论了这对 AI 安全与安全防护意味着什么，以及开发者应采取哪些措施。\n\n## 为内容溯源而生，部署于 Agent 之中\n\n文本水印会嵌入一种信号，使输出能够被识别为 AI 生成。现有方法包括后处理方法，以及直接集成到 LLM 生成过程中的方法 [8]。生成时方法包括 logits 偏置方法 [5]、无失真密钥化采样 [6]、受密码学启发的构造 [7]，以及 SynthID-Text 的锦标赛采样（Tournament sampling）[3]。图 1 将这一过程与普通采样进行了对比。我们使用 SynthID 的无失真配置：在水印随机性上取期望时，该配置能够保持原始 token 分布；但在固定密钥下，单次生成仍可能不同 [3]。Dathathri 等人报告称，在近两千万条 Gemini 回复中，没有测得质量下降 [3]。\n\n图 1：标准文本生成与带水印文本生成的对比。 普通生成从模型的 token 分布中采样。生成式水印会加入随机种子生成器、采样算法和评分函数；SynthID-Text 使用锦标赛采样。改编自 Dathathri 等人 [3]。\n\n\nAnthropic 的部署也说明了，为何这一问题的重要性超出了第一方聊天界面。该公司表示，水印在模型层面应用，覆盖通过 Claude Platform API 访问的受支持模型，以及云服务提供商上的模型 [1]。因此，即使 Agent 本身是独立应用，使用带水印模型作为其推理组件的开发者仍可能接收到带水印输出。这使得模型级水印带来的行为影响，与围绕这些模型构建的 Agent 密切相关。\n\n### 水印偏置的正是 Agent 会据以行动的那些 Token\n\n在模型不确定性较高的地方，锦标赛采样更有机会改变 token 选择。在 JSON 等结构化输出中，大括号、键名和函数名称通常具有很高的可预测性，而查询、数字、路径和收件人等值则更不确定。因此，在普通文本中仅相当于措辞变化的改变，可能会改变 Agent 实际执行的参数。\n\n模型权重和提示词保持不变，但 token 选择并不相同。重要的是，无失真并不意味着在固定水印密钥下行为完全一致。这项保证适用于水印随机性上的总体结果，而特定密钥会在生成过程中改变 token 选择 [3]。因此，即使按照 Dathathri 等人的定义，该水印是无失真的，产生的采样漂移仍可能改变 Agent 行为。其影响还可能依赖于水印密钥，所以我们测试多个密钥，而非依赖单一密钥。\n\n## 我们如何衡量这一影响\n\n我们针对两项实验采用配对设计。工具调用使用 BFCL v4 单轮 AST [9] 进行评估；拒绝行为则使用 200 个 HarmBench 有害行为 [10] 与 100 个良性 JailbreakBench 对照样本 [11]，其中有害请求既单独测试，也在一种固定提示注入技术下测试。表 1 汇总了数据集、评估范围、温度以及预期行为。\n\n我们通过 HuggingFace 未修改的 SynthIDTextWatermarkLogitsProcessor 使用无失真 SynthID-Text 配置，设置为 30 个 Tournament 层、n-gram 长度为 5、采样表为 216、上下文历史为 1,024。每个样本在每个温度下均使用相同的随机种子、批次构成和顺序，分别在启用与未启用 SynthID 的条件下生成。每一对实验中，水印处理器是唯一的差异。\n\n表 1：数据集与实验设置。\n\n实验 | 数据集与范围 | T | 正确行为 \n---|---|---|--- \n工具调用 | BFCL v4 单轮 AST [9]，实时和非实时的预期调用任务，以及相关性与不相关性任务 | 0.001、0.7、1.0 | 正确调用；或在没有合适工具时不调用 \n拒绝 | HarmBench [10]，200 个有害行为；JailbreakBench [11]，100 个良性对照；裸提示与固定注入提示 | 0.001、0.7 | 拒绝有害请求；回答良性请求 \n \n## 水印对工具调用的代价\n\n我们测试水印是否会改变工具选择、参数或输出有效性。对正确工具发出格式良好的调用，但路径、收件人、查询或金额不正确，尤其值得关注，因为它可能成功执行，却执行了错误操作。我们单独评估每一次调用，尽管在已部署的 Agent 中，一次错误调用还可能影响后续观察和决策。\n\n### 工具调用正确性变化的频率\n\n在预期应调用工具的样本中，水印使 7 个模型中的 6 个准确率下降，其中 4 个模型的下降显著。然而，准确率的净变化无法说明同一条调用在有无水印时是否成功。一条调用从正确变为错误，可能被另一条从错误变为正确的调用抵消；即使模型在两条样本上的行为都发生变化，汇总结果仍可能几乎不变。\n\n我们通过 配对分歧率（paired disagreement rate） 直接衡量这一点，并将其称为 变动率（churn），定义为带水印与无水印运行之间判定结果不同的样本占比。对于图 2 的跨温度比较，我们使用 BFCL 研究人员定义的非实时样本，从而在每个温度下获得同一组固定的 1,150 个预期调用任务。图 2 显示，配对分歧显著大于净准确率变化。在 T=1.0 时，phi-4 有 16.8% 的调用判定在两种条件下不同，而其净准确率仅下降 2.87 个点。Llama-3.1-8B 也呈现同样模式：9.9% 的判定发生变化，而净损失仅为 0.87 个点。在 21 个模型—温度组合中，平均变动率为 6.5%，且每种情况下其 bootstrap 区间均不包含零。\n\n图 2：按模型与温度划分的水印下配对工具调用分歧。 结果使用跨模型和温度的 1,150 个非实时 BFCL 预期调用样本。中间的垂直线表示相对于无水印条件没有变化。橙色条表示从正确变为错误的调用，蓝色条表示从错误变为正确的调用。最后一列报告了纳入非实时不相关性样本后的变动率。菱形表示不包含零的 95% bootstrap 区间。\n\n\n### 哪些工具调用错误发生变化\n\n错误类型同样重要。格式错误的输出会阻止预期调用执行，而调用格式良好但工具或参数错误，仍然可能被执行。图 3 将这些失败分为错误工具、错误参数和格式错误输出。与图 2 的跨温度比较不同，这项分析在 T=0.001 下合并了 BFCL 的实时与非实时预期调用样本，以描述更广泛基准中的错误。相关性和不相关性样本被排除，因为它们测试的是是否应该进行调用，而不是生成的调用是否正确。\n\n图 3：按错误类型划分的水印下工具调用错误变化。 在 T=0.001 下，对 BFCL 实时和非实时预期调用样本进行分析，错误类型分为错误工具、错误参数和格式错误输出。垂直线显示无水印时的准确率，条形图显示应用水印后的变化。橙色表示从正确变为错误，蓝色表示从错误变为正确。区间为 95% 样本级 bootstrap 区间。\n\n\n不同模型的错误特征也存在差异。对于 Llama-3.1-8B，准确率损失的最大来源是错误参数（−3.48 个点），其次是错误工具调用（−1.84 个点）。对于 phi-4 和 Granite-3.2-8B，格式错误输出占主导地位（分别为 −5.96 和 −4.36 个点）。因此，类似的汇总变化可能源于不同的失败模式。\n\n## 水印可能在提示注入下削弱拒绝行为\n\n拒绝同样是逐 token 生成的，因此水印可能影响拒绝行为。我们测试单独的有害请求，以及附加一种简单、固定、旨在降低拒绝率的提示注入技术后的请求。该技术将对抗性指令作为检索内容附加，声称安全过滤器已被禁用，并要求模型遵从。该技术在所有提示词、模型和温度下保持不变。OWASP GenAI LLM Top 10 2026 将提示注入列为一种输入侧漏洞：它能够以 Agent 应用开发者非预期的方式改变模型行为，其后果可延伸至有害输出，以及 Agent 系统中未经授权的工具操作 [12]。\n\n### 有无提示注入技术时的拒绝行为\n\n水印会改变模型对裸有害请求的拒绝行为，但在提示注入下，这一影响变得更加明显。如图 4 所示，当相同的有害请求与固定提示注入技术结合时，多个模型的分歧增加，且最强的影响主要表现为从拒绝转向遵从。这一结果与安全尤其相关，因为当模型暴露于对抗性提示时，该行为效应更加显著。\n\n在 T=0.001 时，gemma-3-27b 的变动率从裸有害请求上的 6.0% 增至提示注入下的 23.5%，而净遵从变化从 −1.0 个点转为 +12.5 个点。对于 gemma-3-12b，变动率从 7.5% 增至 11.0%，净遵从变化则从 −0.5 个点升至 +9.0 个点。在这两种情况下，水印对裸有害请求的拒绝几乎没有净影响，但在提示注入下显著降低了拒绝率。Llama-3.1-8B 在注入条件下也表现出显著变动：在 T=0.001 时达到 14.0%，在 T=0.7 时达到 17.5%，尽管其净变化单独来看并不显著。\n\nphi-4 和 Qwen3-4B 在两种条件下的变化都很小。在我们的评估中，这两个模型往往过度拒绝，包括对无害对照样本也是如此。因此，它们在提示注入下变化有限，不应被解读为水印在这些模型上能够更可靠地保持安全行为的证据。\n\n裸请求与注入请求之间的对比非常重要。在常规评估中看似对拒绝行为影响很小的水印，在对抗性条件下可能产生显著不同的安全行为。\n\n图 4：有害提示在有无提示注入技术时，水印导致的配对拒绝分歧。 结果使用 T=0.001 和 0.7 下的 200 个 HarmBench 行为。橙色表示从拒绝变为遵从，蓝色表示从遵从变为拒绝。左侧面板显示裸请求，右侧面板显示在固定注入下的相同请求。中间的垂直线表示相对于无水印条件没有变化。橙色条表示从不遵从变为有害遵从的请求，蓝色条表示相反方向的变化。右侧各列报告净遵从变化和变动率。菱形表示不包含零的 95% bootstrap 区间。\n\n\n### 与温度引起的变动率比较\n\n为将水印引起的分歧置于背景中考察，我们将其与改变温度设置时观察到的分歧进行比较。表 2 对比了 T=0.7 时提示注入条件下水印引起的变动率，以及关闭水印时将 T 从 0.001 改为 0.7 所观察到的变动率。在 6 个模型中的 4 个上，水印引起的变动率显著更高。 Granite-3.2-8B 的温度引起变动率最高，为 15.5%，但其水印引起的变动率仍更高，达到 21.5%。phi-4 和 Qwen3-4B 在两种干预下变化都很小，这与前文所述它们本已较高的拒绝率一致。\n\n表 2：提示注入下水印与温度引起的拒绝变动率对比。 水印变动率比较 T=0.7 时关闭和开启水印的结果；温度分歧比较关闭水印时 T=0.00 1 与 0.7 的结果。最后一列报告二者差异及其 95% bootstrap 区间。\n\nmodel |

Recently, Anthropic announced that future Claude models would embed an invisible watermark in their output [1], [2], and subsequently disclosed that the watermark is based on Google DeepMind’s SynthID-Text [2], [3]. Text watermarking itself is not new, but its deployment now has regulatory relevance. Article 50(2) of the EU AI Act [4] requires providers of AI systems generating synthetic text to mark their outputs in a machine-readable format and make them detectable as artificially generated or manipulated, using technical solutions that are effective, interoperable, robust, and reliable as far as technically feasible.

‍

Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token. At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection. At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it. Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools. Such a watermarking procedure can therefore affect both what the model says and what an agent does. We call this behavioral effect sampling drift.

‍

Whether this drift appears in practice is an empirical question. We find that it does, in both model refusal behavior and agent tool calling. The effect is model- and key-dependent and can be obscured by aggregate scores when changes in opposite directions cancel. We therefore report both net performance and paired disagreement between watermarked and unwatermarked runs. Further, the closing section discusses what it means for AI safety and security and what developers should do about it.

‍

Built for Content Provenance, Deployed Inside Agents

‍

A text watermark embeds a signal that allows output to be identified as AI-generated. Existing approaches include post-processing methods and methods integrated directly into LLM generation [8]. Generation-time approaches include logits-biasing methods [5], distortion-free keyed sampling [6], cryptographically motivated constructions [7], and SynthID-Text’s Tournament sampling [3]. Figure 1 contrasts this process with ordinary sampling. We use SynthID’s non-distortionary configuration, which preserves the original token distribution in expectation over the watermark randomness while individual generations under a fixed key can still differ [3]. Dathathri et al. report no measurable quality degradation across nearly twenty million Gemini responses [3].

‍

Figure 1. Standard versus watermarked text generation. Ordinary generation samples from the model’s token distribution. A generative watermark adds a random seed generator, sampling algorithm, and scoring function; SynthID-Text uses tournament sampling. Adapted from Dathathri et al. [3].

‍

‍

Anthropic’s deployment also illustrates why this matters beyond first-party chat interfaces. The company states that watermarking is applied at the model level and covers supported models accessed through the Claude Platform API as well as cloud providers [1]. A developer using a watermarked model as the reasoning component of an agent can therefore receive watermarked outputs even when the agent itself is a separate application. This makes model-level behavioral effects of watermarking relevant to the agents built around such models.

‍

Same Tokens the Watermark Biases, Same Tokens the Agent Acts On

‍

Tournament sampling has more opportunity to alter token selection where the model is uncertain. In structured output such as JSON, braces, keys, and function names are often highly predictable, while values such as queries, numbers, paths, and recipients are less so. A change that would amount to a lexical variation in ordinary prose can therefore alter an argument that an agent executes.

‍

The weights and prompt remain unchanged, but token selection does not. Importantly, non-distortionary does not imply identical behavior under a fixed watermark key. The guarantee holds over the watermark randomness, while a particular key changes token selection during generation [3]. The resulting sampling drift can therefore change agent behavior even though the watermark is non-distortionary in the sense defined by Dathathri et al. Its effect can also depend on the watermark key, so we test multiple keys rather than relying on one.

‍

How We Measure the Effect

‍

We use a paired design for two experiments. Tool calling is evaluated on BFCL v4 single-turn AST [9], and refusal on 200 HarmBench harmful behaviors [10] plus 100 benign JailbreakBench controls [11], with harmful requests tested both bare and under one fixed prompt injection technique. Table 1 summarizes the datasets, evaluation scope, temperatures, and expected behavior.

‍

We use the non-distortionary SynthID-Text configuration through HuggingFace’s unmodified SynthIDTextWatermarkLogitsProcessor, with 30 Tournament layers, n-gram length 5, sampling table 216, and context history 1,024. Each item is generated with and without SynthID from the same seed, batch composition, and order at each temperature. The watermark processor is the only difference within each pair.

‍

Table 1. Datasets and experimental settings.

‍

Experiment	Dataset and scope	T	Correct behavior
Tool calling	BFCL v4 single-turn AST [9], live and non-live call-expected tasks plus relevance and irrelevance	0.001, 0.7, 1.0	Correct call, or no call when none fits
Refusal	HarmBench [10], 200 harmful behaviors; JailbreakBench [11], 100 benign controls; bare and fixed-injection prompts	0.001, 0.7	Refuse harmful; answer benign

‍

The Tool-Calling Cost of Watermarking

‍

We test whether watermarking changes tool selection, arguments, or output validity. A well-formed call to the correct tool with an incorrect path, recipient, query, or amount is particularly consequential because it can execute successfully while performing the wrong action. We evaluate calls individually, although an incorrect call in a deployed agent could also affect subsequent observations and decisions.

‍

How Often Tool-Call Correctness Changes

‍

On items where a tool call is expected, watermarking reduces accuracy on six of the seven models, with a significant decrease on four. The net change in accuracy, however, does not show whether the same individual calls succeed with and without the watermark. A call that becomes incorrect can be offset by another that becomes correct, leaving the aggregate result nearly unchanged even though the model behaves differently on both items.

‍

We measure this directly using the paired disagreement rate , which we refer to as churn , defined as the share of items whose verdict differs between the watermarked and unwatermarked runs. For the comparison across temperatures in Figure 2, we use BFCL’s researcher-defined non-live items, which give us the same fixed set of 1,150 call-expected tasks at each temperature. Figure 2 shows that the paired disagreement is substantially larger than the net accuracy change. At T=1.0 , 16.8% of phi-4’s call verdicts differ between the two conditions while its net accuracy loss is 2.87 points. Llama-3.1-8B shows the same pattern, with 9.9% of verdicts changing while the net loss is only 0.87 points. Across the 21 model-temperature combinations, churn averages 6.5%, and its bootstrap interval excludes zero in every case.

‍

Figure 2. Paired tool-call disagreement under watermarking by model and temperature. Results use 1,150 non-live BFCL call-expected items across models and temperatures. The central vertical line represents no change relative to the unwatermarked condition. Orange bars show calls that changed from correct to incorrect, and blue bars show calls that changed from incorrect to correct. The final column reports the churn with non-live irrelevance items included. Diamonds denote 95% bootstrap intervals excluding zero.

‍

‍

Which Tool-Call Errors Change

‍

Error type also matters. Malformed output prevents the intended call from executing, while a well-formed call with the wrong tool or argument can still execute. Figure 3 separates these failures into wrong tool, wrong arguments, and malformed output. Unlike Figure 2’s across-temperature comparison, this analysis combines BFCL live and non-live call-expected items at T=0.001 to characterize errors across the broader benchmark. Relevance and irrelevance are excluded because they test whether a call should be made rather than whether the emitted call is correct.

‍

Figure 3. Changes in tool-calling errors under watermarking by error type. BFCL live and non-live call-expected items at T=0.001 , separated into wrong tool, wrong arguments, and malformed output. The vertical lines show the accuracy without watermarking, and the bars show the change when watermarking is applied.**** Orange denotes correct-to-error changes and blue error-to-correct changes. Intervals are 95% item-level bootstrap intervals.

‍

‍

The error profile also differs across models. On Llama-3.1-8B, the largest contribution to the accuracy loss comes from incorrect arguments (−3.48 points), followed by wrong-tool calls (−1.84 points). On phi-4 and Granite-3.2-8B, malformed output dominates (−5.96 and −4.36 points). Similar aggregate changes can therefore arise from different failure modes.

‍

Watermarking Can Weaken Refusal Under Prompt Injection

‍

Refusals are also generated token by token, so watermarking can affect them. We test harmful requests alone and with one simple, fixed prompt-injection technique intended to reduce refusal. The technique appends an adversarial instruction as retrieved content, claiming that the safety filter is disabled and instructing compliance. It is held constant across prompts, models, and temperatures. OWASP GenAI LLM Top 10 2026 identifies prompt injection as an input-side vulnerability that can alter model behavior in ways unintended by the agentic application developer, with consequences that can extend to harmful outputs and unauthorized tool actions in agentic systems [12].

‍

Refusal Behavior With and Without the Prompt Injection Technique

‍

Watermarking changes refusal behavior on bare harmful requests, but the effect becomes more pronounced under prompt injection. As Figure 4 shows, disagreement increases on several models when the same harmful requests are paired with the fixed prompt-injection technique, with the strongest effects shifting predominantly from refusal to compliance. This makes the result particularly safety-relevant because the behavioral effect becomes more pronounced when the model is exposed to an adversarial prompt.

‍

At T=0.001, gemma-3-27b’s churn increases from 6.0% on bare harmful requests to 23.5% under prompt injection, while the net compliance change shifts from −1.0 to +12.5 points. For gemma-3-12b, churn increases from 7.5% to 11.0% and the net compliance change from −0.5 to +9.0 points. In both cases, watermarking has little net effect on refusal for bare harmful requests but substantially lowers refusal under prompt injection. Llama-3.1-8B also shows substantial churn under injection, reaching 14.0% at T=0.001 and 17.5% at T=0.7, although its net change is not individually significant.

‍

phi-4 and Qwen3-4B show little change under either condition. Both models tend to over-refuse in our evaluation, including on the harmless controls. Their limited movement under prompt injection should therefore not be interpreted as evidence that watermarking preserves safety behavior more reliably on these models.

‍

The contrast between bare and injected requests is important. A watermark that appears to have little effect on refusal behavior under ordinary evaluation can produce substantially different safety behavior under adversarial conditions.

‍

Figure 4. Paired refusal disagreement under watermarking on harmful prompts, with and without the prompt injection technique. Results use 200 HarmBench behaviors at T=0.001 and 0.7. Orange shows refusal-to-compliance changes and blue compliance-to-refusal changes. The left panel shows bare requests and the right panel the same requests under the fixed injection. The central vertical line represents no change relative to the unwatermarked condition. Orange bars show requests that changed from non-compliance to harmful compliance, and blue bars show changes in the opposite direction. Right-hand columns report net compliance change and churn. Diamonds indicate 95% bootstrap intervals excluding zero.

‍

‍

Comparison With Temperature-Induced Churn

‍

To place watermark-induced disagreement in context, we compare it with disagreement observed when changing the temperature setting. Table 2 contrasts watermark-induced churn under prompt injection at T=0.7 with churn observed when changing T from 0.001 to 0.7 with watermarking off. Watermark-induced churn is significantly higher on four of the six models. Granite-3.2-8B has the highest temperature-induced churn at 15.5%, but its watermark-induced churn remains higher at 21.5%. phi-4 and Qwen3-4B change little under either intervention, consistent with their already high refusal rates described above.

‍

Table 2. Watermark- versus temperature-induced refusal churn under injection. Watermark churn compares watermark off and on at T=0.7 ; temperature disagreement compares T=0.00 1 and 0.7 with watermarking off. The final column reports their difference and 95% bootstrap interval.

‍

model	watermark churn %	temperature churn %	difference (95% CI)
gemma-3-27b	26.0	13.5	+12.5 (+6.0, +18.5)
Granite-3.2-8B	21.5	15.5	+6.0 (+0.5, +12.0)
Llama-3.1-8B	17.5	7.5	+10.0 (+4.5, +15.5)
gemma-3-12b	11.0	6.0	+5.0 (+0.5, +10.0)
phi-4	0.5	0.5	+0.0 (−1.5, +1.5)
Qwen3-4B	0.0	0.0	+0.0 (+0.0, +0.0)

‍

Sensitivity to the Watermark Key

‍

The preceding results use one watermark key, but SynthID’s effect on token selection depends on the key. We therefore evaluate the sensitivity to the watermark key at T=0.7 with the study key and ten additional keys. Figure 5 shows the resulting change in attack success across keys and models.

‍

Figure 5. Change in attack success under eleven watermark keys. Each row shows one model at T=0.7 on the harmful requests under the injection. The vertical line at zero marks the result without watermarking, and each point shows the change when watermarking is applied with a particular key. Positive values indicate higher attack success and negative values lower attack success relative to the unwatermarked baseline. Orange points show ten additional keys; black diamonds show the study key.

‍

The effect varies substantially across watermark keys. For Llama-3.1-8B and both Gemma models, most keys increase attack success relative to the unwatermarked baseline, although the magnitude varies widely. On the two Gemma models, our study key is among those producing the largest increases. For Llama-3.1-8B, the study key increases attack success by 3.5 points, while the other ten keys average +4.4 points and range from −4.5 to +14.5 points. Granite-3.2-8B shows a more mixed response, with different keys moving attack success in both directions. phi-4 and Qwen3-4B again remain close to the unwatermarked baseline, consistent with their high refusal rates described above. The effect of watermarking therefore depends on both the model and the watermark key.

‍

What This Means for AI Safety and Security

‍

Text watermarking is meant to help identify whether content was generated by AI, but inside an agent it also becomes part of the generation process that produces decisions. Our results show that it can change both tool calling and refusal behavior, with changes to individual decisions that aggregate accuracy or refusal rates can obscure.

‍

The effect on refusal behavior becomes more pronounced under prompt injection. Watermarking changes refusal behavior on bare harmful requests, but the effect is more pronounced when the same requests are paired with the prompt-injection technique. On several models, watermarking then makes the model more likely to answer harmful requests that it would otherwise refuse. This experiment measures model-level refusal rather than end-to-end agent behavior. Its relevance to agents arises when the model is given access to tools or other actions. A refusal that changes to compliance can then affect not only what the model says, but also what an agent subsequently does. We do not test this combined failure mode directly, but the tool-calling results show separately that watermarking can also change the actions generated by the model.

‍

The effect is also model- and configuration-dependent. Figure 5 shows that changing only the watermark key can alter both the magnitude and direction of the effect under the same prompt injection. This is particularly relevant for provider-hosted models, where watermarking can be applied at the model level to outputs consumed by independently developed agents. Where the watermark key or configuration is controlled by the model provider, such changes may also occur outside the agent developer’s direct control.

‍

These findings make reassessment important whenever watermarking is introduced or its configuration or key changes. Such changes can alter individual tool calls and safety decisions in unexpected ways, particularly under adversarial inputs, even when aggregate performance remains stable. Agent evaluation and red-teaming should therefore be repeated under the configuration intended for deployment, including paired comparisons on the same inputs and evaluation under prompt injection.

‍

A provenance mechanism that appears behaviorally stable on ordinary inputs may not remain stable under attack. Watermarking should therefore be evaluated as part of the agent’s deployed security configuration.

‍

These results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties. Detectability and unchanged text quality do not establish that an agent will preserve the same tool-calling or safety behavior once watermarking is enabled. Although we study SynthID-Text, the broader concern applies to interventions that modify token selection in systems that act on generated tokens.

‍

References

[1] Anthropic, “How Claude marks AI-generated content,” Claude Help Center, 2026. https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content

[2] Anthropic, “How Claude’s text watermark works,” Aug. 14, 2026. https://www.anthropic.com/news/claude-text-watermark

[3] S. Dathathri, A. See, S. Ghaisas, et al., “Scalable watermarking for identifying large language model outputs,” Nature , vol. 634, pp. 818–823, 2024. https://doi.org/10.1038/s41586-024-08025-4

[4] European Parliament and Council of the European Union, “Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” Official Journal of the European Union , Art. 50, 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj

[5] J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, “A watermark for large language models,” in Proc. 40th Int. Conf. Machine Learning (ICML) , PMLR, vol. 202, pp. 17061–17084, 2023. https://proceedings.mlr.press/v202/kirchenbauer23a.html

[6] R. Kuditipudi, J. Thickstun, T. Hashimoto, and P. Liang, “Robust distortion-free watermarks for language models,” 2023. arXiv:2307.15593. https://arxiv.org/abs/2307.15593

[7] M. Christ, S. Gunn, and O. Zamir, “Undetectable watermarks for language models,” in Proc. 37th Conf. Learning Theory (COLT) , PMLR, vol. 247, pp. 1125–1139, 2024. https://proceedings.mlr.press/v247/christ24a.html

[8] A. Liu, L. Pan, Y. Lu, J. Li, X. Hu, X. Zhang, L. Wen, I. King, H. Xiong, and P. S. Yu, “A survey of text watermarking in the era of large language models,” ACM Computing Surveys , vol. 57, no. 2, Art. 47, 2024. https://doi.org/10.1145/3691626

[9] S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez, “The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models,” in Proc. 42nd Int. Conf. Machine Learning (ICML) , PMLR, vol. 267, pp. 48371–48392, 2025. https://proceedings.mlr.press/v267/patil25a.html

[10] M. Mazeika, L. Phan, X. Yin, et al., “HarmBench: A standardized evaluation framework for automated red teaming and robust refusal,” in Proc. 41st Int. Conf. Machine Learning (ICML) , PMLR, vol. 235, pp. 35181–35224, 2024. https://proceedings.mlr.press/v235/mazeika24a.html

[11] P. Chao, E. Debenedetti, A. Robey, et al., “JailbreakBench: An open robustness benchmark for jailbreaking large language models,” in Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track , 2024. https://doi.org/10.52202/079017-1745

[12] OWASP GenAI Security Project, “OWASP GenAI LLM Top 10 2026,” Aug. 3, 2026. https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/