Noisegate：面向不可信 AI Agent 的差分隐私网关

让 AI Agent 可以查询敏感数据，同时提供数学保证：即使 Agent 出错、遭到操控或具有对抗性，任何个人记录也不会泄露。

一段录制的 Claude Desktop 会话（回复经过删减；图表卡片来自该会话本身）。AI Agent 按诊断类别拆分 20 名患者，±12 的噪声淹没了每一个柱状分组。系统提醒它不能关闭噪声后，它耗尽了仅可回答三次的预算，最终网关返回的是拒绝，而不是一个噪声更小的答案。在包含 32,561 行的人口普查数据上，过窄的数据切片会在信任边界被拒绝，而完整教育程度分布在大规模数据下能够清晰返回。拒绝和拦截均为在线网关的真实执行结果，通过 python scripts/render_demo_gif.py 复现。可通过 Claude Desktop 使用。

• 可运行的攻击，而非口头宣称。 三类经典隐私攻击（差分攻击、成员推断和通过重识别进行的个体锁定）均针对系统自身引擎运行。每种攻击都展示了在关闭隐私时如何成功、在开启隐私后如何失败，并在 CI 中进行回归测试，确保防御机制不会悄然失效。
• 独立验证的数学机制。 手写噪声机制在全部 35 项噪声尺度检查中与行业参考实现 OpenDP 相符，误差在 1e-9 以内；50 万样本的分布测试通过，同时设置了能证明测试本身会失败的正向对照。
• 通过更优记账获得 3 倍效用。 混合 zCDP 组合允许执行 308 次查询，而朴素预算记账在 100 次时即停止：隐私保证相同，可提问的问题数量是后者的三倍。
• 为 AI Agent 构建。 可作为 Claude Desktop 的 MCP 服务器运行。连接的 Agent 在设计上即被视为不可信，所有隐私属性都在其下游强制执行。
• 小而确定的信任边界。 LLM 提出查询；一个朴素但经过充分测试的验证层作出最终裁决。模型输出的任何内容都不能扩大其自身权限。
• **技术栈：**Python · DuckDB · FastAPI · Streamlit · MCP SDK · Docker · GitHub Actions，并在 CI 中运行 250 余项测试。

用五分钟完成评估：观看 GIF，然后浏览攻击展示。若要深入审查，请参阅 DESIGN.md。

---

你可以用自然语言就敏感数据集提问。LLM 会将每个问题编译为一个小型、受约束的查询；差分隐私引擎会在可追踪的隐私预算内执行该查询，并返回一个刻意加入了噪声、且附带置信区间的答案。这个名字很贴切。就像同名的音频设备，该网关会将低于设定阈值的每个信号压到噪声底以下：任何单个个体的贡献都会被淹没，而群体规模的信号则几乎不受影响地通过。无论提出什么问题、措辞多么巧妙，都无法从答案中重建任何个人记录。

有意思的部分不在于 LLM 能写查询，而在于：**隐私保证并不依赖 LLM 是否可信。**模型只是一个提出查询的便利工具；它不执行任何强制措施。每一项隐私属性都在下游由组件强制执行；即使由人手工输入查询，这些组件的行为也完全相同。这是在生产系统中处理任何不可信输入时都会采用的信任边界原则，如今被应用于 AI Agent。

---

核心看点：攻击展示

任何人都可以声称具备隐私保护。这个仓库附带了足以击破此类声称的攻击代码，将其用于自身引擎，并在 CI 中固定预期结果。理解该网关究竟保证了什么，最快的方法是观察它如何击败三种能攻破朴素“查询数据库”系统的经典攻击。

攻击 1——差分攻击

差分攻击通过提出两个仅相差某一个人的聚合问题，来隔离这个人。

Query A: "Total income of all 100 people in department X."        → $7,240,000
Query B: "Total income of all people in department X except Alice." → $7,135,000
Attacker computes: A − B = $105,000  ← Alice's exact salary, leaked.

两个查询都“只是聚合”。两者都没有点名任何单独的行；但组合起来却暴露了一名个体。展示程序（attacks/differencing.py）证明，此攻击在禁用隐私时能够成功：目标的私有值被精确恢复。（上面的工资示例仅用于说明；在真实的 UCI Adult 数据中，“Alice”是其群组中拥有最高资本收益的唯一者。）随后，它又展示同样的攻击如何在启用 DP 后失效：每个答案上经过校准的噪声使相减操作失去意义，预算会计系统会对两次查询所释放的信息统一计费，而不是将其视为彼此独立的查询。

攻击 2——成员推断

成员推断攻击用于判断某个特定个人是否存在于数据集中。对于许多数据集（如医疗研究或违约者名单），这一事实本身就是敏感信息。仅拥有查询权限的攻击者会尝试判定：“这个确切的人是否在数据中？”

攻击展示会在一系列隐私预算下运行该攻击（ε，即在答案准确度与隐私之间权衡的旋钮），并绘制结果：

一个最优（Neyman–Pearson）攻击者根据单次带噪 COUNT 判断一个人的成员身份，在真实引擎上跨 ε 运行。经验成功率（蓝色，95% Wilson 区间）紧贴解析 Laplace 曲线，并低于最坏情况下的 DP 上限（虚线）；随着 ε 缩小，其成功率会从确定无疑（DP 关闭）降至接近 0.5 的抛硬币水平。绿色的效用曲线（右轴）展示了同一轮扫描中聚合查询的相对误差：在攻击被阻止的区间内，其变化极小。通过 python -m attacks.membership 生成（每个 ε 进行 10,000 次试验）。

随着 ε 缩小（隐私增强），攻击者的成功率会跌向抛硬币水平。效用叠加曲线则展示了所付出的代价：能挫败单人攻击的同一份噪声，对群体规模的聚合结果几乎没有影响。隐私并非免费，图表准确展示了你为此交换了什么。扫描范围覆盖 ε = 8 至 0.5，涵盖整个区间；随附部署方案为每次查询扣除 ε = 0.05，位于该图左侧范围之外。在那里，单次查询攻击已与随机猜测无法区分。

攻击 3——通过重识别锁定个体

Latanya Sweeney 在 2002 年指出，邮政编码、出生日期和性别可以唯一识别约 87% 的美国人。要在数据集中定位一个人，并不需要对方的姓名；几个看似无害的属性就足够了。展示中的第三种攻击（attacks/patients_alice.py）在 20 名合成患者上复现了这一结构：仅凭性别和年龄，就能将队列缩小到恰好一个人——64 岁以上的唯一女性 Alice。

关闭隐私后，两个完全普通的人口统计直方图相减，即可精确恢复 Alice 的诊断结果。这不需要异常值；仅仅具备可重识别性就足够了，因此它是攻击展示中最强的一种攻击。启用隐私后，三道独立防线将其阻断：过滤器防护会直接拒绝明显过窄的查询（不消耗任何预算）；经过校准的噪声淹没两次查询的相减结果（信噪比约为 0.13，因此恢复出的“诊断”本质上是随机抽样）；预算机制会在平均化升级攻击可能奏效前很早就拒绝请求（约需重复 256 次，但仅负担得起 10 次）。这正是页面顶部演示 GIF 和 Claude Desktop 操作演示所实时复现的攻击。结果在 tests/test_attack_patients.py 中进行回归测试。

这些针对系统自身引擎运行的实验，是证明该保证真实、可理解，而非仅从库中导入并盲目信任的核心证据。

与 OpenDP 交叉核验

攻击验证的是我们自己的数学保证。为防范实现内部自洽但实际上错误的问题，该机制还会与作为独立参考的 OpenDP 进行交叉核验（attacks/crosscheck_opendp.py，opendp 0.15.1）。

**简而言之：手写机制与行业参考实现精确到小数点后九位一致；而一个故意错误校准的版本会在同样的测试中失败，因此这种一致性具有实际意义。**具体而言：

• 尺度一致性。 针对 COUNT、对每个已发布数值列进行裁剪的 SUM，以及直方图，在五个 ε 值下进行测试（共 35 次比较）：我们的敏感度 Δf = max(|L|, |U|) 与 OpenDP 的“先裁剪再求和”稳定性边界完全一致；在所有情况下，我们的噪声尺度 Δf/ε 均与 OpenDP 认证为 ε-DP 的最小尺度一致，误差在 1e-9 以内。
• 分布一致性。 生产采样器生成的 500,000 个噪声样本，与 OpenDP Laplace 测量机制在相同尺度下生成的 500,000 个样本，通过了双样本 KS 检验（在 α = 0.001 时，D = 0.0014，p = 0.70）。正向对照证明测试具有检验力：将同一采样器故意错误缩放 5% 后，会以 p ≈ 4×10⁻²² 被拒绝。

OpenDP 是参考实现，绝非实际运行时的实现：它仅存在于开发额外依赖中，运行系统不会导入它。尺度一致性在 tests/test_crosscheck_opendp.py 中进行回归测试（本地未安装 OpenDP 时跳过；CI 会安装它并设置 REQUIRE_OPENDP=1，因此该检查绝不会在 CI 中被静默跳过）。

攻击会在进程内针对真实 DP 引擎运行。由于它们构建的是手写 AST 而非经过 LLM 编译器，因此不需要 API key 或服务器：

python -m attacks.patients_alice # Attack 3: re-identify Alice and recover her diagnosis
                                 #           with DP off; watch guard + noise + budget
                                 #           defeat the same attack (no data fetch needed)

python scripts/fetch_data.py     # fetch UCI Adult into data/ (gitignored; ~4 MB)

python -m attacks.differencing   # Attack 1: recover Alice exactly with DP off, then watch
                                 #           noise + budget defeat the same subtraction
python -m attacks.membership     # Attack 2: sweep ε and regenerate the chart above

python -m attacks.crosscheck_opendp  # cross-check vs OpenDP (needs the dev
                                     # extras: pip install -e ".[dev]")

这些结果本身也在 tests/test_attack_differencing.py、tests/test_attack_membership.py 和 tests/test_attack_patients.py 中进行回归测试。因此，任何悄然削弱隐私保证的改动都会导致 CI 失败，而不只是生成一张更好看的图表。

---

三个阶段。只有第一个不可信，而这种不对称正是整个设计的关键。

  Natural language question
            │
            ▼
  ┌───────────────────┐
  │   LLM compiler    │   Emits a CONSTRAINED QUERY AST (not free SQL),
  │   (UNTRUSTED)     │   via structured/schema-forced output.
  └───────────────────┘
            │  query AST
            ▼
  ┌───────────────────┐
  │  Validation /     │   Policy check: allowed columns only, aggregates
  │  guardrail layer  │   only, declared value ranges, group-cardinality
  │   (TRUSTED)       │   caps. Invalid → repair loop or refusal.
  └───────────────────┘
            │  validated AST
            ▼
  ┌───────────────────┐
  │  Privacy engine   │   Clamp to declared range (bounds sensitivity),
  │   (TRUSTED)       │   add calibrated Laplace noise, decrement budget.
  └───────────────────┘
            │
            ▼
  Noisy answer + confidence interval + remaining budget

最重要的设计选择是标为 UNTRUSTED（不可信） 的方框。验证层之上的一切都被视为具有敌意；其下的一切则构成负责强制执行保证的可信计算基。LLM 编译到的是受约束的 AST，而不是任意 SQL，因为这种约束能够限制查询敏感度；而有界敏感度正是使噪声校准可靠的基础。自由形式 SQL 会重新打开该项目旨在关闭的攻击面。

完整架构、威胁模型、敏感度与噪声数学，以及预算记账设计，请参阅 DESIGN.md。

---

如实说明隐私模型

简明版本是：每个答案都会加入噪声；系统会按身份保存累积信息披露的精确账本；网关

Noisegate: a differential-privacy gateway for untrusted AI agents

Give an AI agent query access to sensitive data, with a mathematical guarantee that no individual's record can leak — even if the agent is wrong, manipulated, or adversarial.

A recorded Claude Desktop session (replies trimmed; the chart cards are the session's own). An AI agent breaks 20 patients down by diagnosis, and the ±12 noise swamps every bin. Reminded that it cannot turn the noise off, it drains a three-answer budget until the gate returns a refusal instead of a quieter answer. On the 32,561-row census, a too-narrow slice is rejected at the trust boundary , while a full education breakdown comes back clean at scale. The refusal and the rejection are the live gateway's real enforcement, reproduced by python scripts/render_demo_gif.py. Try it from Use it from Claude Desktop.

• Working attacks, not claims. Three classic privacy attacks (differencing, membership inference, and singling-out by re-identification) run against the system's own engine. Each is shown succeeding with privacy off, defeated with privacy on, and regression-tested in CI so the defense cannot quietly rot.
• Independently verified math. The hand-rolled noise mechanism matches OpenDP, the industry reference implementation, in all 35 noise-scale checks to within 1e-9, and 500,000-sample distribution tests pass with a positive control proving the test can fail.
• 3× the utility from better accounting. Hybrid zCDP composition admits 308 queries where naive budget accounting stops at 100: the same privacy guarantee, three times the questions.
• Built for AI agents. Runs as an MCP server for Claude Desktop. The connecting agent is untrusted by design, and every privacy property is enforced below it.
• A small, deterministic trust boundary. The LLM proposes queries; a boring, fully tested validation layer disposes. Nothing the model outputs can widen its own authority.
• Stack: Python · DuckDB · FastAPI · Streamlit · MCP SDK · Docker · GitHub Actions, with a 250+ test suite in CI.

Five minutes to evaluate: watch the GIF, then skim the attack gallery. Deep review: DESIGN.md.

---

You ask questions about a sensitive dataset in plain English. An LLM compiles each question into a small, constrained query; a differential-privacy engine executes it under a tracked privacy budget and returns a deliberately noisy answer with a stated confidence interval. The name fits. Like its audio namesake, the gateway keeps every signal below a set threshold under the noise floor: any one individual's contribution is drowned out, while population-scale signals pass through nearly untouched. No individual record can be reconstructed from the answers, no matter what is asked or how cleverly it is phrased.

The interesting part isn't that an LLM can write queries. It's that the privacy guarantee does not depend on the LLM being trustworthy. The model is a convenience that proposes a query; it enforces nothing. Every privacy property is enforced downstream, by components that would behave the same way if a human typed the query by hand. This is the trust-boundary discipline you'd apply to any untrusted input in a production system, applied here to an AI agent.

---

The headline: an attack gallery

Anyone can claim privacy. This repository ships the exploits that would break the claim, runs them against its own engine, and pins the outcomes in CI. The fastest way to understand what the gateway guarantees is to watch it defeat three classic attacks that break naive "query a database" systems.

Attack 1 — The differencing attack

A differencing attack isolates one person by asking two aggregate questions that differ by exactly that person.

Query A: "Total income of all 100 people in department X."        → $7,240,000
Query B: "Total income of all people in department X except Alice." → $7,135,000
Attacker computes: A − B = $105,000  ← Alice's exact salary, leaked.

Both queries are "just aggregates." Neither names a single row. Yet together they expose an individual. The gallery (attacks/differencing.py) shows this attack succeeding with privacy disabled : the target's private value is recovered exactly. (The salary sketch above is illustrative; on the real UCI Adult data, "Alice" is the unique holder of her group's maximum capital gain.) It then shows the same attack defeated once DP is on : the calibrated noise on each answer makes the subtraction useless, and the budget accountant charges for the information released across both queries rather than treating them as independent.

Attack 2 — Membership inference

A membership-inference attack determines whether a specific individual is in the dataset at all. For many datasets (a medical study, a list of defaulters), that fact is itself sensitive. An attacker with only query access tries to decide: "is this exact person in the data?"

The gallery runs this attack across a sweep of privacy budgets (ε — the dial that trades answer accuracy for privacy) and plots the result:

An optimal (Neyman–Pearson) attacker deciding one person's membership from a single noisy COUNT, run against the real engine across ε. Empirical success (blue, 95% Wilson intervals) hugs the analytic Laplace curve and stays below the worst-case DP ceiling (dashed); it collapses from certainty (DP off) toward the 0.5 coin-flip as ε shrinks. The green utility curve (right axis) shows an aggregate query's relative error over the same sweep, barely dented where the attack is defeated. Generated by python -m attacks.membership (10,000 trials per ε).

As ε shrinks (stronger privacy), the attacker's success collapses toward a coin flip. The utility overlay shows the price paid: the same noise that defeats the one-person attack barely moves a population-scale aggregate. Privacy is not free, and the chart shows exactly what you trade for it. The sweep spans ε = 8 down to 0.5 to cover the whole range; the bundled deployment charges ε = 0.05 per query , off the left edge of this chart, where the single-query attack is already indistinguishable from chance.

Attack 3 — Singling out by re-identification

Latanya Sweeney showed in 2002 that ZIP code + birth date + sex uniquely identify roughly 87% of Americans. You do not need someone's name to find them in a dataset; a few innocuous attributes will do. The gallery's third attack (attacks/patients_alice.py) reproduces that structure on 20 synthetic patients: sex and age alone narrow the cohort to exactly one person, Alice, the only woman over 64.

With privacy off, subtracting two perfectly ordinary demographic histograms recovers Alice's diagnosis exactly. No outlier value is required; being re-identifiable is enough, which makes this the strongest attack in the gallery. With privacy on, three independent defenses end it: the filter guard rejects the obvious narrow query outright (spending nothing); calibrated noise drowns the two-query subtraction (signal-to-noise ≈ 0.13, so the recovered "diagnosis" is essentially a random draw); and the budget refuses the averaging escalation long before it could work (~256 repetitions needed, 10 affordable). This is the attack the demo GIF at the top of this page, and the Claude Desktop walkthrough, replay live. Outcomes are regression-tested in tests/test_attack_patients.py.

These experiments, run against the system's own engine, are the core evidence that the guarantee is real and understood, not imported from a library and taken on faith.

Cross-checked against OpenDP

The attacks validate the guarantee against our own math. To guard against a self-consistent-but-wrong implementation, the mechanism is also cross-checked against OpenDP as an independent reference (attacks/crosscheck_opendp.py, opendp 0.15.1).

In short: the hand-rolled mechanism agrees with the industry reference to nine decimal places, and a version that is intentionally mis-calibrated fails the same tests, so the agreement means something. The details:

• Scale agreement. For COUNT, clamped SUM over every published numeric column, and a histogram, at five ε values (35 comparisons): our sensitivity Δf = max(|L|, |U|) equals OpenDP's clamp-then-sum stability bound exactly , and our noise scale Δf/ε matches the smallest scale OpenDP certifies as ε-DP to within 1e-9, in every case.
• Distribution agreement. 500,000 noise samples from the production sampler vs. 500,000 from OpenDP's Laplace measurement at the same scale pass a two-sample KS test (D = 0.0014, p = 0.70 at α = 0.001). A positive control proves the test has power: the same sampler, intentionally mis-scaled by 5%, is rejected at p ≈ 4×10⁻²².

OpenDP is the reference, never the implementation: it lives in the dev extras only, and the running system does not import it. The scale agreement is regression-tested in tests/test_crosscheck_opendp.py (skipped without OpenDP locally; CI installs it and sets REQUIRE_OPENDP=1 so the check can never silently skip there).

The attacks run in-process against the real DP engine. No API key and no server are needed, since they build hand-written ASTs rather than going through the LLM compiler:

python -m attacks.patients_alice # Attack 3: re-identify Alice and recover her diagnosis
                                 #           with DP off; watch guard + noise + budget
                                 #           defeat the same attack (no data fetch needed)

python scripts/fetch_data.py     # fetch UCI Adult into data/ (gitignored; ~4 MB)

python -m attacks.differencing   # Attack 1: recover Alice exactly with DP off, then watch
                                 #           noise + budget defeat the same subtraction
python -m attacks.membership     # Attack 2: sweep ε and regenerate the chart above

python -m attacks.crosscheck_opendp  # cross-check vs OpenDP (needs the dev
                                     # extras: pip install -e ".[dev]")

The outcomes are themselves regression-tested in tests/test_attack_differencing.py, tests/test_attack_membership.py, and tests/test_attack_patients.py, so a change that quietly weakened the guarantee would fail CI rather than just producing a prettier chart.

---

Three stages. Only the first is untrusted, and that asymmetry is the whole idea.

  Natural language question
            │
            ▼
  ┌───────────────────┐
  │   LLM compiler    │   Emits a CONSTRAINED QUERY AST (not free SQL),
  │   (UNTRUSTED)     │   via structured/schema-forced output.
  └───────────────────┘
            │  query AST
            ▼
  ┌───────────────────┐
  │  Validation /     │   Policy check: allowed columns only, aggregates
  │  guardrail layer  │   only, declared value ranges, group-cardinality
  │   (TRUSTED)       │   caps. Invalid → repair loop or refusal.
  └───────────────────┘
            │  validated AST
            ▼
  ┌───────────────────┐
  │  Privacy engine   │   Clamp to declared range (bounds sensitivity),
  │   (TRUSTED)       │   add calibrated Laplace noise, decrement budget.
  └───────────────────┘
            │
            ▼
  Noisy answer + confidence interval + remaining budget

The most important design choice is the box labeled UNTRUSTED. Everything above the validation layer is treated as hostile. Everything below it is the trusted computing base that enforces the guarantee. The LLM compiles to a constrained AST rather than arbitrary SQL because that constraint is what bounds query sensitivity, and bounded sensitivity is what makes the noise calibration sound. Free-form SQL would reopen exactly the attack surface this project closes.

See DESIGN.md for the full architecture, the threat model, the sensitivity and noise math, and the budget-accounting design.

---

The privacy model, stated honestly

The plain version: every answer is noised, an exact ledger of cumulative disclosure is kept per identity, and the gateway refuses to answer before that ledger can exceed the guarantee it advertises. The rest of this section spells out the ledger's math. It matters to privacy engineers, and everyone else can skim it.

Each answer is produced by the pure-ε Laplace mechanism, with calibrated noise and a stated confidence interval. Cumulative privacy loss across queries is tracked by a hybrid accountant that maintains two simultaneously valid bounds: the pure sum of ε, and a zero-concentrated DP (zCDP) total in which each ε-DP release costs ρ = ε²/2 and ρ adds across queries. It refuses further queries only when both are exhausted. Light usage keeps the conservative pure-ε capacity; bulk usage gains zCDP's √k composition, which at small per-query ε answers several times more queries at the same total budget.

At the default ε_q = 0.05 against the same (ε = 5, δ = 10⁻⁶) budget, plain sum-of-ε accounting refuses after 100 queries; the hybrid answers 308 before refusing, the same guarantee for three times the questions. The gain grows as the per-query ε shrinks, and above the break-even it degrades cleanly: the hybrid falls back to the pure-ε capacity rather than dropping below it. Regenerate this, the break-even, and the δ-sensitivity views with python benchmarks/composition_comparison.py.

The cost is worth stating just as clearly: the headline guarantee is (ε, δ)-differential privacy, which is strictly weaker than pure ε-DP. With probability at most δ the ε bound is allowed to fail. δ is not extra noise; it is failure mass. So it is derived from the dataset rather than hard-coded: δ = min(10⁻⁶, 1/(30·n)), at least 30× below 1/n for any served dataset (the same proportional margin the 2020 US Census production system chose), which for the bundled Adult set (n = 32,561) gives 10⁻⁶. A user whose pure-ε ledger never crosses the budget also keeps the full pure ε-DP guarantee. DESIGN.md sections 3 and 5.4 give the math, the break-even, and the trade in full.

---

Quickstart — three ways in

1. No API key, no setup — run the attacks. The gallery above executes in-process against the real engine; python -m attacks.patients_alice needs nothing fetched at all.

2. No API key — connect an AI agent. The gateway runs as an MCP server for Claude Desktop; the walkthrough is below.

3. The full natural-language UI (a local single-tenant demo) — an API key for the untrusted NL→query compiler is the only extra requirement:

export ANTHROPIC_API_KEY=...   # used only by the untrusted NL→query compiler
docker compose up              # brings up the engine, API, and UI
# open http://localhost:8501

This HTTP + Streamlit surface is a local, single-tenant demo. Identity comes from a spoofable X-Identity header, a placeholder rather than an authentication boundary, so it is meant for one trusted operator on their own machine, not a public multi-tenant deployment. The same trusted validator, engine, and persisted budget sit below it, so what it demonstrates about noise, clamping, and budget discipline is real; only the identity binding is a placeholder (see what these surfaces are, and are not).

For a local (non-Docker) setup, running the tests, and configuration knobs (budget size, epoch, identities), see SETUP.md.

Use it from Claude Desktop (MCP)

The gateway also runs as an MCP stdio server, so any MCP-capable agent can ask aggregate questions directly. The agent becomes the untrusted query author, and the same trusted validator, engine, and budget sit below it. The querying surface exposes only structured tools (count, sum, average, histogram, get_budget) whose argument schemas are generated from the dataset policy: published columns only, operations constrained by column kind. No ANTHROPIC_API_KEY is needed, because the client agent is the intelligence.

Install the console entry point once (pip install -e .), then register it in ~/Library/Application Support/Claude/claude_desktop_config.json (macOS):

{
  "mcpServers": {
    "noisegate": {
      "command": "/absolute/path/to/llm-dp/venv/bin/noisegate",
      "cwd": "/absolute/path/to/llm-dp",
      "env": { "DP_IDENTITY": "demo" }
    }
  }
}

By default it serves the bundled Adult dataset (run python scripts/fetch_data.py first). Point DP_POLICY at an owner-authored policy YAML to serve another dataset, BUDGET_DB_PATH at the persistent budget ledger (defaults to data/budget.db), and AUDIT_LOG_PATH at the audit trail (defaults to data/audit.log; set it empty to disable). Budget is keyed to identity, epoch, and dataset, so reconnecting never refreshes it.

Connect in four steps. No ANTHROPIC_API_KEY is needed at any point: the structured tools go straight to the trusted pipeline, and the connecting agent is the intelligence.

1. pip install -e . in a virtualenv. This installs the noisegate entry point (dp-mcp is kept as an alias).
2. Pick a dataset. The 20-row synthetic patients demo ships in-repo (nothing to fetch); the census dataset needs python scripts/fetch_data.py once.
3. Register the server with the JSON block above. For the patients demo, add "DP_POLICY": "/absolute/path/to/llm-dp/datasets/samples/patients.yaml" to the env block. Use an absolute path, because Claude Desktop does not apply cwd when resolving env vars, so a relative path will fail.
4. Restart Claude Desktop and confirm the noisegate tools appear: count, sum, average, histogram, get_budget.

Reproducible walkthrough: the patients demo, then the census at scale (click to expand)

A reproducible session: the patients demo. This replays the gallery's Attack 3 live, with a real agent as the attacker. Twenty synthetic patients (datasets/samples/patients.csv, no PHI), among them Alice, the only woman over 64. The policy publishes age, sex, and diagnosis, with ε 0.25 per query from a total budget of 5.0. Ask these questions in order. The exact values differ every run, since the noise is real and hiding it would misrepresent the system, but the shape of each answer reproduces. Every result carries remaining_epsilon, so you watch the budget deplete without ever asking for it:

1. "How much privacy budget do I have?" — get_budget reports remaining ε 5.00 of 5.00 at ε 0.25 per query: enough for 20 answers, ever, for this identity on this dataset.
2. "Break down the patients by diagnosis." — a histogram with five bins, each a noisy count ± 12.0 (95% interval). Against 20 rows the noise dominates — a bin can even come back negative. That is the honest price of privacy at n = 20; the census section below shows the same guarantee costing almost nothing at n = 32,561. The result reports remaining_epsilon: 4.75.
3. "Show the diagnosis breakdown for women aged 70 to 74." — a disguised row lookup (that cohort is exactly Alice). The filter guard rejects it as a tool error: the 4-year age window is under the required 5% of the declared 0–120 span. Rejected before execution — no budget spent.
4. "Compare the diagnosis breakdown of all women against women 64 and under." — the classic differencing pair: two broad population queries whose cohorts differ by exactly Alice. Both pass the guard and both are answered — and each bin carries ±12 of noise against a ±1 signal, so subtracting the two histograms points at an essentially random diagnosis. The budget ticks down 0.25 for each answer (remaining_epsilon: 4.25), which is the second defense: averaging the noise away would take ~256 repetitions of the pair, and the budget refuses after 10.
5. "Keep counting the patients until you are refused." — each count returns ~20 ± 12 and remaining_epsilon drops another 0.25. When the budget cannot cover one more query, the next call returns a first-class result of kind: "refusal" — never a noisier answer, never an error to retry.
6. Quit Claude Desktop, reopen it, ask for the budget again. Still exhausted. The ledger is persisted in BUDGET_DB_PATH keyed to identity, epoch, and dataset; reconnecting never refreshes it.

Then the census, for scale. Remove DP_POLICY from the env block (and restart Claude Desktop) to serve UCI Adult: n = 32,561 at ε 0.05 per query, and each dataset keeps its own budget ledger. The same mechanism that made a 20-row cohort unreadable barely dents a population-scale aggregate:

• "How many people are in the dataset?" — ≈ 32,500 ± 60 , an error of 0.2%.
• "What is the average hours worked per week?" — ≈ 40.5 ± 0.4.
• "Break the population down by education." — 16 bins, each ± 60 on counts in the thousands.

That asymmetry is the point: the noise is calibrated to hide any one individual, so it drowns questions about individuals and barely touches questions about populations. The membership-inference chart above sweeps the same trade across ε.

What the server guarantees. Every tool call is answered under differential privacy via the Laplace mechanism, or refused. There is no un-noised path and no way to retrieve an individual row. Concretely:

• Per-query cost. Each answered query spends a fixed per_query_epsilon (0.05 on the bundled policy) from a finite, persisted (ε_total, δ) budget keyed to your identity, epoch, and dataset. Every result, and get_budget, reports the remaining budget, so the agent can plan the few questions that matter rather than probe.
• Privacy unit. The unit protected is one row on the bundled dataset (one person = one row); for a policy that declares an entity unit, it is one entity contributing at most k rows, with the noise scaled to k , never rows silently treated as people.
• Refusal semantics. When the budget cannot cover a query the server returns a first-class result of kind "refusal", never a noisier answer and never an error to retry. A refusal is a policy outcome to respect; exhaustion is by design (see the budget-lifecycle discussion in DESIGN.md section 5.4).
• Audit trail. Below the trust boundary the service appends one record per answered release to AUDIT_LOG_PATH: identity, the validated query AST, the epsilon charged, and the remaining budget, with no raw data value and no answer figure. It records what actually ran and what it cost, so a deployment can reconstruct exactly how a budget was spent.

What these surfaces are not. Both the MCP server and the HTTP + Streamlit demo show the trust-boundary and budget discipline; neither is a hardened production service. Each serves a single local principal who owns the machine (the MCP server binds one identity at launch, the HTTP demo reads a spoofable X-Identity header), and that principal can rotate the epoch or delete the budget file to start fresh. So the "an external party burns budget and cannot get it back" property is demonstrated in mechanism, not enforced against a remote adversary ; that needs the verified-identity (OAuth) and multi-tenant split still on the roadmap. The noise sampler also has a known floating-point side channel (Mironov 2012; DESIGN.md section 5.5), out of scope here. Do not point either surface at real regulated data (PHI, or PII under a compliance regime) or treat it as a substitute for a vetted DP library where an untrusted party can authenticate. The realistic scope is one-row-per-subject demo datasets and a local, trusted operator.

The bundled dataset is the public UCI Adult / Census Income dataset, the standard benchmark in the differential privacy literature, so the attack-vs-ε results here are directly comparable to published work. It contains no private data to begin with, so the repository is safe to publish; the privacy machinery is demonstrated as if the records were sensitive.

---

What this project is, and is not

It is a demonstration that an untrusted natural-language layer can sit in front of sensitive data without compromising a formal privacy guarantee, with the guarantee backed by working attacks rather than assertions.

It is not a production privacy product, a substitute for a vetted DP library in a real deployment, or a claim that LLM-generated queries are safe to run unguarded. They are not, which is exactly why the trusted layer exists.

---

Where this sits among the tools a privacy engineer would actually reach for, and what it deliberately does differently.

DP libraries. OpenDP, Google's differential-privacy libraries, IBM's diffprivlib, and Tumult Analytics (which powered U.S. Census Bureau, Wikimedia, and IRS releases before its team joined LinkedIn in 2025 and the library moved into the OpenDP organization) are vetted implementations of the mechanisms themselves, written for a trusted analyst to call from their own code. This project hand-rolls its mechanism instead, so that every line of the privacy-critical path stays small, legible, and testable in one repo — and then cross-checks it against OpenDP, precisely because a hand-rolled mechanism means nothing without an external referee. For a real deployment the advice above stands: use a vetted library.

DP query engines. PINQ (McSherry, 2009) established the architecture this project inherits: an untrusted analyst issues queries through a trusted layer that guarantees differential privacy no matter what is asked. Its production descendants answer SQL under DP: smartnoise-sql, Uber's archived elastic-sensitivity rewriter, and Wilson et al.'s differentially private SQL, shipped today as BigQuery differential privacy. Two deliberate differences here. Those systems accept SQL and analyze or rewrite it; this gateway never accepts SQL at all — the agent submits a small constrained AST, because a query language small enough to enumerate is a trust boundary small enough to test exhaustively. And every one of them puts a human analyst at the keyboard; here the query author is an autonomous agent, and the threat model says so.

Natural-language database interfaces. Text-to-SQL is a large and successful field, but privacy is not its subject: the generated SQL runs with whatever authority the database grants. Where safety is addressed, as in SafeNLIDB (AAAI 2026), it is alignment-based — train the model to reason about what is safe to answer — so the guarantee lives inside the model. This project takes the complementary position: assume the model cannot be made safe, and enforce the guarantee below it with mathematics that holds even when the model is adversarial.

Agent and MCP privacy tooling. The emerging MCP-security stack addresses real risks — tool poisoning, prompt injection, exfiltration — with policy gateways, PII redaction and tokenization, and data-minimization patterns. Those controls govern what an agent can see and send. None of them bounds what a sequence of individually harmless answers cumulatively reveals, and that is the channel the attack gallery exploits: a differencing attack leaks a salary through two aggregates a redaction proxy would happily pass. Differential privacy bounds exactly that channel, which makes it complementary to the rest of that stack rather than competition.

DP near LLMs, different problems. DP-SGD (e.g. Opacus) protects training examples inside model weights; DP in-context learning, DP synthetic text, and DP-RAG protect prompts and retrieval corpora; local DP (Apple's and Google's telemetry) noises data on-device before it is ever collected. This project is the older, central-model setting those all descend from: a trusted curator, an untrusted questioner, and a budget between them.

The claim, stated carefully. There is no new differential-privacy mathematics in this repository. The contribution is the combination, and the evidence: an open, attack-validated artifact that puts agentic natural-language access in front of a DP aggregate backend under an explicit untrusted-model threat model. As far as I can find, no other open MCP server enforces differential privacy with a persisted budget beneath an untrusted agent. If you know of one, open an issue — I would genuinely like to compare notes.

---

.
├── README.md            ← you are here
├── DESIGN.md            ← architecture, threat model, the math, the budget design
├── SETUP.md             ← local setup, running the tests, configuration knobs
├── engine/              ← DP engine: mechanism, RNG, sensitivity, budget accountant + store
├── validation/          ← the trust boundary: policy checks, guards, and the repair loop
├── compiler/            ← the untrusted LLM NL→AST compiler
├── attacks/             ← the attack gallery: differencing, membership ε-sweep, patients singling-out
├── service/             ← AnalyticsService + audit log: the transport-agnostic core every surface shares
├── mcp_server/          ← the stdio MCP server: tool schemas generated from the dataset policy
├── datasets/            ← owner-authored dataset policies, ingestion, and the bundled patients demo
├── identity/            ← the identity seam the budget is keyed to
├── api/                 ← FastAPI service
├── ui/                  ← Streamlit front-end with the live budget meter
├── benchmarks/          ← manual latency benchmarks (run by hand, never by pytest)
├── docs/                ← the demo GIF and the session chart assets
├── scripts/             ← fetch_data.py (pulls UCI Adult into data/, never committed) + the demo GIF renderer
└── tests/               ← unit, trust-boundary, privacy-guarantee, and attack regression tests (DESIGN.md section 8)

This project is where I work through the design of privacy-preserving data systems and least-privilege data access in public: the trust boundary, the budget accounting, and the attack demos are the parts I'd defend in a design review.