Give an AI agent query access to sensitive data, with a mathematical guarantee that no individual's record can leak — even if the agent is wrong, manipulated, or adversarial.

A recorded Claude Desktop session (replies trimmed; the chart cards are the session's own). An AI agent breaks 20 patients down by diagnosis, and the ±12 noise swamps every bin. Reminded that it cannot turn the noise off, it drains a three-answer budget until the gate returns a refusal instead of a quieter answer. On the 32,561-row census, a too-narrow slice is rejected at the trust boundary, while a full education breakdown comes back clean at scale. The refusal and the rejection are the live gateway's real enforcement, reproduced by python scripts/render_demo_gif.py. Try it from Use it from Claude Desktop.

At a glance

• Working attacks, not claims. Three classic privacy attacks (differencing, membership inference, and singling-out by re-identification) run against the system's own engine. Each is shown succeeding with privacy off, defeated with privacy on, and regression-tested in CI so the defense cannot quietly rot.
• Independently verified math. The hand-rolled noise mechanism matches OpenDP, the industry reference implementation, in all 35 noise-scale checks to within 1e-9, and 500,000-sample distribution tests pass with a positive control proving the test can fail.
• 3× the utility from better accounting. Hybrid zCDP composition admits 308 queries where naive budget accounting stops at 100: the same privacy guarantee, three times the questions.
• Built for AI agents. Runs as an MCP server for Claude Desktop. The connecting agent is untrusted by design, and every privacy property is enforced below it.
• A small, deterministic trust boundary. The LLM proposes queries; a boring, fully tested validation layer disposes. Nothing the model outputs can widen its own authority.
• Stack: Python · DuckDB · FastAPI · Streamlit · MCP SDK · Docker · GitHub Actions, with a 250+ test suite in CI.

Five minutes to evaluate: watch the GIF, then skim the attack gallery. Deep review: DESIGN.md.

---

You ask questions about a sensitive dataset in plain English. An LLM compiles each question into a small, constrained query; a differential-privacy engine executes it under a tracked privacy budget and returns a deliberately noisy answer with a stated confidence interval. The name fits. Like its audio namesake, the gateway keeps every signal below a set threshold under the noise floor: any one individual's contribution is drowned out, while population-scale signals pass through nearly untouched. No individual record can be reconstructed from the answers, no matter what is asked or how cleverly it is phrased.

The interesting part isn't that an LLM can write queries. It's that the privacy guarantee does not depend on the LLM being trustworthy. The model is a convenience that proposes a query; it enforces nothing. Every privacy property is enforced downstream, by components that would behave the same way if a human typed the query by hand. This is the trust-boundary discipline you'd apply to any untrusted input in a production system, applied here to an AI agent.

---

The headline: an attack gallery

Anyone can claim privacy. This repository ships the exploits that would break the claim, runs them against its own engine, and pins the outcomes in CI. The fastest way to understand what the gateway guarantees is to watch it defeat three classic attacks that break naive "query a database" systems.

Attack 1 — The differencing attack

A differencing attack isolates one person by asking two aggregate questions that differ by exactly that person.

Query A: "Total income of all 100 people in department X."        → $7,240,000
Query B: "Total income of all people in department X except Alice." → $7,135,000
Attacker computes: A − B = $105,000  ← Alice's exact salary, leaked.

Both queries are "just aggregates." Neither names a single row. Yet together they expose an individual. The gallery (attacks/differencing.py) shows this attack succeeding with privacy disabled: the target's private value is recovered exactly. (The salary sketch above is illustrative; on the real UCI Adult data, "Alice" is the unique holder of her group's maximum capital gain.) It then shows the same attack defeated once DP is on: the calibrated noise on each answer makes the subtraction useless, and the budget accountant charges for the information released across both queries rather than treating them as independent.

Attack 2 — Membership inference

A membership-inference attack determines whether a specific individual is in the dataset at all. For many datasets (a medical study, a list of defaulters), that fact is itself sensitive. An attacker with only query access tries to decide: "is this exact person in the data?"

The gallery runs this attack across a sweep of privacy budgets (ε — the dial that trades answer accuracy for privacy) and plots the result:

An optimal (Neyman–Pearson) attacker deciding one person's membership from a single noisy COUNT, run against the real engine across ε. Empirical success (blue, 95% Wilson intervals) hugs the analytic Laplace curve and stays below the worst-case DP ceiling (dashed); it collapses from certainty (DP off) toward the 0.5 coin-flip as ε shrinks. The green utility curve (right axis) shows an aggregate query's relative error over the same sweep, barely dented where the attack is defeated. Generated by python -m attacks.membership (10,000 trials per ε).

As ε shrinks (stronger privacy), the attacker's success collapses toward a coin flip. The utility overlay shows the price paid: the same noise that defeats the one-person attack barely moves a population-scale aggregate. Privacy is not free, and the chart shows exactly what you trade for it. The sweep spans ε = 8 down to 0.5 to cover the whole range; the bundled deployment charges ε = 0.05 per query, off the left edge of this chart, where the single-query attack is already indistinguishable from chance.

Attack 3 — Singling out by re-identification

Latanya Sweeney showed in 2002 that ZIP code + birth date + sex uniquely identify roughly 87% of Americans. You do not need someone's name to find them in a dataset; a few innocuous attributes will do. The gallery's third attack (attacks/patients_alice.py) reproduces that structure on 20 synthetic patients: sex and age alone narrow the cohort to exactly one person, Alice, the only woman over 64.

With privacy off, subtracting two perfectly ordinary demographic histograms recovers Alice's diagnosis exactly. No outlier value is required; being re-identifiable is enough, which makes this the strongest attack in the gallery. With privacy on, three independent defenses end it: the filter guard rejects the obvious narrow query outright (spending nothing); calibrated noise drowns the two-query subtraction (signal-to-noise ≈ 0.13, so the recovered "diagnosis" is essentially a random draw); and the budget refuses the averaging escalation long before it could work (~256 repetitions needed, 10 affordable). This is the attack the demo GIF at the top of this page, and the Claude Desktop walkthrough, replay live. Outcomes are regression-tested in tests/test_attack_patients.py.

These experiments, run against the system's own engine, are the core evidence that the guarantee is real and understood, not imported from a library and taken on faith.

Cross-checked against OpenDP

The attacks validate the guarantee against our own math. To guard against a self-consistent-but-wrong implementation, the mechanism is also cross-checked against OpenDP as an independent reference (attacks/crosscheck_opendp.py, opendp 0.15.1).

In short: the hand-rolled mechanism agrees with the industry reference to nine decimal places, and a version that is intentionally mis-calibrated fails the same tests, so the agreement means something. The details:

• Scale agreement. For COUNT, clamped SUM over every published numeric column, and a histogram, at five ε values (35 comparisons): our sensitivity Δf = max(|L|, |U|) equals OpenDP's clamp-then-sum stability bound exactly, and our noise scale Δf/ε matches the smallest scale OpenDP certifies as ε-DP to within 1e-9, in every case.
• Distribution agreement. 500,000 noise samples from the production sampler vs. 500,000 from OpenDP's Laplace measurement at the same scale pass a two-sample KS test (D = 0.0014, p = 0.70 at α = 0.001). A positive control proves the test has power: the same sampler, intentionally mis-scaled by 5%, is rejected at p ≈ 4×10⁻²².

OpenDP is the reference, never the implementation: it lives in the dev extras only, and the running system does not import it. The scale agreement is regression-tested in tests/test_crosscheck_opendp.py (skipped without OpenDP locally; CI installs it and sets REQUIRE_OPENDP=1 so the check can never silently skip there).

Reproduce it

The attacks run in-process against the real DP engine. No API key and no server are needed, since they build hand-written ASTs rather than going through the LLM compiler:

python -m attacks.patients_alice # Attack 3: re-identify Alice and recover her diagnosis # with DP off; watch guard + noise + budget # defeat the same attack (no data fetch needed)

python scripts/fetch_data.py # fetch UCI Adult into data/ (gitignored; ~4 MB)

python -m attacks.differencing # Attack 1: recover Alice exactly with DP off, then watch # noise + budget defeat the same subtraction python -m attacks.membership # Attack 2: sweep ε and regenerate the chart above

python -m attacks.crosscheck_opendp # cross-check vs OpenDP (needs the dev # extras: pip install -e ".[dev]")

The outcomes are themselves regression-tested in tests/test_attack_differencing.py, tests/test_attack_membership.py, and tests/test_attack_patients.py, so a change that quietly weakened the guarantee would fail CI rather than just producing a prettier chart.

---

How it works

Three stages. Only the first is untrusted, and that asymmetry is the whole idea.

  Natural language question
            │
            ▼
  ┌───────────────────┐
  │   LLM compiler    │   Emits a CONSTRAINED QUERY AST (not free SQL),
  │   (UNTRUSTED)     │   via structured/schema-forced output.
  └───────────────────┘
            │  query AST
            ▼
  ┌───────────────────┐
  │  Validation /     │   Policy check: allowed columns only, aggregates
  │  guardrail layer  │   only, declared value ranges, group-cardinality
  │   (TRUSTED)       │   caps. Invalid → repair loop or refusal.
  └───────────────────┘
            │  validated AST
            ▼
  ┌───────────────────┐
  │  Privacy engine   │   Clamp to declared range (bounds sensitivity),
  │   (TRUSTED)       │   add calibrated Laplace noise, decrement budget.
  └───────────────────┘
            │
            ▼
  Noisy answer + confidence interval + remaining budget

The most important design choice is the box labeled UNTRUSTED. Everything above the validation layer is treated as hostile. Everything below it is the trusted computing base that enforces the guarantee. The LLM compiles to a constrained AST rather than arbitrary SQL because that constraint is what bounds query sensitivity, and bounded sensitivity is what makes the noise calibration sound. Free-form SQL would reopen exactly the attack surface this project closes.

See DESIGN.md for the full architecture, the threat model, the sensitivity and noise math, and the budget-accounting design.

---

The privacy model, stated honestly

The plain version: every answer is noised, an exact ledger of cumulative disclosure is kept per identity, and the gateway refuses to answer before that ledger can exceed the guarantee it advertises. The rest of this section spells out the ledger's math. It matters to privacy engineers, and everyone else can skim it.

Each answer is produced by the pure-ε Laplace mechanism, with calibrated noise and a stated confidence interval. Cumulative privacy loss across queries is tracked by a hybrid accountant that maintains two simultaneously valid bounds: the pure sum of ε, and a zero-concentrated DP (zCDP) total in which each ε-DP release costs ρ = ε²/2 and ρ adds across queries. It refuses further queries only when both are exhausted. Light usage keeps the conservative pure-ε capacity; bulk usage gains zCDP's √k composition, which at small per-query ε answers several times more queries at the same total budget.

At the default ε_q = 0.05 against the same (ε = 5, δ = 10⁻⁶) budget, plain sum-of-ε accounting refuses after 100 queries; the hybrid answers 308 before refusing, the same guarantee for three times the questions. The gain grows as the per-query ε shrinks, and above the break-even it degrades cleanly: the hybrid falls back to the pure-ε capacity rather than dropping below it. Regenerate this, the break-even, and the δ-sensitivity views with python benchmarks/composition_comparison.py.

The cost is worth stating just as clearly: the headline guarantee is (ε, δ)-differential privacy, which is strictly weaker than pure ε-DP. With probability at most δ the ε bound is allowed to fail. δ is not extra noise; it is failure mass. So it is derived from the dataset rather than hard-coded: δ = min(10⁻⁶, 1/(30·n)), at least 30× below 1/n for any served dataset (the same proportional margin the 2020 US Census production system chose), which for the bundled Adult set (n = 32,561) gives 10⁻⁶. A user whose pure-ε ledger never crosses the budget also keeps the full pure ε-DP guarantee. DESIGN.md sections 3 and 5.4 give the math, the break-even, and the trade in full.

---

Quickstart — three ways in

1. No API key, no setup — run the attacks. The gallery above executes in-process against the real engine; python -m attacks.patients_alice needs nothing fetched at all.

2. No API key — connect an AI agent. The gateway runs as an MCP server for Claude Desktop; the walkthrough is below.

3. The full natural-language UI (a local single-tenant demo) — an API key for the untrusted NL→query compiler is the only extra requirement:

export ANTHROPIC_API_KEY=... # used only by the untrusted NL→query compiler docker compose up # brings up the engine, API, and UI

open http://localhost:8501

This HTTP + Streamlit surface is a local, single-tenant demo. Identity comes from a spoofable X-Identity header, a placeholder rather than an authentication boundary, so it is meant for one trusted operator on their own machine, not a public multi-tenant deployment. The same trusted validator, engine, and persisted budget sit below it, so what it demonstrates about noise, clamping, and budget discipline is real; only the identity binding is a placeholder (see what these surfaces are, and are not).

For a local (non-Docker) setup, running the tests, and configuration knobs (budget size, epoch, identities), see SETUP.md.

Use it from Claude Desktop (MCP)

The gateway also runs as an MCP stdio server, so any MCP-capable agent can ask aggregate questions directly. The agent becomes the untrusted query author, and the same trusted validator, engine, and budget sit below it. The querying surface exposes only structured tools (count, sum, average, histogram, get_budget) whose argument schemas are generated from the dataset policy: published columns only, operations constrained by column kind. No ANTHROPIC_API_KEY is needed, because the client agent is the intelligence.

Install the console entry point once (pip install -e .), then register it in ~/Library/Application Support/Claude/claude_desktop_config.json (macOS):

{ "mcpServers": { "noisegate": { "command": "/absolute/path/to/llm-dp/venv/bin/noisegate", "cwd": "/absolute/path/to/llm-dp", "env": { "DP_IDENTITY": "demo" } } } }

By default it serves the bundled Adult dataset (run python scripts/fetch_data.py first). Point DP_POLICY at an owner-authored policy YAML to serve another dataset, BUDGET_DB_PATH at the persistent budget ledger (defaults to data/budget.db), and AUDIT_LOG_PATH at the audit trail (defaults to data/audit.log; set it empty to disable). Budget is keyed to identity, epoch, and dataset, so reconnecting never refreshes it.

Connect in four steps. No ANTHROPIC_API_KEY is needed at any point: the structured tools go straight to the trusted pipeline, and the connecting agent is the intelligence.

1. pip install -e . in a virtualenv. This installs the noisegate entry point (dp-mcp is kept as an alias).
2. Pick a dataset. The 20-row synthetic patients demo ships in-repo (nothing to fetch); the census dataset needs python scripts/fetch_data.py once.
3. Register the server with the JSON block above. For the patients demo, add "DP_POLICY": "/absolute/path/to/llm-dp/datasets/samples/patients.yaml" to the env block. Use an absolute path, because Claude Desktop does not apply cwd when resolving env vars, so a relative path will fail.
4. Restart Claude Desktop and confirm the noisegate tools appear: count, sum, average, histogram, get_budget.

Reproducible walkthrough: the patients demo, then the census at scale (click to expand)

A reproducible session: the patients demo. This replays the gallery's Attack 3 live, with a real agent as the attacker. Twenty synthetic patients (datasets/samples/patients.csv, no PHI), among them Alice, the only woman over 64. The policy publishes age, sex, and diagnosis, with ε 0.25 per query from a total budget of 5.0. Ask these questions in order. The exact values differ every run, since the noise is real and hiding it would misrepresent the system, but the shape of each answer reproduces. Every result carries remaining_epsilon, so you watch the budget deplete without ever asking for it:

1. "How much privacy budget do I have?" — get_budget reports remaining ε 5.00 of 5.00 at ε 0.25 per query: enough for 20 answers, ever, for this identity on this dataset.
2. "Break down the patients by diagnosis." — a histogram with five bins, each a noisy count ± 12.0 (95% interval). Against 20 rows the noise dominates — a bin can even come back negative. That is the honest price of privacy at n = 20; the census section below shows the same guarantee costing almost nothing at n = 32,561. The result reports remaining_epsilon: 4.75.
3. "Show the diagnosis breakdown for women aged 70 to 74." — a disguised row lookup (that cohort is exactly Alice). The filter guard rejects it as a tool error: the 4-year age window is under the required 5% of the declared 0–120 span. Rejected before execution — no budget spent.
4. "Compare the diagnosis breakdown of all women against women 64 and under." — the classic differencing pair: two broad population queries whose cohorts differ by exactly Alice. Both pass the guard and both are answered — and each bin carries ±12 of noise against a ±1 signal, so subtracting the two histograms points at an essentially random diagnosis. The budget ticks down 0.25 for each answer (remaining_epsilon: 4.25), which is the second defense: averaging the noise away would take ~256 repetitions of the pair, and the budget refuses after 10.
5. "Keep counting the patients until you are refused." — each count returns ~20 ± 12 and remaining_epsilon drops another 0.25. When the budget cannot cover one more query, the next call returns a first-class result of kind: "refusal" — never a noisier answer, never an error to retry.
6. Quit Claude Desktop, reopen it, ask for the budget again. Still exhausted. The ledger is persisted in BUDGET_DB_PATH keyed to identity, epoch, and dataset; reconnecting never refreshes it.

Then the census, for scale. Remove DP_POLICY from the env block (and restart Claude Desktop) to serve UCI Adult: n = 32,561 at ε 0.05 per query, and each dataset keeps its own budget ledger. The same mechanism that made a 20-row cohort unreadable barely dents a population-scale aggregate:

• "How many people are in the dataset?" — ≈ 32,500 ± 60, an error of 0.2%.
• "What is the average hours worked per week?" — ≈ 40.5 ± 0.4.
• "Break the population down by education." — 16 bins, each ± 60 on counts in the thousands.

That asymmetry is the point: the noise is calibrated to hide any one individual, so it drowns questions about individuals and barely touches questions about populations. The membership-inference chart above sweeps the same trade across ε.

What the server guarantees. Every tool call is answered under differential privacy via the Laplace mechanism, or refused. There is no un-noised path and no way to retrieve an individual row. Concretely:

• Per-query cost. Each answered query spends a fixed per_query_epsilon (0.05 on the bundled policy) from a finite, persisted (ε_total, δ) budget keyed to your identity, epoch, and dataset. Every result, and get_budget, reports the remaining budget, so the agent can plan the few questions that matter rather than probe.
• Privacy unit. The unit protected is one row on the bundled dataset (one person = one row); for a policy that declares an entity unit, it is one entity contributing at most k rows, with the noise scaled to k, never rows silently treated as people.
• Refusal semantics. When the budget cannot cover a query the server returns a first-class result of kind "refusal", never a noisier answer and never an error to retry. A refusal is a policy outcome to respect; exhaustion is by design (see the budget-lifecycle discussion in DESIGN.md section 5.4).
• Audit trail. Below the trust boundary the service appends one record per answered release to AUDIT_LOG_PATH: identity, the validated query AST, the epsilon charged, and the remaining budget, with no raw data value and no answer figure. It records what actually ran and what it cost, so a deployment can reconstruct exactly how a budget was spent.

What these surfaces are not. Both the MCP server and the HTTP + Streamlit demo show the trust-boundary and budget discipline; neither is a hardened production service. Each serves a single local principal who owns the machine (the MCP server binds one identity at launch, the HTTP demo reads a spoofable X-Identity header), and that principal can rotate the epoch or delete the budget file to start fresh. So the "an external party burns budget and cannot get it back" property is demonstrated in mechanism, not enforced against a remote adversary; that needs the verified-identity (OAuth) and multi-tenant split still on the roadmap. The noise sampler also has a known floating-point side channel (Mironov 2012; DESIGN.md section 5.5), out of scope here. Do not point either surface at real regulated data (PHI, or PII under a compliance regime) or treat it as a substitute for a vetted DP library where an untrusted party can authenticate. The realistic scope is one-row-per-subject demo datasets and a local, trusted operator.

The bundled dataset is the public UCI Adult / Census Income dataset, the standard benchmark in the differential privacy literature, so the attack-vs-ε results here are directly comparable to published work. It contains no private data to begin with, so the repository is safe to publish; the privacy machinery is demonstrated as if the records were sensitive.

---

What this project is, and is not

It is a demonstration that an untrusted natural-language layer can sit in front of sensitive data without compromising a formal privacy guarantee, with the guarantee backed by working attacks rather than assertions.

It is not a production privacy product, a substitute for a vetted DP library in a real deployment, or a claim that LLM-generated queries are safe to run unguarded. They are not, which is exactly why the trusted layer exists.

---

Related work

Where this sits among the tools a privacy engineer would actually reach for, and what it deliberately does differently.

DP libraries. OpenDP, Google's differential-privacy libraries, IBM's diffprivlib, and Tumult Analytics (which powered U.S. Census Bureau, Wikimedia, and IRS releases before its team joined LinkedIn in 2025 and the library moved into the OpenDP organization) are vetted implementations of the mechanisms themselves, written for a trusted analyst to call from their own code. This project hand-rolls its mechanism instead, so that every line of the privacy-critical path stays small, legible, and testable in one repo — and then cross-checks it against OpenDP, precisely because a hand-rolled mechanism means nothing without an external referee. For a real deployment the advice above stands: use a vetted library.

DP query engines. PINQ (McSherry, 2009) established the architecture this project inherits: an untrusted analyst issues queries through a trusted layer that guarantees differential privacy no matter what is asked. Its production descendants answer SQL under DP: smartnoise-sql, Uber's archived elastic-sensitivity rewriter, and Wilson et al.'s differentially private SQL, shipped today as BigQuery differential privacy. Two deliberate differences here. Those systems accept SQL and analyze or rewrite it; this gateway never accepts SQL at all — the agent submits a small constrained AST, because a query language small enough to enumerate is a trust boundary small enough to test exhaustively. And every one of them puts a human analyst at the keyboard; here the query author is an autonomous agent, and the threat model says so.

Natural-language database interfaces. Text-to-SQL is a large and successful field, but privacy is not its subject: the generated SQL runs with whatever authority the database grants. Where safety is addressed, as in SafeNLIDB (AAAI 2026), it is alignment-based — train the model to reason about what is safe to answer — so the guarantee lives inside the model. This project takes the complementary position: assume the model cannot be made safe, and enforce the guarantee below it with mathematics that holds even when the model is adversarial.

Agent and MCP privacy tooling. The emerging MCP-security stack addresses real risks — tool poisoning, prompt injection, exfiltration — with policy gateways, PII redaction and tokenization, and data-minimization patterns. Those controls govern what an agent can see and send. None of them bounds what a sequence of individually harmless answers cumulatively reveals, and that is the channel the attack gallery exploits: a differencing attack leaks a salary through two aggregates a redaction proxy would happily pass. Differential privacy bounds exactly that channel, which makes it complementary to the rest of that stack rather than competition.

DP near LLMs, different problems. DP-SGD (e.g. Opacus) protects training examples inside model weights; DP in-context learning, DP synthetic text, and DP-RAG protect prompts and retrieval corpora; local DP (Apple's and Google's telemetry) noises data on-device before it is ever collected. This project is the older, central-model setting those all descend from: a trusted curator, an untrusted questioner, and a budget between them.

The claim, stated carefully. There is no new differential-privacy mathematics in this repository. The contribution is the combination, and the evidence: an open, attack-validated artifact that puts agentic natural-language access in front of a DP aggregate backend under an explicit untrusted-model threat model. As far as I can find, no other open MCP server enforces differential privacy with a persisted budget beneath an untrusted agent. If you know of one, open an issue — I would genuinely like to compare notes.

---

Repository layout

.
├── README.md            ← you are here
├── DESIGN.md            ← architecture, threat model, the math, the budget design
├── SETUP.md             ← local setup, running the tests, configuration knobs
├── engine/              ← DP engine: mechanism, RNG, sensitivity, budget accountant + store
├── validation/          ← the trust boundary: policy checks, guards, and the repair loop
├── compiler/            ← the untrusted LLM NL→AST compiler
├── attacks/             ← the attack gallery: differencing, membership ε-sweep, patients singling-out
├── service/             ← AnalyticsService + audit log: the transport-agnostic core every surface shares
├── mcp_server/          ← the stdio MCP server: tool schemas generated from the dataset policy
├── datasets/            ← owner-authored dataset policies, ingestion, and the bundled patients demo
├── identity/            ← the identity seam the budget is keyed to
├── api/                 ← FastAPI service
├── ui/                  ← Streamlit front-end with the live budget meter
├── benchmarks/          ← manual latency benchmarks (run by hand, never by pytest)
├── docs/                ← the demo GIF and the session chart assets
├── scripts/             ← fetch_data.py (pulls UCI Adult into data/, never committed) + the demo GIF renderer
└── tests/               ← unit, trust-boundary, privacy-guarantee, and attack regression tests (DESIGN.md section 8)

Author's note

This project is where I work through the design of privacy-preserving data systems and least-privilege data access in public: the trust boundary, the budget accounting, and the attack demos are the parts I'd defend in a design review.