Research on whether AI systems for benefits and public services do what the law says they do.
I test AI claims against ground truth that can be checked: benefit formulas defined in
statute, synthetic data with exact labels, and payment records with coded errors. The
views represented here are my own.
What I'm interested in
Small, domain-specific models
Models trained for one job in one domain, small enough to run on the device where
the work happens. Most of the publications here are this: an 11-million-parameter
redaction model, a two-million-parameter speech model, tiny networks that compute
a single benefit rule exactly.
AI alignment
Whether a system keeps doing what it was meant to do as it gets more capable —
and whether the humans around it keep making the judgment calls instead of
deferring to it. In benefits work the intent is unusually concrete: it is written
down in statute, so misalignment can be measured rather than argued about.
AI safety
Evaluation and oversight for systems that decide things about real people:
gates set before runs, guardrails that live outside the agent, and evidence a
reviewer can check after the fact. Safety work that only lives inside the model
is a request, not a control.
Research areas
Exact and verifiable models
1 publication
Benefit formulas are defined by law, so a model that computes one can be checked exactly — to the cent, everywhere, not on average. I write neural network weights directly from those formulas and compare them against trained models, which fail in ways average accuracy hides.
Sensitive text should not have to leave the device to be useful. I train small, specialised models — phi-redact is 11 million parameters — that run in a browser tab or a phone app, paired with deterministic rules for the identifiers that have checksums.
Recovering SNAP's effective formula from real payment-error data (experiments 116–122). So far: the formula layer checks out against the statute, and the payment error lives in the fact layer — earnings above all. Write-up in progress.
An independent test of the claim that AI models can write, read, and edit neural network weights directly, run against US benefit formulas where the correct answer is defined by law. Written weights were exact. Trained weights failed in ways average accuracy hides. And one step of the law cannot be represented at all.
› Seven benefit rules compiled into network weights matched their statutory formulas with a worst error under four trillionths of a dollar.
› A trained SSI network changed a couple's payment by up to $25.15 depending on which spouse earned the income — a distinction the law does not make — in 98.9% of equivalent households.
› Total API cost for the whole program: $2.48. Every experiment, including the failures, is public.
I trained my first model to redact PII and PHI on-device: an 11-million-parameter token classifier that fits in 11.3 MB. Nine versions of synthetic data, strict span scoring, and a failure ledger got it to 0.974 full-value recall on my golden set, with the misses still named in the post.
› 11.13 million parameters, distilled from a 66-million-parameter teacher. The shipped int8 build is 11.3 MB and runs in the browser.
› Version 9 was a data revision, not a new architecture: golden full-value recall 0.958 to 0.974, clean-case false positives 0.44 to 0.20.
› The failure ledger is published with the results, including the types the model scored 0.000 on for six versions.
Protocols and pass/fail gates are written before the experiments run, so a result cannot quietly redefine what success was supposed to be.
Failures published
Failed versions, corrected results, and the cases a model gets wrong ship next to the successes, in the same document.
Numbers re-checked
Every decisive number is re-produced before publication. If a re-check disagrees with an earlier figure, the corrected figure is the one that ships.
More writing on AI and government
The publications above are the long-form research. Shorter pieces on evaluation,
agentic systems, and AI delivery in government live on the blog, and the advisory page
collects the applied work.