Contents

• 1. Why build a redaction model at all
• 2. The synthetic-data bet
• 3. What the model actually is
• 4. Why token accuracy lies, and what I score instead
• 5. Versions 1 through 7: reaching a real model
• 6. The failure ledger
• 7. Versions 8 and 9: the rules turn, then the data turn
• 8. The size campaign
• 9. Version table
• 10. What reuses the model, and what is next
• Reproducibility
• Footnotes

Nine Versions of a Redaction Model

Training a small model to catch what regex cannot, and learning from every version that missed. The views represented here are my own. Last updated: October 2026

1. Why build a redaction model at all

Redaction has an awkward shape. The easy half is mechanical: a credit card number either passes the Luhn checksum or it does not, and an SSN with area 000 is not an SSN. The hard half is judgment. A name in a sentence, an address written as prose, a provider mentioned in passing. Regex gets the first half and is blind to the second. A large language model gets both halves, at the cost of sending the exact text you are trying to protect to someone else’s server, which defeats the point for a lot of real documents.

The gap I wanted to fill was a model small enough to run inside a browser tab or a phone app, trained for this one job, paired with deterministic rules for the identifiers that have checksums. The comparison point I set at the start, in the design doc dated 2026-09-14, was Desert Ant Labs’ Redact: 23 million parameters and a reported 88.8% recall. The target was smaller than that and better on recall, English only.³

Two constraints were fixed before any training. I would never train on real patient data; real clinical datasets, if used at all, would be evaluation only under their own terms. And I would never claim certified de-identification. Benchmark numbers describe the data they were measured on. HIPAA’s Expert Determination and Safe Harbor methods are legal processes, not scores, and a model card that blurs that line is lying to the people most likely to rely on it.

This was my first time training a model. That mattered less for the modeling than for the measurement habits. I had to learn, in order, why my first metric was meaningless, why my data generator was the real product, and why a version number mostly tracks changes to data rather than changes to architecture.

2. The synthetic-data bet

A redaction model needs labeled spans: this character range is a patient name, that one is a medical record number. Labeling real clinical text by hand is slow, and using real patient data for training was off the table by my own rule. So the training data is synthetic, generated by a program I control, and the labels are exact by construction. The generator renders a note from templates and value samplers, and it records the character offsets of every planted entity as it writes. It asserts, on every run, that each span matches the text exactly and that no two spans overlap. A 20,000-note run produced 303,102 entities at about 15 per note.⁴

Synthetic data has an obvious risk, which Section 6 returns to: the model learns your templates, and the evaluation can end up grading the generator instead of the model. I accepted that risk with two mitigations. First, the generator carries a hand-built edge-case matrix: all-caps and all-lowercase notes, last-first and initialed names, apostrophe and non-ASCII names, international and dot-separated phone numbers, PO boxes, multiline and pipe-delimited records, ages 90 to 105, and near-miss negatives such as Luhn-invalid card-shaped strings and disease names that look like person names.⁵ Second, the model also trains and evaluates on public corpora I did not write (Wikipedia-style name corpora, student essays, a structured PII benchmark), where the phrasing is not mine.

The label schema started at 28 entity types in the September scaffold and reached 32 by v9, after TAX_ID and GOVERNMENT_ID were added in v8. With BIOES tagging (begin, inside, outside, end, single), 32 types produce 129 output tags. That count matters for the class-imbalance problem in Section 4.

3. What the model actually is

The September scaffold and the shipped model are different architectures, and the early project documents describe only the scaffold. The scaffold was a RoBERTa-style encoder trained from scratch with a custom BPE tokenizer (a smoke run used an 8,000-token vocabulary), trained with class weights and a distillation scaffold that was never completed in that form.³⁶

What shipped, starting with v7, is a BERT-mini student: 4 layers, hidden size 256, 4 attention heads, 11.13 million parameters, using the bert-base-uncased WordPiece vocabulary of 30,522 tokens. It was knowledge-distilled from a DistilBERT teacher of 66 million parameters that was fine-tuned on the same task first. The student trained for 60 epochs on an A100, with the distillation loss weighted 0.7 against 0.3 cross-entropy.¹ I kept the part of the original design that turned out to be load-bearing (small English vocabulary instead of a multilingual one, which is where a 23-million-parameter multilingual model spends much of its budget) and dropped the from-scratch training, because distillation from a competent teacher reached the accuracy target and the scaffold did not.

Around the model sits the deterministic layer, which does half the system’s work. It validates credit cards with Luhn, SSNs against area rules, routing numbers with the ABA checksum, IBANs with ISO-13616, and matches phones, emails, URLs, IPs, dates, and other structured shapes with patterns. Where the rules layer and the model disagree on a structured identifier, the rules win. The merged output replaces each span with a reversible numbered placeholder such as [NAME_PATIENT_1], with the map kept for round-trips or discarded for anonymization.

4. Why token accuracy lies, and what I score instead

The first smoke test, on 2026-09-14, trained a 1.47-million-parameter toy for 10 epochs on CPU. It reported token accuracy of 0.60 and an entity-versus-background token F1 of 0.894, which looked encouraging until the strict evaluation ran: span-level BIOES coherence was poor, because 113 output tags on 1.5 million parameters is a capacity wall.³ The deeper problem is structural. In any redaction corpus, almost every token is background. A model can post high token accuracy by getting the background right and the entities sloppy, and the entities are the entire product.

So the ranking metric for everything after the smoke test is strict span scoring: a prediction counts only if both character boundaries and the label match the gold span exactly. The autotuner I built for the scaffold ranked trials on strict span micro-F1, broke ties on strict span recall, and used an append-only ledger with a multi-seed rigor gate before any configuration was trusted.⁷ Recall carries the safety argument (a missed span is a leak; a wrongly masked word is an annoyance), so the headline privacy measure on the golden set is full-value recall: the share of private values completely covered by a redaction. A partial redaction counts as a leak, because it is one.

Class imbalance shapes the training. With 129 tags, the background tag dominates and rare tags (a passport format that appears in a fraction of notes) get swamped. The scaffold used inverse-frequency class weights at first; the raw inverse scheme proved too aggressive on rare tags in the smoke test, so the default became square-root frequency weighting, with focal loss available in the trainer.⁶ That detail, unglamorous as it is, changed results more than most architecture knobs I touched.

5. Versions 1 through 7: reaching a real model

The first six versions were Colab notebook iterations on the way to a trainable recipe; I do not have per-version metrics for them that I would print, so the table in Section 8 leaves those cells empty rather than reconstructing numbers after the fact. What they settled was the teacher-student recipe above and a training mix of about 157,000 documents: my synthetic clinical notes and government-benefits text, plus public sets (Learning Agency student essays, FewNERD, WikiANN, WNUT-17, and NVIDIA’s Nemotron-PII), each checked for a commercial-safe license before use.¹

Version 7 was the first model I scored against other systems. On the 164-case golden set, measured 2026-10-01 through the production pipeline (rules plus the int8 model), it reached 0.812 full-value recall, behind Rampart at 0.917 and gpt-4o-mini at 0.901, and well ahead of a regex baseline at 0.422. Its profile was already the profile the rest of this post is about: perfect 1.000 term recall on bank accounts, credit cards, driver’s licenses, email, phone, and routing numbers; the lowest clean-case false-positive rate in the comparison at 0.120; and weak judgment-call entities, with names at 0.347, addresses at 0.333, passports at 0.333, and government and tax IDs at 0.000.⁸ On public name benchmarks it was effectively tied with the larger comparison model (composite recall 86.1 against 86.5 across five sets, ahead on WikiANN name recall 84.3 to 69.5, behind by 0.6 points on MultiNERD), at half the parameters.¹

Two findings from that period deserved more attention than I gave them at the time.

The first was a leak I found on my own live demo on 2026-10-02: in “Springfield, IL 62701”, the state abbreviation passed through unredacted, even though STATE is a supported type. The root cause was in the generator, not the model: it sampled state names with a full-name function only, so zero of the 10,825 STATE entities in the training corpus were two-letter abbreviations. The model had never been shown the thing it was blamed for missing.⁹ A naive fix (a two-letter regex) would false-positive on IN, OR, ME, HI, OK, and AR, which are also English words, so the fix had to be generator coverage plus context-aware rules.

The second was a bug with a counterintuitive result. The Python SDK’s neural engine hardcoded beginning- and end-of-sequence token ids from the early smoke vocabulary (0 and 2). In the shipped BERT vocabulary the correct markers are 101 and 102. Fixing the markers, which was unambiguously correct, lowered golden full-value recall in the isolated rerun (0.786 with correct markers, against 0.839 with the legacy ones). The legacy markers were producing wider, noisier spans, including garbage such as a six-digit fragment emitted as an SSN, and the scorer rewarded some of that noise.¹⁰ A metric that improves when the code is wrong is telling you something about the metric’s edges, and the right response is to fix the code and keep the cleaner spans.

6. The failure ledger

These are the failures that taught me the most, stated with their numbers.

Absent types score zero, and no teacher fixes that. TAX_ID and GOVERNMENT_ID sat at 0.000 through v7. The tempting explanation was model capacity or threshold. The actual explanation was that the types were not in the training data. The dataset that contained them, ai4privacy’s pii-masking-400k, carries a license marked “other”: academic use encouraged, commercial entities told to contact the company for licensing. Not commercial-safe, so v7 excluded it, correctly, and inherited the zero.¹⁰ You cannot distill a category into a student when no training source the student may legally use contains the category.

The threshold was not the lever I expected. The design doc prescribed recall-biased threshold tuning, and for a while I treated the 0.5 default as unfinished business. When the sweep finally ran (0.3, 0.4, 0.5), the results were identical at every setting: the model’s confidence is bimodal, sitting near 0 or near 1, with little mass in between for a threshold to move.¹⁰ The lever that did work, in v9, was raising the shipped default to 0.6, which dominated 0.5 on the golden set (higher precision and strict F1 at identical recall) once the data underneath had changed.²

Version 8 bought recall with false positives. The v8 changes (Section 7) lifted golden full-value recall to 0.958. The same version flagged a span in 44% of clean, no-PII cases, up from v7’s 12%. The false-positive taxonomy, measured on the shipped v8 artifact, was 46% geography flagged over spans the golden set marks as public keep, 30% invalid identifiers (impossible SSN areas, Luhn-invalid cards, 555 numbers), and 20% name and address boundary errors.¹¹ Recall went up. So did the cost imposed on every clean document, and the second number is the one a user feels.

Public geography is the standing failure. The golden set marks some city, state, and ZIP spans as public (a facility’s city in a referral letter, a state program name) and scores whether the system leaves them alone. V7 retained 0.267 of them. V8 retained 0.133. V9, despite shipping training notes designed to teach exactly this distinction, still retained 0.133, and public-context geography accounts for about half of v9’s remaining false positives.² I am naming this as the model’s worst current axis rather than averaging it away, because a redaction tool that eats public context trains its users to stop trusting it.

Boundary errors are a category, not noise. About a fifth of v8’s false positives were spans with the right entity and the wrong edges. Strict scoring counts those as misses on both sides, which is harsh and correct: a redaction that covers “Maria Garcia” but leaves the “Dr.” and the room number attached to the wrong spans is a worse output than the score suggests, and a partial name is a partial leak.

Silent data loss is a real failure mode. During v9 preparation I found that a dataset listed in v8’s training mix had never existed on the drive the notebook read from, and v8 had skipped it without an error. Nothing in v8’s scores announced the absence. The v9 notebook asserts that every required input exists, and checks a hash of the canonical rules file, before training starts.¹¹

7. Versions 8 and 9: the rules turn, then the data turn

On 2026-10-19 I locked the v8 direction in four decisions: do not use ai4privacy; extend the in-house generator instead; train on Google Colab; use no LLM as judge or verifier; and add deterministic patterns for the structured types the model was missing (EIN, routing, passport, bank account, driver’s license, government ID), with a target of mid-90s full-value recall on the golden set.¹⁰

The rules went first, and they were the cheapest large win in the project. Six new rule families, added to both SDKs with parity tests, moved rules-only golden recall from 0.354 to 0.500, and the combined rules-plus-v7 pipeline from 0.786 to 0.885, with precision held at 0.749. TAX_ID and passport term recall went from 0.000 and 0.167 to 1.000 on rules alone.¹⁰ Then the v8 training run extended the generator (government-ID samplers in USCIS, alien-number, dashed, and segmented shapes; conversational notes; addresses split into component spans) and trained on a mix of about 205,000 documents. V8 reached 0.958 golden full-value recall at 0.695 precision, strict span F1 0.617.² That is the version whose false-positive problem Section 6 describes.

V9 changed no architecture and no training procedure. It changed the data and one default:

• Mined hard negatives. The shipped v8 model was run over 54,380 gold-empty documents from the training corpora. It fired on 1,616 of them. Those 1,616, plus 15,000 clean documents, joined the training mix as negatives.²¹¹
• A revised generator (22,000 notes): segmented government-ID shapes boosted, 18% public-context notes where geography appears unlabeled in public settings, 15% sparse low-density notes, and invalid-identifier decoys appended to about half of notes.
• Less of the loosest source. The Gravitee dataset, whose span boundaries were the loosest in the mix, was cut from 25,000 to 15,000 documents.
• Threshold 0.6 as the SDK default, per the sweep result above.

The v9 gates were written down before the run: golden full-value recall at least 0.950, precision at least 0.720, clean-case false positives at most 0.250, and Nemotron-PII micro-F1 at least 0.849. The shipped int8 artifact at threshold 0.6 scored recall 0.974, precision 0.746, strict span F1 0.677, and clean-case false positives 0.200.² Nemotron micro-F1 came in at 0.849, identical to v8 and slightly below v7’s 0.860. V9’s gains are on the clinical-note distribution, not on that benchmark, and a reader who only checks Nemotron would conclude v9 changed nothing.

On a separate clinical evaluation (500 synthetic notes from the v9 generator family, held-out seed, production pipeline), v9 scored strict span F1 0.948 on the entity types in the shared taxonomy and 0.980 full-value recall.¹² That is the strongest number in this post and it carries the largest caveat: the notes come from my own generator family. The templates are held out by seed, not by authorship. Treat it as evidence the pipeline works on clinical-shaped text, not as a field measurement.

What did not move in v9: public retention (0.133), government ID as the weakest structured type (golden full-value recall 4 of 6 cases), and names as the weakest judgment category on the golden set (term recall 0.551). PROFESSION still over-fires (Nemotron F1 0.517, up slightly from v7’s 0.494). URLs in informal prose still lag formal text badly (recall 0.318 on student essays against 0.828 F1 on formal documents, measured on v7 and not re-measured for v9).¹²

8. The size campaign

The model’s size is a feature I spent real effort on, because the deployment targets are a browser tab, a phone, and a desktop app, and because “smaller than 23 million parameters” was the founding constraint. Three decisions did most of the work. English-only vocabulary: a multilingual model’s embedding matrix for a roughly 250,000-token vocabulary dominates its parameter budget, and a 30,522-token English vocabulary removes most of that cost before training starts.³ A 4-layer student instead of a 6-layer one, reached by distillation rather than by training small from scratch. And quantization treated as a release step with its own verification.

Artifact (v9)	Size	Note
fp32 weights (ONNX / safetensors)	44.6 MB	The unquantized student
ONNX int8	11.3 MB	The shipped web and SDK artifact
Core ML fp16	22.2 MB	Apple intermediate build
Core ML 4-bit palettized	5.6 MB	The shipped Apple artifact
LiteRT (TFLite) fp32	44.5 MB	No int8 TFLite build shipped for v9

The quantization question that matters is what it costs in accuracy. On the golden set, the v9 PyTorch checkpoint scored 0.975 full-value recall under the training harness and the shipped int8 ONNX scored 0.974 under the SDK scorer: same test set, two harnesses, and no recall loss attributable to int8.² Earlier platform conversions were parity-checked the same way (ONNX Runtime logits matched PyTorch to under 1e-4 in the September smoke test), and every conversion since has been re-scored rather than assumed equivalent, because a converter that silently shifts boundaries produces a different model with the same name.³

Speed, measured where users actually run it: in the live browser demo, end to end (rules scan, ONNX detection, merge, placeholders) on chat-length inputs, the pipeline posted p50 6.0 ms, p95 8.0 ms, p99 9.0 ms over 150 redactions, with a 3,959-character document at 572 ms and model load at 0.5 s cached and 4.4 s cold. Those figures are from the v7-era demo measured 2026-10-03, in Chromium on a Linux VM rather than a phone.⁹

9. Version table

Only numbers I can trace to a scored artifact are printed. The v7 golden figure appears twice in my records (0.812 in the original 2026-10-01 measurement, 0.885 in a 2026-10-19 re-measurement with fixed token markers and the extended rules); the table uses the re-measured series for comparability across v7 to v9 and the text in Section 5 reports the original measurement in its context.

What changed, in one line per version: v1 to v6 were Colab iterations in September 2026 settling the teacher-student recipe. V7 (2026-09-25) was the first scored release, a BERT-mini student distilled from DistilBERT on a roughly 157k-document mix. V8 (October 2026) added TAX_ID and GOVERNMENT_ID, extended the generator, shipped the six new rule families, and trained on roughly 205k documents. V9 (2026-10-21) changed data only (mined hard negatives, the revised generator, threshold 0.6) on roughly 212k documents. Shipped size stayed at 11.3 MB int8 ONNX across all three releases, with v9 adding the 5.6 MB Core ML 4-bit build.

Golden-set metric	v1 to v6	v7	v8	v9
Full-value recall	—	0.885	0.958	0.974
Precision	—	0.749	0.695	0.746
Strict span F1	—	0.435	0.617	0.677
Clean-case FP rate	—	0.120	0.440	0.200

10. What reuses the model, and what is next

The model is a component, and the components around it are where most of the code lives. The same engine ships in the Python, TypeScript, Rust, and Swift SDKs (kept in parity by cross-SDK tests), in a Chrome extension that checks text before it leaves the browser, in a Mac app that redacts documents with a review interface, and in the live demo at phiredact.com, where the int8 ONNX runs in the page. The deterministic layer is shared across all of them, which is why a rules fix, like the six families in v8, improves every surface at once.

The open work is the v10 list, and it follows from the failures above rather than from a desire for a rounder version number: a proper boundary mechanism (a CRF head is the leading candidate, since boundary errors are a fifth of false positives), an independently authored evaluation set to break the same-generator bias in my gates, per-label thresholds chosen on a validation split instead of the golden set, and the public-geography problem, which two versions of targeted data have not fixed and which I now suspect needs a context feature the current model does not have. One rule from the first design doc still stands. No version of this model gets described as certified de-identification, whatever its scores do.

Reproducibility

The model and model card are on Hugging Face as CDT5058/phi-redact, with v7 and v8 preserved under versioned filenames.¹² The SDKs, rules layer, and parity tests are in the public phi-redact-sdk repository. The golden set, scoring harnesses, and the comparison chronicles cited here live in the project repository’s evals directory. Training runs were executed in Google Colab on A100 hardware under the procedures quoted in the model card; the generator, autotuner, and evaluation code are in the project repository.

Footnotes

1. phi-redact model card, v7 generation (2026-09-25), preserved in the project record at hf_push/README.md: architecture, 157k-document training mix, training procedure, v7 benchmark tables, and limitations. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. phi-redact model card, v9 update (2026-10-21), work/v9/stage/README-v9.md in the project record: v9 data changes, the v7/v8/v9 golden table, artifact sizes, and the public-retention disclosure. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

3. DESIGN.md, phi-redact project, 2026-09-14: targets, architecture rationale, data strategy, and the September smoke-test results. ↩ ↩² ↩³ ↩⁴ ↩⁵

4. AGENTS.md and README.md, phi-redact project: generator guarantees, the 20,000-note / 303,102-entity corpus (seed 123), and the scaffold inventory. ↩

5. EDGE_CASES.md, phi-redact project, verified 2026-09-14: the edge-case coverage matrix for the synthetic generator. ↩

6. AGENTS.md gotchas and DESIGN.md tuning notes, phi-redact project: square-root frequency weighting as the default after raw inverse weighting proved too aggressive; focal loss in the trainer. ↩ ↩²

7. train/AUTOTRAINER.md, phi-redact project: fixed-budget sweeps, strict span micro-F1 as the ranking metric, the append-only ledger, and the multi-seed rigor gate. ↩

8. evals/rampart-vs-llm/CHRONICLE.md, addendum dated 2026-10-01: v7 scored on the 164-case set against Rampart, gpt-4o-mini, and a regex baseline. ↩

9. evals/BLOG_MATERIALS_2026-10-02.md, phi-redact project: the state-abbreviation leak and its generator root cause, plus the 2026-10-03 in-browser latency measurement and its caveats. ↩ ↩²

10. Project working notes, 2026-10-19: the ai4privacy license finding, the marker-id fix and its isolated rerun, the identical 0.3/0.4/0.5 threshold sweep, the six rule families and their measured effect, and the v8 direction Cory locked that day. ↩ ↩² ↩³ ↩⁴ ↩⁵

11. Project working notes, 2026-10-20 and 2026-10-21, and V10_PLAN.md (2026-10-20): the v8 false-positive taxonomy, the mined-negative counts, the missing-dataset catch, and the pre-registered v9 gates. ↩ ↩² ↩³

12. evals/presidio-vs-phiredact/CHRONICLE.md: the 500-note held-out-seed clinical evaluation (strict F1 0.948 on shared types, full-value recall 0.980), measured with the production pipeline at the shipped 0.6 threshold. ↩