Skip to main content
Contents
Seeded Flora banner for "synthetic-data-provenance" — dusk paletteKOCD4DKK↑11O4C6CK04↑C1C/KD441K//\oDCO→C64↑C/OCK0KC//C↑4↑/o1C4ODC↑D6\44D6K→↑CDoCCKCK6DK464D↑/DKKo4O1DCC→↑C4K61o/666\10CCoCKKD→DCCKK6CC/D/oCCKO\\C6/O\↑4D46K166404→↑OKKO6KC→/ooCCODC4C4C/6/4C00C→C/O4/6CoKC\DOK1K\64COK\O40KDK64/1C0o/↑CoC→↑/o4K64/6→K44→66→CKC/44/1K/C16COC6C6↑4o614o6\/4↑0CCKD6\K/↑6D/C//DC6\/DOCCCC\/DK0\1K↑OC→O4K64DK4→KoKD\\6D414KDCOCOC→D→40Do/4↑/\4D→C6CCC///\D64↑→/K↑↑C/oDCD6\D6O↑KC6604//C4460K\↑/K4OD14C116CD1C→4D4D/↑44///4CDDC4\OOCCC4↑↑6/0o/6C1D/014oKOooCD4CKC6/C1C→C4→O→→O1C64oC/CD1→11K↑6D6DC6O4CC↑CO4C40DDDK/D46→Ko4CK6C414D0/16oD/↑↑K6CCK\→DODKD/
Seeded Flora banner · seed: synthetic-data-provenance · flora · dusk — reproduce it at Flora

Synthetic Data Is Only as Good as Its Provenance

Generating benefits cases from the rulebook instead of from real people’s files, and what it took to trust the labels. The views represented here are my own. Last updated: November 2026

Executive summary

I build synthetic test cases for government benefits programs: a household, an income picture, a scenario, and an eligibility outcome that is correct by construction, with a rationale trace that cites the rule that decided it. The households come from real Census microdata; the rules are encoded by hand from primary sources: the CFR, state manuals, federal charts. Nothing is trained on real case files, and no real constituent’s data is in the loop.

  • The labels are only as good as the parameters, and the parameters rot. Five years of my SNAP categorical-eligibility files (FY2021–FY2025) carried parameters marked estimated, carried back from a current chart because the historical charts were hard to reach. When I finally verified each year against its own FNS chart, most rows held. Some did not. Virginia is absent from the May 2020 chart entirely; Nebraska was 130% of poverty in 2021, not the 165% I had carried back.
  • Sometimes the data was right and the world had changed. Alaska and Arkansas showed as “No BBCE” in three editions of the State Options Report and as BBCE states in the August 2025 chart. Both readings were correct. The states adopted the policy during 2025.
  • The uses rank themselves. Edge-case testing of constituent-facing services is the strongest use and the one I would defend anywhere. Post-training on synthetic rationale traces is promising and dangerous in a specific way: the synthetic set can be a training ground and a regression test, but never a validation set. Modeling past behavior is the weakest use, because past behavior includes past misapplication.
  • The thesis. Synthetic data does not eliminate the hard part of working with benefits data. It moves the hard part from data collection to source verification. That trade is worth it if you do the verification.

1. Five files marked “estimated”

Broad-based categorical eligibility (BBCE) is the policy that lets a household qualify for SNAP through a TANF-funded benefit instead of the standard income and asset tests. Each state sets its own gross income limit and asset rule, and FNS publishes the table as a chart that changes over time.

My generator needs those parameters per state per year, so I keep a data file per fiscal year. For FY2026 I had the August 2025 chart. For FY2021 through FY2025, I had verified which states had BBCE from the FNS State Options Reports, but each state’s income percentage and asset rule were carried back from the 2026 chart and marked, row by row, "verification_status": "estimated". The dated charts for the earlier years sat behind archive snapshots and bot checks I could not get past from my build environment.

That is a common way for synthetic data to go wrong quietly. The file looks complete. The cases generate. The labels are internally consistent. They are just, in places, wrong, and nothing downstream can tell, because the generator is the only witness.

Getting the charts took a second environment with broader web access and an afternoon of archive archaeology: the chart is a PDF attachment whose link does not exist in the earliest archived pages at all, and the archived PDFs only parse under the archive’s raw-capture URL form. But each fiscal year does have its own chart, and once I had all five, every row could be checked against its own year instead of a borrowed one.

Mostly, the carried-back values survived. The failures:

  • Virginia is not in the May 2020 chart. It appears in July 2021, at 200%. My FY2021 file had a Virginia row, borrowed from the present.
  • Nebraska was 130% in July 2021, Oregon 185%. Connecticut was 185% and Pennsylvania 160% in June 2022; Michigan’s asset limit was $15,000. Louisiana was 130% and Maine 185% through August 2024. New Mexico rose from 165% to 200% only in the August 2025 chart.
  • Alaska and Arkansas are absent from all five historical charts and present in August 2025: Alaska at 200% with no asset limit, Arkansas at $5,500 of assets with a two-tier income limit. CBPP’s current explainer dates Alaska’s adoption to 2025. The “discrepancy” that started the investigation was two true statements about different years.

The files are verified now, and the verification is the research result. A synthetic dataset is a claim about the world, parameter by parameter, and each parameter has a source and a date or it has a guess.

2. What the generator actually is

“Synthetic” covers very different machines, so here is exactly what this one is.

The generator does not learn from case files. It combines two things that are each real in a different way. The population is real: households sampled from Census ACS microdata, with their actual sizes, incomes, and compositions, rather than independent draws from marginal rates that produce households no census has ever seen. The rules are stated: eligibility logic encoded from the governing documents (for SNAP, the federal rules plus each state’s BBCE parameters; for other programs, the state manual, the regulation, the federal chart), with an effective date on every parameter, because the four states I have worked through in detail update on four different cycles.

A generated case is a household run through the encoded rules. The label is exact by construction: the generator knows which check failed, and the design rule is that an ineligible outcome has exactly one determinative failure, so the rationale trace can say “denied because gross income exceeds 200% of poverty under this state’s BBCE limit” and that sentence is the whole causal story, not a post-hoc explanation. Where my encoding is wrong, the label is wrong, which is why Section 1 is the paper under this one.

3. The strongest use: edge-case testing of services constituents actually touch

Eligibility systems fail at boundaries. The household at 130% of poverty in a 130% state and the same household at 131%. The state whose rule is the exception: Maryland’s LIHEAP limit is not a single percentage but a ramp by household size, and Arkansas runs two BBCE tiers by age and disability. The case type that is 0.3% of a real caseload and therefore never appears in a pilot, a training set, or a vendor’s demo data.

You cannot test those cases with real constituent data, for privacy reasons that should not be negotiated. And you cannot wait for production to find them. In production the cost of a wrong boundary is a person wrongly denied food assistance, discovering it weeks later through an appeal.

Synthetic cases with known-correct outcomes are, in software terms, property-based testing for public services: generate the boundary, state the expected result from the rule, run the service (a rules engine, a chatbot, a caseworker-facing tool, an LLM with a policy manual in its context), and diff. The generator’s job is coverage of the rule’s corners, including corners no real caseload has visited yet. This use makes no claim about the world beyond the rulebook, which is exactly why it is the defensible one.

4. Post-training: promising, with a circularity trap

A rationale trace is also training material. A model fine-tuned on cases where every answer comes with the deciding rule, applied in the right order, is being taught the form of eligibility reasoning: categorical checks before income tests, the gross test before the net test, the state exception before the federal default. I believe that is worth doing, and it is the direction this work points.

The trap is measurement. A model post-trained on my generator’s distribution will perform beautifully on my generator’s test set, and I will have learned nothing, because the same assumptions wrote both. Synthetic data can be the training ground and the regression suite, run after every change to catch reasoning that broke. But validation has to happen somewhere the generator does not control: real adjudicated outcomes where they can be used lawfully, or sealed cases built from primary sources by a process that does not share the generator’s code. The moment a synthetic benchmark becomes the headline number, it has stopped measuring the model and started measuring the generator.

5. The belief I hold more carefully: modeling previous behavior

People say synthetic data lets you model how a system behaved in the past: how cases were decided, where they piled up, what a caseload looked like under rules that no longer exist. There is a version of that I believe. Running a measured population through a historical rulebook (the FY2021 parameters against 2021 households) is a legitimate counterfactual. It tells you what the stated rules would have done, and it is how I would want to evaluate a policy change before enactment.

The version I do not believe is behavior cloning. Observed past decisions are not ground truth about the rules; they are ground truth about the rules as applied, by offices with backlogs, under guidance that was sometimes wrong, to applicants who sometimes never applied because the process was hostile. Train on that and you have built a machine for reproducing the historical error rate with confidence. My own SNAP work points the other way: when I recovered the effective formula from real payment-error data, the formula layer checked out against the statute and the error lived in the fact layer, in the inputs rather than the rule. The stated rule is the better witness. Model the rulebook and the population; treat observed behavior as something to explain, not something to copy.

This argument also has a scope limit. It works where a rulebook exists and can be checked: benefits eligibility, tax provisions, permitting rules. It says nothing about synthetic data for domains where the “rules” are taste, sentiment, or clinical judgment, and I do not think the provenance discipline transfers there in the same way.

6. What verification costs

Verification is not a one-time tax. Parameters change under you: the District of Columbia cut its adult Medicaid limits effective January 1, 2026 while I was verifying a file built from January 2026 tables. Sources contradict each other in ways that take real reading to settle. Texas’s handbook contains both “$120 and 20%” and “$120 and one-third” for TANF deductions, and both are correct, in different tests, for different programs sharing a page; two secondary summaries I relied on had each grabbed one and labeled it the rule. Some sources are simply gone or stale: a federal eligibility page I cross-checked against turned out to be dated December 2023, presented without fanfare as current. And some sources stay unreachable. The Urban Institute’s online Welfare Rules Database tables are blocked behind a bot wall from every environment I have, so the TANF work rests on state manuals and regulations instead, which turned out to be the better foundation.

The generator, the data files with their per-row verification status, and the experiments are public.1 The verification status field is the part I would most like other people to steal: a synthetic dataset that cannot tell you which of its parameters are verified and which are estimated is asking for trust it has not earned.

Footnotes

  1. github.com/ctrimm/synthetic-gov-data-kit ↩

Working on an agentic or LLM-powered system?

I advise teams building these systems — architecture, tool and orchestration design, retrieval, and evaluation for high-stakes, compliance-sensitive domains.