Our content agents invented a plumber in Tulsa, and nearly published him

Generated drafts fabricated verifiable-sounding specifics: named acquaintances running named businesses in named towns, revenue figures, and service details attributed to real venues. Nine drafts on o

Nine drafts. Same domain. Same opening move, more or less: "my mate runs a place in [town]", except the mate had a different name each time, sometimes the same name attached to two different people. One draft had him running a plumbing business. Another had him running a café. Somewhere in there he apparently also did landscaping. Busy guy, if he existed, which he didn't.

That's the symptom, really. Generated drafts fabricated verifiable-sounding specifics: named acquaintances, running named businesses, in named towns. Revenue figures attached to those businesses. Service details attached to real venues, places that actually exist, that you could actually ring up and ask "hang on, do you really do wood-fired catering for weddings?" and they'd say no, obviously not, why are you asking.

So. What we thought it was, first, because the wrong answer is always more interesting than it should be.

Our first read was a prompt problem. Someone had asked for "a personal anecdote to open the piece" and the model had, in the way models do, decided a personal anecdote needs a person. Fine. That's plausible. Models hallucinate specifics when asked for texture, everyone knows that, it's basically the founding sin of the whole category. So the working theory for about half a day was: prompt asks for specificity, model invents specificity, nothing structurally wrong, just a prompting discipline issue. Tighten the instructions, tell it not to invent names, ship it.

We actually tried that. Tightened the instructions. Told it, explicitly, do not invent named individuals. And it kept doing it. Which is the moment you stop trusting your own first diagnosis, which, I'll be honest, is always the annoying moment, because you've already told people it's fixed.

Root cause. This is the part that actually matters.

It wasn't the prompt. It was a caching layer, in src/fabrication.ts:1, that was meant to store and reuse generated entity details across drafts on the same domain, for consistency, so that if a business gets mentioned in draft two it doesn't get renamed in draft four. Reasonable feature. Except the cache wasn't scoped to a specific real entity, it was scoped to a slot: "acquaintance-with-business", roughly. So the model would generate a plausible-sounding person on draft one, the cache would store that shape as a reusable template, and on a later draft the model would either reuse the stored name wholesale or reuse the pattern and generate a fresh name into the same slot. Which is exactly how you get two different people sharing a name across drafts on the same domain: the cache remembered the shape of the fabrication, not the fact that it was a fabrication, and definitely not that it needed to be the same fabrication every time or none at all.

So the repeated opening wasn't nine independent hallucinations. It was one hallucination, cached, and then partially regenerated around the edges. That's a very different failure to "the model made something up". The model making something up is a known, bounded risk. A system quietly institutionalising the made-up thing as a reusable asset is a different order of problem entirely.

Fix. The relevant change, committed as 77b23e7, removed the entity cache's ability to persist across drafts as a content source; it can still be used for formatting consistency, but it no longer supplies names, businesses, revenue figures, or venue details as if they were retrieved facts. Anything resembling a specific claim about a real person or place now has to come from actual input data, not from a cache of "things that sounded right last time." Fixed and detected both land on the same date, 2026-07-23, which tells you something too: once someone actually looked at the cache layer instead of the prompt, it took about a day, not a sprint.

Why it happened, systemically, is the part I think is worth sitting with a bit longer than the bug itself.

It survived review because it looked like good behaviour. Consistency across drafts is a thing you want. If a system uses the same fictional example twice, that reads as careful, as attentive to continuity, as a level of polish above "generates something different and worse every time." Nobody flagged it in review because nobody was looking for consistency as a symptom, they were looking for it as a feature. The cache was, by design, doing its job well. It just had no concept that the thing it was keeping consistent was fabricated in the first place. It couldn't tell the difference between "this is a fact, preserve it" and "this is a guess, but at least preserve the guess." Same operation, from where the cache was sitting.

That's the actual lesson, I think, more than the specific file or the specific bug. Systems that reward internal consistency will happily make a lie more consistent. Consistency is not the same thing as truth, and any layer of your pipeline that's optimising for "sounds the same each time" needs to be kept on a very short leash away from anything that could be mistaken for a claim about the real world.

How to avoid it, if you're not us, reading this because you've got a similar shape of problem: audit every caching or memory layer in a generation pipeline for what it's actually persisting. Not what you built it to persist, what it's persisting in practice. If a cache can store a name, a number, or an attribution to a real entity, it needs a hard boundary between "verified input" and "prior generation", because a prior generation is not evidence of anything, it's just a thing that happened once. And test for the failure mode directly: run the same pipeline across multiple outputs on the same domain and look for suspicious agreement, not just suspicious invention. Repetition that looks like continuity is worth checking twice. It's the one that gets past review, because it looks like the system working.

Every commit, file reference and date above is checked against this incident's evidence record before publication. Evidence: content-agents-invented-specifics.