# Noctem Labs

> Production failures, diagnosed. Every claim traceable to the commit that caused it.

Post-mortems of real production failures in a multi-agent system running on
one server. Every claim is checked against a stored evidence record (commits,
file references, exact error strings, dates) before publication. Nothing is
published while the underlying fault is still live.

Site: https://noctemlabs.co.uk
Author: Jack Stovell

## Failures

### Supabase image transform returns 400 for format=jpeg, and every placeholder silently disappeared
URL: https://noctemlabs.co.uk/failures/supabase-image-transform-returns-400-for-format-jpeg-and-every/
Repository: paws-of-london
Symptom: format=jpeg
Symptom: quality=10
Cause: getPlaceholderUrl asked Supabase for format=jpeg at quality=10. The transform accepts neither — format must be origin|webp|avif and quality must be >= 20 — so every low-quality image placeholder on the site 400d and fell back to flat colour. It shipped 2026-03-01 and was found 2026-07-23. The unit test asserted the broken behaviour, so the suite stayed green for the entire time.
Timeline: introduced 2026-03-01, fixed 2026-07-23

### Our content agents invented a plumber in Tulsa, and nearly published him
URL: https://noctemlabs.co.uk/failures/our-content-agents-invented-a-plumber-in-tulsa-and-nearly-published-hi/
Repository: claude-editor
Cause: Generated drafts fabricated verifiable-sounding specifics: named acquaintances running named businesses in named towns, revenue figures, and service details attributed to real venues. Nine drafts on one domain opened with the same invented-mate construction, including two contradictory people of the same name.
Timeline: introduced unknown, fixed 2026-07-23

### Alphabetical ordering quietly starved four sites of every content slot
URL: https://noctemlabs.co.uk/failures/alphabetical-ordering-quietly-starved-four-sites-of-every-content-slot/
Repository: claude-editor
Cause: listSiteConfig returns sites ORDER BY site, and commissionEmptySlots drained each site's entire empty-slot queue before moving to the next, spending one global per-cycle budget. The alphabetical head consumed the whole cap every cycle and the tail received nothing — permanently, not occasionally. Measured on the live backlog the day it was fixed: marketframe, a parked product with no revenue, took all 10 commissions; paws, the site with the largest audience, and reviewnudge, the only site that has ever taken money, took zero.
Timeline: introduced unknown, fixed 2026-07-30

### Approve all approved everything except the backlog it existed to clear
URL: https://noctemlabs.co.uk/failures/approve-all-approved-everything-except-the-backlog-it-existed-to-clear/
Repository: claude-editor
Cause: Batch approval resolved "all" through getSlots({from: new Date()}), which renders as slot_at >= now. Every past-due filled slot was invisible to the one command that exists to approve it. It approved the future, skipped the backlog, and returned ok.
Timeline: introduced unknown, fixed 2026-07-23

### Our QA judge called articles incomplete. It caused the truncation.
URL: https://noctemlabs.co.uk/failures/qa-judge-called-articles-incomplete/
Repository: claude-editor
Symptom: Article cuts off mid-sentence in the final section ('makes you sc'). This is a critical structural failure that renders the piece unpublishable.
Cause: validate.ts truncated draft content at 8,000 chars before handing it to the editorial judge, which then reported the truncation it had itself caused as a critical structural failure. The defect scaled with length: the longer and more thorough the article, the more certainly it failed QA. The workaround in use was to trim articles until they passed.
Timeline: introduced unknown, fixed 2026-07-30