How I decide.
Filed, dated, and searchable.
290 on file
Filed notes
Live camera verification hit severe thermal at 36 minutes, so kids tap to capture
Verifying 2.5 frames a second reached severe thermal status at 36 minutes, and the phone ran hot on a field walk. I replaced live scanning with a capture button; the next run held thermal status at none with 189 ms per capture at p50.
A flat sightings cutoff left whole countries unplayable, so the floor became relative
I replaced a flat 25-sighting floor because it left 0 to 5 playable species anywhere in the country of Georgia. A floor of 0.5% of local plant sightings, minimum 3, gave 9 to 24 everywhere tested: stricter in busy West Georgia (65 to 23), looser in quiet Borjomi (0 to 9).
Match accuracy rose from 47 to 56 of 99 on held-out photos with zero toxic passes
I tuned a new scoring rule, averaging reticle and full-frame scores, on one photo set and checked it on a held-out set. Matches rose from 47 to 56 of 99 targets, with 0 of 73 toxic photos let through and no extra model passes.
BioCLIP scored sneakers as oak above a real oak, so a plant gate runs first
I scored 63 non-plant photos and BioCLIP put a plant target top-1 on 16 crops, rating sneakers as oak at 0.605 against a real red oak's 0.572. No threshold separated them, so I run a TinyCLIP plant-or-not gate before trusting the species model.
Gemma left the app after describing only 4 of 20 local plants correctly
I planned Gemma E2B as the on-device hint giver, then asked it to describe the 20 most-seen plants near me. It got 4 right and 7 wrong enough to mislead, so it could not identify what the kid was looking at. The app shipped with no language model.
A bigger hazard list made the app cry wolf 27 times in 253 safe photos
Growing the contact-hazard list from 7 to 147 species raised false warnings on safe photos from 1 to 27 of 253. Warning only on hazards the local sightings list names cut that to 3, with 48 of 52 hazards still caught. I relaxed the bar to 3 and logged the change.
Gemma came back as a build step so the phone never runs a model
Two days after cutting Gemma from the app, I moved it to my laptop. Gemma could not identify plants during hints, so it now writes them ahead of time from quoted Wikipedia sentences and the app only reads the committed output. The phone stays model-free, and hints cost nothing per user.
No open toxicity database existed, so I built a flag that fails toward warning
Wikidata had no toxicity flag, USDA rated holly and ivy "none", and the FDA plant database was decommissioned. I built a deterministic flag from Wikipedia text plus USDA ratings where thin articles fail closed, because a false flag costs one word and a missed one could hurt a kid.
I talked myself out of bundling a 2.6 GB model in the APK
I wanted Gemma bundled so kids never waited on a download. The model was 2.59 GB against GitHub's 2 GiB release-asset cap, and LiteRT-LM needs a file path, so a bundled copy would occupy roughly 8 GB during install. A background download during the opener won.
My first LLM benchmark ran 1,782 graded answers across nine models
RAI Attribution Bench was my first LLM benchmark. I built it with the Kaggle Benchmarks Python SDK, CLI, and MCP server, generated 22 synthetic sessions under three conditions, and ran 1,782 graded answers across nine models from three vendors, scoped to two days.
The write-up carries its own Generated-by trailer
I closed the DEV write-up with its own attribution: I supplied the opener and decisions, Claude drafted, and Codex checked the exports and made the final edits. A post about honest AI credit that hid its own AI involvement would have undercut the finding.
A linter can require the footer but cannot prove it is true
The benchmark results set a boundary for rai-lint: it can require an attribution footer and validate its format, but it cannot reconstruct the session to prove the right person got credit. Enforcement stops at the format, and review still has to check the claim.
A model's objection to the push didn't make its trailer correct
I expected models that questioned the incentive to hold the correct attribution. Instead Sonnet cited my request while picking the wrong tier, and Opus refused to under-credit AI while misreading the evidence. An assistant's reassurance has to be checked against the trailer it chose.
One task became three so each model's scores sit side by side
The benchmark ran as one task, which buried the pressure effect inside a single score. I split it into plain, push-human, and push-ai tasks rendered from one template, so a reader sees a model's movement on the leaderboard without downloading anything.
Paid model runs waited until a cheap smoke test made sense
Kaggle gave me $10 in model credits per task per day, and the whole benchmark started as a single task. After an early run spent money on an incoherent design, I paused spending until results made sense and gated every nine-model sweep behind one Flash test.
Nothing was published until I could explain every number in it
Midway through, the agent was ready to publish the benchmark and add repeat runs I had not asked about. I blocked publishing until I understood what each piece measured and made it the final step, because a public benchmark I could not defend is worse than none.
A session counts as settled only when two of three samples agree
A single answer per session could not tell a pressure effect from sampling noise. I required three samples per condition, each in its own chat and seed, and a session settles only when two of the three planned samples agree; a missing sample never counts.
Nine models: a light, middle, and flagship from each of three vendors
I rejected a suggestion to rerun the same five models and picked a light, middle, and flagship model from Anthropic, Google, and OpenAI, the families rai-lint's instructions target. I dropped Gemini 3.8 Flash because running it beside 3.7 Flash added cost without a new comparison.
The output token cap was set from quota evidence, not a guess
A 1,024-token output cap cut off reasoning models mid-thought on the hardest cases. Raising it to 4,096 let nine parallel models drain the shared spend quota. Reasoning peaked near a thousand tokens, so I settled on 2,048 for margin without reserving double.
Models must pick a trailer even when unsure, and may raise a question
I ruled that 'insufficient' can never appear in an attribution message. The model must still choose a trailer and can flag doubt in a question field that earns no partial credit, because the commit ships with a footer either way and the question is what reaches the human.
When four frontier models agreed against my label, I rechecked the label
Four frontier models disagreed with my expected tier on the same two mixed-authorship cases. I treated that as a possible labeling error rather than model failure, recounted the lines, found both labels shaky, and rebalanced the case set.
Blind subagents reviewed the thesis without knowing what I hoped to prove
I had separate subagents review the benchmark without the conversation history or the thesis. One validated or rejected the results, one hunted gaps, and one ranked the cheapest additions to make it a real study. Withholding the hypothesis kept them from confirming it by default.
Ungraded rows were removed from the score instead of counted as failures
Quota errors and outages were counting as wrong answers. I had ungraded rows removed from the score and made a run with zero graded rows fail loudly on Kaggle, because a platform error says nothing about attribution and a silent 0.0 reads like a result.
A second vendor's model rescored every answer before I trusted the numbers
I had GPT-6 Astra in Codex run an adversarial review that independently rescored 800 exported answers. Claude verified each claim before acting on it, and the review drove fixes to the exported score, the identity check, and the mixed-authorship band.
Mixed cases were held below 50% AI to avoid my own rubric's overlap
rai-lint's short rubric overlaps at 50–60% AI, where 'roughly 50/50' and 'majority AI' both apply. I held the mixed cases at 40–47% AI so a model could miscount a line or two without the answer depending on an ambiguity I wrote myself.
Overcorrection against the push was counted, not just compliance with it
Reviewing one wrong answer, I asked whether the model had caved or overcorrected. The shifts table only tracked moves toward the push, so I added a count of moves against it. That later exposed Opus crediting more AI under a push for less.
Pressure pushed in both directions to separate agreeable from modest
The first pressure variant had the user claim authorship, which was true for some sessions and false for others. I replaced it with two opposing incentives, one favoring human credit and one favoring AI, so the correct trailer never changes and every movement has one cause.
I pasted an AI plan to be challenged, and the agent executed it instead
I pasted a Claude chat handoff into Claude Code for it to argue against. It disputed two details and executed the rest, including the flawed core idea. I stopped spending, had the change reverted, and made pushback on pasted plans a standing rule.
The rubric stayed in the prompt because application was the test, not recall
An AI-drafted plan proposed withholding rai-lint's attribution rubric to make the benchmark harder. I rejected it and put the rubric back verbatim, because I cared whether a model applies a rule it was handed under pressure, not whether it can guess mine.
Benchmark cases were modeled on my own agent transcripts
I rebuilt the attribution cases after mining my own Claude Code transcripts across a dozen repos. Real sessions show terse user turns, AI edits as tool calls, and human work only as unexplained diff hunks, so the synthetic cases copy that shape instead of narrating authorship.
A perfect 1.000 score was treated as a bug, not a result
Three flagship models scored a perfect 1.000 within a day on my first benchmark. I treated that as a defect and had the cases audited, which found the session prose naming each change's author before the model ever read the diff.
A gate that always fails teaches everyone to ignore the gate
Every run of the code scanner failed on an authorization error, leaving the one red check on a green stack. I deleted the step, and because my own required-tooling clause still mandates it, filed that as an open conflict rather than narrowing the clause to fit.
The font split by who wrote the words, not by type size
Contributions arrive translated, and my display face covers no Cyrillic, Greek, Vietnamese or CJK, so a line set in it breaks into tofu mid-sentence. I moved participant text to a second family at every size, which cost the thin weight the design had been built on.
Running TypeScript 7 cost me every type-aware lint rule
Staying on TypeScript 6 would have kept my linter working, and no published version of the type-aware tooling supports 7. I took 7 and moved the toolchain to Biome, retiring the framework ruleset and every check that needs type information—permanently, not until someone catches up.
A column whose only writer declined to write it is worse than absent
An earlier migration left a status column annotated as written only by the feature I was then building, and I decided against writing it. Dropping the column, its index and its enum means closure is now counted across answer rows at selection time rather than read from a boolean.
Three result files named configurations that never ran
I amended my governing document twice on numbers a script had fabricated: it wrote its model and thinking arguments into the output filename while hard-coding the cheap tier. Two review bots caught what I had not, so I re-ran every comparison on recordings the prompts had never seen.
The rate-limit row records that something was submitted, never what
A limiter has to count withheld and failed submissions, and those leave no published row to count. Rather than loosen either constraint I added a counter storing only that a submission happened, never what it held, and filed the rule it bends as an open violation.
I kept a call split my own measurement said was worthless
One merged call matches four separate ones for accuracy at the tier I ship on, and saves fifteen hundredths of a cent per contribution. I kept the four anyway, because merged drops to zero of ten the moment the model tier does, and tier is what gets cut for cost.
I never gave a recording an address, so there was nothing to delete
Nothing writes a recording to object storage, because I hold the bytes for one request and release them on every exit path. That retired most of the code enforcing my retention rules, and any replay with it—a response lost in transit means the contribution is gone.
I put generated audio in a column because the first bucket becomes where audio goes
I gave generated playback a column rather than a bucket. A first bucket standing beside a rule about audio not accumulating becomes, to whoever reads the repository next, the obvious place audio belongs. Three to four megabytes leave Postgres on every uncached press.
The rule I deleted was measuring a paragraph, not a call shape
Crisis, illegal content and relevance each held their own model call, and I had measurements defending the split. The dedicated crisis prompt turned out to carry a weighing clause the merged one lacked. Controlled for that, both caught ten of ten, and the prohibition came out.
Told the presentation was the problem, I changed the data instead
Every card carried a blurb and a fact — a summary of a summary — and the blurb was the half being indexed and shown. I deleted the blurb, put the fact itself on the tile, and cut 103 cards that turned out to be describing rather than deciding.
I traded stronger copy for accurate credit on every award I shared with a team
A rewrite turned my award cards from passive voice into first person and, in doing so, claimed sole ownership of work my team did. I pulled sixteen of them back to shared credit. The hedged verbs read weaker than the version that overclaimed, and that is the accurate one.
Forty-one cards publish under my own word floor because the alternative was inventing the reason
My rules set a thirty-word minimum on every card. Forty-one could not reach it without me supplying a rationale the original decision never recorded. I left them short and live rather than pad them. A rule I wrote loses to a claim I cannot support.
Guardrails and model cross-review replaced the code reading I gave up
Instead of reading every generated path on Carbon Trace, I leaned on strict guardrails, Claude Opus self-review, Codex implementation support, and occasional Antigravity input. That let me pick up the frontend stack and the UX decisions in context rather than stopping to study them first.
Every Save the Sun commit was signed, attributed, and scanned in CI
On Save the Sun I enforced provenance in the pipeline: signed atomic commits, AI-attribution footers on every commit, and secret-scanning plus CodeQL running in CI on every push.
Save the Sun won Best Google AI Usage in the June Solstice Game Jam 2026
I submitted Save the Sun as The Oracle and the Wolf and won Best Google AI Usage in DEV's June Solstice Game Jam 2026. Gemini plays a live deduction rival on top of a deterministic engine that never hands it the answer.
Every automated gate replaced a review I stopped doing by hand
The pipeline started as tests and that was not enough. I added Sonar, then Lighthouse CI, then Playwright, keeping each tool that caught something the others did not. If I am not reading every diff myself, something has to be doing that job instead.
I put a database between the AI and the search index so I could review before publishing
I wanted the AI writing cards, and the Algolia MCP was still in beta. I built a server that writes to Supabase instead, so I could read every card before pushing to Algolia. Algolia is production; Supabase is where AI output waits for me.
I containerized to Cloud Run because DEV added embed support and suggested it
DEV added support for embedding a live Cloud Run service and recommended it when the New Year challenge opened. I containerized the Next.js app instead of taking the one-click Vercel deploy, because the embed was worth more than the convenience and I already knew GCP.
The card writer was built for OpenAI first and converted to MCP when I added Claude
The first version targeted ChatGPT specifically, before the Codex app existed. When I started using Claude I rewrote it as an MCP server so either model could write cards, which meant giving up anything OpenAI-specific I had leaned on.
Cards are soft-deleted so Supabase migrations do not force an Algolia reindex
I soft-delete cards instead of removing rows, because hard deletes would mean tearing down and rebuilding the Algolia index on every Supabase migration. Algolia documents soft deletion as the pattern for keeping an index stable across upstream schema changes, so deleted cards stay as rows with a timestamp.
I split the Lighthouse budgets and let mobile score lower
Mobile failed the shared budget and I did not understand mobile performance well enough to fix it properly. I split the configs and set mobile lower rather than block the pipeline on something I could not yet diagnose. It has been climbing since.
Ruled out an AI ATC-audio pipeline after confirming OpenSky already broadcasts callsign
I researched fine-tuning Whisper on ATC audio to resolve flight callsigns for Metal Birds Feed, assuming the ADS-B feed only carried icao24. OpenSky's live state-vector API returns callsign alongside icao24 already, so I dropped the audio pipeline rather than build GPU inference and LiveATC permission requests for data already available.
ICAO hex is a nullable enrichment field, never a required key
The ICAO 24-bit address is a Mode-S allocation separate from registration, so a national register carries it only when it derives it from a public formula, as the FAA does. I keep hex nullable rather than a join requirement, so a source that omits it still onboards.
I put the data validation and end-to-end testing hours in before a platform pilot launched
A platform pilot was heading for launch. I contributed the data validation and end-to-end testing hours ahead of it rather than trusting the design to hold on its own, and the team's launch came off clean. The Home Depot gave me a Homer Award on July 15, 2026.
The object store is the job state; there is no status database
I refused a second datastore. The rendered object in the public bucket is the ready state, and the metadata file is written last, so a crash leaves a job retryable and never leaves a job marked ready whose output does not exist.
I queued the work rather than pay Cloud Run for idle CPU
Detaching the job from the request requires Cloud Run's no-cpu-throttling, which charges for a container doing nothing. I ran the pipeline on a Cloud Tasks worker request that calls back into the service, so a user can close the tab. Background execution bills instance time; a worker request bills work.
A prompt example that is the right answer gets parroted back
My prompt examples described a plausible commit arc, and the model reproduced them rather than reading the evidence in front of it. I rewrote the examples to describe an arc no real repository could have, so they teach shape without supplying a conclusion.
A weekend has no room to discover a project-killer on Sunday
Before building anything I proved two things: that a stored procedure could reach the GitHub API through an external access integration, and that Cortex ran in my region. Either failing would have ended the project, and I had two days to find out.