2026-09-05  ·  testing · typescript · programming · webdev

Every Test Passed and It Could Not Read a Single Real Record

I wrote a tool that reads GitHub Issues and reports the ones whose declared file scopes overlap.

Eight fixtures, all six rules covered, every test green.

Run against a real repository, it read zero Issues.

What came out

Eleven Issues, one identical complaint for each:

error #21  The scope section is present but declares no path.
error #19  The scope section is present but declares no path.
error #16  The scope section is present but declares no path.
...

The section is there. The contents are not being read.

Here is what a real Issue looks like:

## Scope

```
README.md, .github/workflows/self-check.yml
```

Inside a code block, on one line, comma separated.

My parser treated each line inside a code block as one path, then discarded anything containing whitespace on the grounds that a path has none. README.md, .github/... contains a space, so every one of them was thrown away.

What the fixtures said

## Scope

- `src/auth/**`
- `src/routes/login.tsx`

A bullet list. One path per line, in backticks.

Why did I write them that way? Because that is how I assumed people write them.

I also wrote the Issue template those real Issues came from. My tool could not read the format I designed myself.

Why no test failed

This is the part worth keeping.

The fixtures were written by the same person who wrote the parser, on the same day, holding the same assumption.

parser      implemented assuming "scopes are written as a bullet list"
fixtures    written assuming "scopes are written as a bullet list"
test        the two agree, so it passes

The assumption is duplicated on both sides, so the test cannot examine it. What it examines is that the implementation matches the fixtures.

This is not an implementation bug. It is a bug in how the tests were designed, and no amount of coverage finds it. More coverage adds more paths — through the same wrong premise.

The fix is not "write more tests"

Ten fixtures instead of eight would still be ten fixtures written by me, holding the same assumption.

Add one input you did not write.

gh api "repos/owner/name/issues?state=closed&per_page=100" \
  --jq '[.[] | select(.pull_request|not) | {number,title,body}]' \
  > test/fixtures/real-issues.json

Then pin it as a regression test:

// The real Scope sections were a single line inside a code block. The parser
// as written read none of them and called every Issue scope-empty.
const real = JSON.parse(readFileSync(`${F}real-issues.json`, "utf8"))
  .map((i) => ({ id: `#${i.number}`, title: i.title, body: i.body ?? "" }));

ok(!check(real, { files }).some((f) => f.rule === "scope-empty"),
   "reads a Scope written inside a code block");

That test sits outside my assumptions, because I did not author the data. Past reality did.

The parser side:

if (inFence) {
  // Inside a code block these are usually listed on one line, comma or space
  // separated. Real Issues were written that way, and treating the line as a
  // single path read none of them.
  for (const tok of line.split(/[,\s]+/)) if (tok) out.push(tok);
  continue;
}

The same shape again, the same day

The README said:

Requires Node 22.6 or newer

CI failed.

SyntaxError: Cannot use import statement outside a module

Without "type": "module", Node loads a .ts file as CommonJS. It worked locally because the local Node is 25.

Same structure:

README    claims it runs on 22.6
CI        verifies on 25
verified      that it runs on 25
not verified  that it runs on 22.6 — which is the claim

The claim and the place it was checked were not the same place.

The fix is to test at the boundary you claim:

strategy:
  matrix:
    node: ["22.18", "24"]

(22.6 turned out to need a flag for type stripping, so the floor moved to 22.18. I only looked that up because CI failed. Had it passed, the claim would have shipped false.)

Generalising

Data you produce yourself reproduces your assumptions along with it.

The exposure is highest here:

• anything that interprets input from outside (parsers, importers, webhook receivers)
• anything handling text other people wrote
• any format you defined yourself — the worst case, because you believe people write it the way you specified

One rule covers it:

Anything that interprets input gets one test with input you did not write.

Take one real record, freeze it, make it a regression test. That is one foothold outside your own assumptions.

And for claims:

Verify at the boundary you claim. If it says "N or newer", test on N.

Takeaways

• Fixtures carry the same assumptions as the implementation, so they cannot test those assumptions
• Coverage does not help. It adds paths through the wrong premise
• Add one input you did not write. Take it from production and freeze it
• The riskiest format is the one you defined, because you trust it is followed
• A README claim is unverified until it is tested at the boundary it claims
• I only checked because CI broke. Had it stayed green, the false claim would have shipped

---

Related

• Treating "It Worked" as Verification Ships Code That Only Works With One Commit
• What You Refuse to Check Decides the Quality of a Linter
• Write the Same Decision in Two Places and Only One of Them Gets Fixed

---

I publish the configuration for splitting Claude Code into separate personas — Architect, Coder, Reviewer, Conflict Resolver — under MIT. Copy it, run ./setup.sh, and it works. It does not depend on your tech stack.

https://github.com/quintetkit/quartet

I built one real tool using nothing but this workflow. Every Issue, PR, review and merge is still there. The parts that went wrong were not deleted.

https://github.com/quintetkit/mdlinkcheck

The version that adds a UI Designer persona, review criteria, a per-Issue parallel execution script and a 10-chapter guide is on the product page.

The full kit — five personas, the scripts and the complete guide in English and Japanese — is on BOOTH, a Japanese store with an English interface that takes international cards.

https://quartet-dev.booth.pm/items/8807156