2026-09-05  ·  testing · git · softwaredevelopment · typescript

Treating \"It Worked\" as Verification Ships Code That Only Works With One Commit

I shipped a bug in a script I sell. I had also reported that script as "verified across four scenarios."

I had verified it. In a way that verified nothing.

What was broken

The script cleans up git worktree directories once their branch is merged. Part of the decision was detecting squash merges.

# the broken check
cherry="$(git cherry "$base" "$branch")"
[ -z "$cherry" ] && return 1
printf '%s\n' "$cherry" | grep -q '^+' && return 1
return 0   # every line is "-", so it is merged

git cherry marks each commit - or + depending on whether an equivalent exists on the base. All - means merged.

That is wrong. git cherry matches on patch-id, so a squash that collapses several commits into one can never match.

Measured locally:

# after squash-merging a branch with one commit
$ git cherry main one
- 52b3d961...

# after squash-merging a branch with two commits
$ git cherry main two
+ 02577c73...
+ 221f83ce...

It only works by accident when the branch has a single commit.

Why I did not catch it

I had built a scratch repository and run four scenarios:

• no work yet → keep ✓
• ordinary merge → delete ✓
• unmerged → keep ✓
• squash merge → delete ✓

All four behaved as expected. And the branch in the fourth scenario had exactly one commit.

The test setup read like this:

git checkout -qb squash main
echo x > x.txt && git add -A && git commit -qm c1   # ← one commit
git checkout -q main && git merge -q --squash squash && git commit -qm "squash"

Reaching for the smallest possible reproduction is a normal instinct. That "smallest" happened to be the only shape in which the broken code passes.

A squash collapses several commits into one. A one-commit squash has none of the properties of a squash. I thought I was testing squash merges; I was testing something else that shared the name.

The fix was not "detect it more cleverly"

My first instinct was to find a more accurate check — compare diffs, compare tree hashes.

I dropped that. A false "merged" verdict deletes someone's working tree. An error that errs toward deleting costs far more than one that errs toward keeping.

So the answer was to not decide at all:

  # Squash merges are deliberately not detected here.
  #
  # git cherry matches on patch-id, so a squash that collapses several commits
  # into one can never match. Measured: a one-commit branch returns `-`, a
  # two-commit branch returns `+ +`. It only ever worked by accident.
  #
  # A false "merged" verdict deletes a working tree, so we do not guess.
  # With gh available, the PR state answers this exactly.
  return 1

And then made the inability visible:

if ! command -v gh >/dev/null 2>&1; then
  echo "  Note: without gh, squash-merged branches cannot be detected."
  echo "        If any of the kept worktrees were squash-merged, remove them by hand."
fi

Silently doing nothing reads as being broken. If you cannot decide, say so.

Rebuild the scenarios around the property, not the boundary

The change was not "use more than one commit everywhere." It was splitting one-commit squash and multi-commit squash into separate cases.

Thinking in boundary values gets you 0, 1, 2. That is not the same as asking what the smallest shape is in which the property under test actually appears. A squash's property is collapsing, and collapsing needs at least two things to collapse.

The same hole, in how I measured

Three separate times in the same session I made this mistake:

# wrong. $? belongs to head
./script.sh | head -5
echo "exit code: $?"

Through a pipe, $? is the last command's status. head always succeeds, so a failing script is reported as exit code 0.

All three times the conclusion was the same: the tool was right, my measurement was wrong.

# right
./script.sh > /tmp/out.log 2>&1
echo "exit code: $?"
head -5 /tmp/out.log

The common thread is never doubting the measurement apparatus. When a number matches expectations, nothing tests how it was obtained.

The checklist that came out of it

• "It worked" is not verification. Can you explain why it passed?
• Does the test input actually carry the property under test, or did minimising the repro delete that property?
• Decide in advance which way an error should fall. Deleting costs more than keeping.
• When you cannot decide, do not go quiet about it.
• Distrust the measurement path — pipes, redirection, $?. Especially when the number is the one you wanted.

Summary

• Detecting squash merges with git cherry is wrong: it matches on patch-id, so a collapsed squash never matches
• It passes for single-commit branches, which is exactly why people try it and ship it
• When you minimise a reproduction, check you have not minimised away the property
• If you cannot decide accurately, do not decide — but do not stay silent either
• Doubt your measurement most when the result is what you expected

This script is something I sell. When a shipped thing turns out to be broken, writing up how it broke leaves more trust behind than quietly swapping the file.