The Art of Not Reading Test Output

CC BY 4.0

Why running tests more makes them harder to fix

The conversation history is sent again in full with every response. The output of a run does not disappear there. After 5 runs, 5 runs' worth rides along every time.

Not all of it arrives, though. Claude Code puts a limit on the output of a command, and anything past it is cut. Of those 26,147 lines, what the 🤖AI receives is an excerpt of the head and the tail. If the failure sits in the middle, the 🤖AI reads it as "everything passed" without ever seeing it.

In tasks that ask a model to find the line it needs from a long input, a single irrelevant line was enough to lower the accuracy. Test output is the least related thing there is to the one failure you are fixing.

The mechanism: turn the way out into a single door

Replace the entry point for running tests with a single wrapper. The full text goes to a log; only a summary goes to the conversation. This is a reduced version of the real thing (the example is Rails / Minitest, but you can use it for anything by swapping the lines you grep for whatever your own test runner prints).

#!/bin/bash
# scripts/test.sh — keeps the whole output in a log, returns only a summary to the conversation
log=tmp/test_last.log
mkdir -p tmp
bin/rails test "$@" >"$log" 2>&1
status=$?

echo "--- summary (full log: ${log}) ---"
grep -E "^Finished in " "$log" | tail -1
grep -E "^[0-9]+ runs, " "$log" | tail -1   # the line with the counts. ⚠️ never cut this one (see below)

failures=$(grep -cE "^(Failure|Error):" "$log" || true)
if [ "${failures:-0}" -gt 0 ]; then
  echo "--- ${failures} failure(s)/error(s) (first excerpt) ---"
  grep -A 3 -E "^(Failure|Error):" "$log" | head -40
fi
exit $status

This is all that comes out.

--- summary (full log: tmp/test_last.log) ---
Finished in 277.851362s, 50.2067 runs/s, 318.7712 assertions/s.
13950 runs, 88571 assertions, 0 failures, 0 errors, 228 skips

My real summary is 29 lines long, and the minimal form above is its core 3 lines. The remaining lines are things like the list of areas that were never run. The whole text stays in tmp/test_last.log every time, and the first line of the summary says where that is. Throwing it away is no better. The 🤖AI has to be able to read it back from there.

What matters is that not one line has been added to the instructions given to the 🤖AI. You could write "summarize the output before you read it" in your instruction file, but as the introduction showed, that is a solution on the side that dilutes as you add to it. I changed the entrance, not the instructions. Rules can be broken; an entrance is the only way through.

Placing it is not enough, though, because the 🤖AI will not go through this entrance on its own. A standard command like rails test is a word burned into the 🤖AI by its training. The scripts/test.sh you put down is a word it is seeing for the first time today. Even if you write "use the wrapper" in your instruction file, what surfaces the moment it goes to run the tests is the standard command it has seen hundreds of millions of times. One line in an instruction file has to compete with a habit and a priority ingrained in the 🤖AI, and it usually loses.

The official docs draw the same line. What goes into the instruction file is "bash commands the 🤖AI cannot guess." What stays out is "standard language conventions the 🤖AI already knows." rails test is the latter and scripts/test.sh is the former. An instruction file can only hold the side the 🤖AI does not know. So I block off the standard command instead. Blocking it off completely is no better. I leave one way out, named explicitly.

On Claude Code, adding one hook that stops a bare rails test and points to the wrapper makes the entrance genuinely single. A hook is a script that cuts in immediately before a command runs, and you register it in .claude/settings.json like this.

{
  "hooks": {
    "PreToolUse": [{
      "matcher": "Bash",
      "hooks": [{
        "type": "command",
        "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/use_test_wrapper.sh"
      }]
    }]
  }
}

All it does is "if the command is a bare rails test, print how to use the wrapper and finish without letting it run."

#!/bin/bash
# .claude/hooks/use_test_wrapper.sh
command=$(cat | python3 -c 'import json,sys; print(json.load(sys.stdin).get("tool_input",{}).get("command",""))')
case "$command" in
  *scripts/test.sh*|*SKIP_TEST_GUARD=*) exit 0 ;;   # let the wrapper through, and the explicit escape hatch
esac
if printf '%s' "$command" | grep -qE '(^|[;&|[:space:]])(bin/)?rails[[:space:]]+test([[:space:]]|$)'; then
  echo "Run tests with scripts/test.sh <same args> (full output stays in tmp/test_last.log)" >&2
  exit 2   # exit 2 = do not let it run; make the AI read the guidance on stderr
fi
exit 0

The 🤖AI cannot move with the bare command, and gets to go on only once it has read the guidance. What works here is not the wording of the guidance but where it lands. The guidance from a hook comes back at the very spot where the 🤖AI typed the command, as a refusal to run it. When a human cuts into the conversation with "use the wrapper," that lines up behind as a separate exchange. It is the difference between a position you can choose to read and a position you cannot get past without reading.

What I stopped reading

The raw output of the tests. Neither the human nor the 🤖AI reads it. It is not thrown away, so I open the log only when a cause really has to be investigated with the full text.

Caveat: anomalies other than failures stop being visible

The blind spot of this mechanism is that anomalies that do not take the shape of a failure get hidden. A surge in skips, warnings, a drop in the number of tests run — even when a whole file of tests stops being loaded, it reads "0 failures" as long as everything left passes.

On my own machine, the test run itself once failed to come up. Not one test ran, yet the record of having run them stayed. Failures were 0, and the summary stayed quiet.

That is why I never cut the line with the counts from the summary. The 13950 runs, 88571 assertions, 0 failures, 0 errors, 228 skips in the run above is it. Cutting it down too far is no better. I never print the numbers alone; the item names always go beside them.

How to verify: confirm once that failures do arrive

Prerequisites

• The wrapper from this article (scripts/test.sh) is in place, and the hook that stops the bare test command is registered
• You have 1 or more tests, and they run on your machine

Time required: 5 minutes

Steps

1. You deliberately break 1 test. Open 1 test file that finishes quickly and change a single character of an expected value (assert_equal "abc", ... becomes "abd"). You put it back in the cleanup, so note down the file name and the line number
2. Ask the 🤖AI to "run the tests." Do not invoke the wrapper yourself — ask the 🤖AI, because what you want to confirm is what arrives when the 🤖AI goes through the door

Pass conditions (the summary that comes back has to satisfy all of them)

• The location of the full log is there (a single line like tmp/test_last.log)
• The name of the test you broke, and the gap between expected and actual, are there (the lines matching Expected: "abc" and Actual: "abd". The file name you noted in step 1 shows up here)
• The count line has failures at 1 or more (a line like 13950 runs, 1 failures)

If it does not pass

• It said 0 failures — the test you broke never ran in the first place. Check that the file you picked in step 1 is inside the target of the run, then redo step 2
• It failed, but no test name or gap is there — the summary is not picking up enough lines. Widen the range of the grep in the wrapper

Cleanup

• Put the single character back and have the 🤖AI run it once more. You are done when the count line returns to 0 failures
• Forget to put it back and everything you do after this keeps failing

What you confirmed is only the "a failure always arrives" side of it. If you begin operating without seeing this, you will be using it without being able to tell whether a quiet summary means "there are no failures" or "the wiring never delivers failures."

Next time, the art of not reading memory. I do not read back the notes the 🤖AI has been accumulating. And the notes growing has never been a problem.

Series: The Art of Not Reading

• ← Previous: Introduction: I Barely Read What the 🤖AI Outputs Anymore
• → Next: 2. The art of not reading memory
• All articles: Introduction: I Barely Read What the 🤖AI Outputs Anymore

End of CC BY 4.0