The Art of Not Asking AI to Summarize

CC BY 4.0

Why only the denominator drops when the 🤖AI summarizes

The denominator drops because a report is a summary. That report comes out unasked. As article 5 has it, an 🤖AI stops where the work looks finished. Even short of finished, the report comes out there.

The "0 mismatches" in that report is not lettering carried over from what the command printed. As last time has it, it is a sentence the 🤖AI wrote on the spot. The command was printing the denominator too. What the 🤖AI wrote was only the "0 mismatches" part of it.

The official prompting guidance says that telling a model "only serious problems" or "be conservative" when asking for a review makes it follow the instruction as written and report less. The page for another model spells out the drop a little further. It investigates to the same depth, yet fewer findings turn into a report. The amount investigated holds still, and only the amount that lands in the report moves.

What the guidance describes is the case of moving it by instruction. With no instruction, a report leans the same way, because how much gets written is settled on the training side.

Put a counting mechanism down and the number still gets rewritten wherever it passes through an 🤖AI.

The "0 mismatches" the 🤖AI wrote enters the conversation later than the original line. And as the conversation nears its limit, Claude Code clears older tool outputs first, then summarizes the conversation if that is not enough. The original line sits on the tool output side, so it goes first.

What is left afterward is the "0 mismatches" with the denominator gone. It is also what the 🤖AI takes as material next. By then the denominator is no longer around, so whether 169 is too few is beyond checking.

The mechanism: a fixed prefix on number lines, and a hook that checks the final reply

You can also just ask for it to be pasted as it is. But that is an instruction. As article 6 has it, an instruction works only when it is read.

In fact, right after I put down a policy file saying "place the denominator alongside," the 🤖AI got it wrong in the same shape.

So instead of asking, I put 2 things down: a fixed prefix on the number line, and a hook that looks at whether that line is in the final reply.

The command first. A fixed prefix goes at the head of the number line. Mine is [observed]. The denominator goes on the same line.

# the glossary compared against its table. "0 mismatches" alone leaves the reader no way to doubt whether 169 is too few
puts "[observed] glossary: #{rows.size} terms x #{chapters.size} chapters, #{bad.size} mismatches"

Then the hook. After a tool has run it remembers the lines carrying that prefix, and once the 🤖AI has finished writing its reply it looks at whether those lines are in the final reply as they stand. If they are not, the hook prints them itself.

# a shortened version of the code on my machine. pending is the lines remembered, last_message is the AI's final reply,
# squeeze is the function that drops whitespace before comparing (decorate a line and it is a different string)
# after the tool: pick the lines carrying that prefix out of the output and remember them
for line in tool_output.splitlines():
    if line.lstrip().startswith("[observed]"):
        append(line.strip())

# end of the reply: lines missing from the final reply get printed by the hook as they stand
missing = [l for l in pending if squeeze(l) not in squeeze(last_message)]
if missing:
    emit({"systemMessage": "lines the command printed:\n" + "\n".join(missing)})

All that is asked for is that the 🤖AI paste the line, prefix and all, in the final reply. Writing the denominator in as well, or explaining the number, is not part of it.

This mechanism does not stop the summarizing in the replies along the way. Once that line appears at the end, though, the original number enters the conversation there. It goes to the screen as it stands, and it becomes material the 🤖AI uses afterward.

A line the hook printed goes only to my screen. It does not reach the 🤖AI, so the material the 🤖AI works from does not grow.

Putting the prefix down brought out another problem of its own. What dropped the line was not an 🤖AI. It was my wrapper for running the tests. It keeps the whole log in a file and prints only a summary to the screen, and that summary was picking up nothing but the result line. That day, the observation lines that reached the screen came to 0. These days the wrapper picks up the lines carrying that prefix and prints them. What the hook watches is the 🤖AI side alone.

What I stopped reading

Numbers the 🤖AI restated.

Even where the report reads "0," that part goes unread. What I read is the line the command printed.

How to verify: confirm a line appears only after a restatement

Prerequisites

• You have picked one command that prints a number and put a fixed prefix at the head of that line. The denominator goes on the same line as well
• That line reaches the screen when the command runs. From here on, what gets watched is the 🤖AI side alone
• The mechanism that watches for that prefix is registered in 2 places (after the tool = remember the line / end of the reply = compare against the final reply)

Time required: 10 minutes

Steps

1. You ask the 🤖AI to run that command and report the resulting number. You word the request so that it invites a restatement, on purpose
2. You watch whether the line the command printed appears after the 🤖AI's reply
3. You ask for the same thing again, this time having the line pasted as it stands, and watch that nothing appears
4. You ask for the same thing with a command that carries no prefix, and watch that nothing appears

Pass conditions (all of them have to hold)

• At step 2 the line the command printed appears as it stands
• Nothing appears at step 3 (without this, a mechanism that prints the same line twice every time would look like a pass)
• Nothing appears at step 4 (output with no prefix is not being picked up as well)

If it does not pass

• Nothing happens: it is registered on one side only. You need it both after the tool and at the end of the reply
• It appears even after the paste: the line is being decorated. Put it in bold or mix words into it and it is a different string. As it stands means undecorated
• Lines from earlier work appear too: the remembered lines are not being emptied at each ending

Cleanup

• Delete the record of the lines the mechanism remembered (so that old lines are not left over the next time you verify)

Next time, the art of not cutting in. Even when the 🤖AI gets it wrong, I do not point it out. And it still comes out right every time.

Series: The Art of Not Reading

• ← Previous: 9. The art of not reading handoffs
• → Next: 11. The art of not cutting in
• All articles: Introduction: I Barely Read What the 🤖AI Outputs Anymore

End of CC BY 4.0