I no longer check what the AI tells me. Not "the tests pass," not "I confirmed it on the screen." I can stop checking because I put down, ahead of time, the mechanisms that stop verification from flowing to the expensive means. The series "The Art of Not Checking" is the record of building that state one mechanism at a time, and this article is the finale. The mechanism from each article can be reached from the list in the introduction.
There is one thing I have to confess.
I do check, in all 5 articles
At the end of each article there is a section called "How to verify." Every one of those is a record of me breaking it and checking for myself. I opened 2 terminals and hit a 2nd test run, I deleted 1 line of the forbidden-word check from the hook and watched it go straight through, I confirmed that a declaration with nothing inside it stops the screenshot, and I touched 1 file and then watched the date and time on --last stay where it was. On top of that, I hand the reader the same steps every time. A series titled "The Art of Not Checking" ends, in all 5 articles, with steps for checking.
Counting them, the number of times I checked comes to 1 per mechanism. Just the 1 time on the day I put it down, and I have not checked once since.
There is one more shape those sections have in common. The pass conditions are decided before the steps are run. What you hand over, and what has to happen for it to count as a pass — there is not 1 article where I decided that after the run. It was the same shape as what I was doing in the body of the articles.
The 5 mechanisms sit at only 2 entrances
| Article | What I put down | Where I put it |
|---|---|---|
| 1 | The 2 questions before a screenshot, and a ladder from the cheapest | A single sheet of instructions (not an entrance) |
| 2 | A hook that makes the existence of the declaration file a precondition for the run | The entrance for screenshots and smoke tests |
| 3 | --last, which reads the last result back | The entrance for running tests |
| 4 | A lock that lets only 1 run through | The entrance for running tests |
| 5 | Thinning out, and the list of areas that did not run | The entrance for running tests |
I created 2 new files. The remaining 3 were additions to the single wrapper I put down in article 1 of the first series, "The Art of Not Reading." What I added over the 5 articles comes to a few dozen lines in all.
I could decide how to check — the number of times, the number of runs, the scope, the means, the trace — ahead of the run because there was only 1 place to put what I had decided. If the entrance for running tests had been split into 3, then neither the lock nor --last nor the list would work, wherever you put them. As I wrote in the caveats of articles 2 and 4, what those look at is only the runs that came through that entrance. Whether the entrance has been brought down to 1 was the premise of this series.
The part I cut shows up somewhere
I made one promise in the introduction. Every article puts down, alongside the mechanism that cuts checking, 1 place where the part you cut shows up. I will line them up.
| Article | What I stopped doing | What comes out every time instead |
|---|---|---|
| 1 | Not taking screenshots | On the images that remain, 1 sentence saying what the image verified |
| 2 | Not trusting the declaration | The date and time it stopped and the reason, as 1 line in the record |
| 3 | Not re-running tests | The last summary, and the date and time of that run |
| 4 | Not reading false failures | Another run in progress, and the guidance to the way out |
| 5 | Not running everything | The areas that did not run, 22 lines of them |
Not one check has gone away. What went away is the work of deciding the checking on the spot. The right-hand column comes out in front of me without my going to read it. Only what you do not have to go and read is what you can get by without checking.
The "what I stopped doing" lined up in the finale of the previous series came to 31. Add these 5, and it is 36.
The loopholes are all still there
In the caveat of each article I have written that mechanism's blind spot. The one deciding whether it is cheap or expensive is the AI itself, and whether what is inside the 2 questions is right is looked at by no one (article 1). The hook looks only at the form as well, so it goes through if 1 plausible-looking sentence is in there. And what it can pick up is only the entrances whose names are written in the hook, so the 14 wrappers I type day to day have not been looked at once since the day I put them down (article 2). --last is a shortcut only for when you have not changed the code (article 3). Outside the lock there are another clone and CI (article 4). The list only makes a forgotten run visible, it does not prevent it (article 5).
I have closed none of them off. The reason is the same as the starting point of article 1: a program cannot decide whether "this verification can be done with a unit test" is true or false. Trying to put a judgment where it cannot be decided means building one more stage that judges, and that stage ends up being someone's judgment again.
On top of that, the 5 blind spots have something in common. Each of them is only not looking; none of them pretends to look. The hook is set not to look at what is inside, and the list puts out what did not run every time. I meant to put them on the other side of what the introduction said: work that makes checking faster hides what you are not looking at.
The miscount was in the articles
These 5 articles began as an addendum to series 1. When they were made a series of their own, the number of articles in the earlier series moves. The finale of series 3 has that count printed in it. It is the sentence where the total sits in front of "these 3 are the only ones I wrote in the imperative." I wrote up a record of the move, redid all the references, and went over it afterward. Even so, 11 of the printed numbers were left as the old ones. I was about to hand the reader a scale that does not exist.
I did not find it by reading back over them. I found it when I put down an automated test that counts from the articles themselves and matches that against the printed numbers. It was the same shape as article 5. What has not been counted is put in front of you every time.
Around the same time, I got something wrong once more.
In the caveat of article 2, I first wrote that I had forgotten to add to the hook an entrance I added later. Looking at git it was the other way around: the entrance was there first, and the hook had not looked at it once since the day I put it down. I counted after writing it, so it was fixed before it went into the article.
The conclusion
The art of not checking is the art of not deciding how to check on the spot.
Checking looks like a choice between doing it and not doing it. Underneath that hang when, how many times, how far, by what means, and what trace to leave, and the moment you answer yes or no, the AI that happens to be there fills them in.
How it gets filled in differs every time, and I cannot notice that it changed.
So I moved the place where it gets decided to before the run. Where I moved it to is 2 entrances, and what I put down is a single sheet of instructions and 1 hook, and the rest is additions to the wrapper. If how to check is decided first, you can get by without reading the result of the check.
Even so, the days when you do check remain. They are the days you put a mechanism down.
Break it, and please confirm just once that what should stop stops and what should pass passes.
If you write the pass conditions first and then run, that does not become "checking decided on the spot" either.
In the introduction I asked you to count the number of times you ran something in your most recent session in order to check. Of those, how many had how far you would go decided before you ran them?
Answering the same question now, on my own machine it comes to "all of the ones that ran." That is because anything not decided stops before it starts running.
For how to place the mechanism from each article, start from the list in the introduction. What was built with these mechanisms is running at typingtube. The screens I built with the time I got back from not checking are lined up there.