Five ways my own test harness lied to me, and the one that nearly got published
The number that gave it away was sitting in a table I had already read three times.
prompt size 97 tokens median, 112 max
That is the grounded question-answering workflow. It sends four reference documents with every question. The documents are 447 words.
Four hundred and forty-seven words cannot be ninety-seven tokens.
I had written that number into a results document as good news about context headroom. “Prompt sizes never approached the 4096-token context window.” It was not good news about headroom. It was the harness telling me, in the only field capable of telling me, that the reference material was not in the prompt. The directory holding it had never been committed to the repository. Every one of those fifteen questions had been asked against an empty reference block.
Both models had behaved impeccably. One had been instructed to answer NOT IN THE MATERIAL when the material did not contain the answer, and the material contained nothing, so it said NOT IN THE MATERIAL thirteen times out of fifteen. The other had no such instruction, so it answered from whatever it had memorized during training and invented a wine.
Every call returned 200. The scorer scored it. The scores were internally consistent, had plausible and different failure modes on each side, and I wrote two thousand seven hundred words about what they revealed regarding the capabilities of small language models.
They revealed nothing about small language models. I had measured what happens when you ask a question with no reference material, which is a question I was not asking and which has an uninteresting answer.
This is the post about that, and about the four other things I found in the same weekend, because the missing directory was not the first one and it was not even the most instructive.
The experiment, briefly
A Raspberry Pi 4, two small models through Ollama, and three workflows: pull an identifier out of a message, route a message to one of six queues, answer a question from four documents. Ninety-five test cases. A deterministic baseline for the first workflow. Predictions committed before anything was installed.
The findings are a separate post. This one is about the code I wrote to produce them, which turned out to be where the actual bugs were.
Five flaws. Every single one was found by reading output, not by reading code. I never once found one of these by looking at the scorer.
Kind one: a metric that rewards the safe failure
This one I hit twice, in two different workflows, and did not recognize the second instance as the same bug until I had already fixed the first.
The classification scorer. Six labels, one of which is escalate. The instruction said to escalate when a message needs a person, including when you cannot tell what it is about. So I built a metric called confident wrong: wrong, and not an escalation. The reasoning is sound. A model that escalates when unsure is usable. A model that routes a trademark notice to billing is not.
Then the 360M model answered escalate to 43 of 50 messages.
It scored 3 on confident wrong. The 1.5B model, which was genuinely attempting the task and getting 46% of it right, scored 26.
My metric had just certified the useless model as roughly six times safer than the mediocre one. Because escalations are exempt, a model that escalates always cannot register a confident error. The exemption I added to reward caution had instead rewarded paralysis, and it did so in the direction that looks best.
The grounded QA scorer. Four of the fifteen questions have no answer in the corpus. Correctly saying so is the behavior that matters most, because a system that invents a plausible answer with no evidence is worse than one that fails loudly: it is indistinguishable from one that worked. So the headline number was correct abstentions out of four.
Then the 1.5B model said NOT IN THE MATERIAL to 13 of 15 questions.
It scored 4 out of 4. A perfect score, on the metric I cared most about, for a model that had answered almost nothing.
Same bug. Different workflow. Two hours apart. I fixed the classification one, wrote a comment in the code explaining the principle, and then failed to apply that principle to the file I opened next.
The shape, stated once: a metric that rewards the safe behavior will certify any model that only ever does the safe thing. If you exempt a behavior because it is cautious, you have to pair that exemption with evidence that the model is making distinctions at all. Otherwise, the exemption is the score.
Both fixes are the same three lines: count how concentrated the answers are, and when one answer dominates, print a warning that the headline number is meaningless and say why.
Kind two: a metric that measures nothing at all
The vacuous fixture. I caught this one before the run, which is the only reason it is not a bigger part of this post. A test case that passed regardless of what the model did, because the fixture was inert either way. It contributed a tick to a column and no information.
The keyword check, which is worse, because it survived longer and looked more useful.
The eleven answerable QA questions each carry a list of words the correct answer should contain. The scorer checks whether any of them appear anywhere in the reply. I labeled it “indicative only” and moved on.
On the invalid run it scored 1 of 11 for both models. The single hit was the word “reservation”, appearing inside this answer:
You owe a tasting fee of $100 for the weekend. You also need to book a reservation for the weekend.
The correct fee is forty dollars. The hundred is invented. That one hit was the entire positive signal in my automated QA scoring and it was noise.
Here is the part that makes it a better story. After I fixed the corpus and reran, the keyword check scored 11 of 11 for the 1.5B model. Vindicated, apparently. I read all eleven answers by hand anyway, and nine of them are right.
So the check was wrong both times. It was wrong in the pessimistic direction when the answers were bad and wrong in the optimistic direction when they were good, and on the rerun one of its eleven hits is an answer that gets the arithmetic wrong and hits on the word “reservation” anyway.
A check that agrees with the truth because the truth happened to be agreeable has not been validated. It has been lucky.
Kind three: a detector with a blind spot in the obvious direction
I wrote a decay check because of a thermal problem. A Pi 4 with heat sinks and no fan hit its soft temperature limit during a thirty-call run and started capping the clock, and nothing anywhere said so. So the scorer now compares generation speed in the first third of a run to the last third and flags a slowdown.
It then reported this, as steady:
across the run 4.3 -> 7.6 tok/s (176% of starting speed, steady)
The hardware got seventy-six percent faster partway through a run. My detector, which was looking for numbers below a threshold, saw a number above it and reported no problem.
Hardware does not speed up. That reading is either contention at the start of the run or, as it turned out, generations too short to time. Either way it is a signal that the measurement is unreliable, and I had built a detector that would only ever notice half of the ways that can happen.
Anything you threshold in one direction, threshold in both. An implausibly good number is as informative as an implausibly bad one and nobody ever writes the check for it.
Kind four: errors that get dropped instead of counted
Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty. The traceback was useless because urllib raises HTTPError and discards the response body, which is where Ollama puts its reason. Reading e.read() gave it up immediately:
{"error":"prediction aborted, token repeat limit reached"}
The 360M model, asked to find an order number in “Thanks for a lovely afternoon yesterday,” where the correct answer is one word, had fallen into a repetition loop and been killed after twenty-one seconds. Reproducible at temperature zero. The input causes it.
My first fix was to record failures and keep going, which was right. My second was to exclude all failures from the scores as infrastructure noise, which was badly wrong.
That call was not infrastructure noise. The model ran. It received the prompt, generated tokens, and produced nothing usable. That is a result, and excluding it would have dropped from the denominator the single case this model demonstrably could not handle, scoring it only on the cases it survived.
So the runner now tags every failure as one of two kinds:
- Model failure. The call reached a working model and the model failed. Stays in the denominator. Counts against the model. Reported on its own line as “no usable answer.”
- Transport failure. The call never reached a working model. Ollama was down, the tag was wrong, the request timed out. Nothing was measured, so it is excluded and the run is reported as incomplete.
The distinction is a five-line change and it is the difference between measuring a model and measuring a model on the subset of inputs it happens to tolerate.
Kind five: inputs that go missing without raising
This is the missing corpus from the top of the post, and it is a different and worse category than everything above it.
The function was four lines:
def corpus_text():
parts = []
for p in sorted((FIX / "corpus").glob("*.md")):
parts.append(p.read_text(encoding="utf-8").strip())
return "\n\n".join(parts)
A glob that matches nothing yields nothing. An empty list joins to the empty string. No exception, no warning, no exit code. It returns a perfectly valid value that happens to be the absence of the thing it was asked for.
And on the same evening, a sibling. The offline test’s entire product is a text transcript. I copied it into the run directory, committed, pushed, and it was not there. The standard Python .gitignore template, which I had pulled in wholesale, carries *.log on line 165, and a pattern without a leading slash matches at any depth.
The copy succeeded. git status came back clean. The push reported success. The one artifact the test exists to produce was gone, and everything in the chain said fine.
Absence does not raise. A missing directory, a missing file and an inert fixture all present as an ordinary successful run. That is narrower and more useful than “test your tests,” and it generalizes: every input a harness depends on needs a loader that refuses to proceed without it, and every artifact it produces needs to be verified as present rather than inferred from an exit status.
The loader now looks like this, and the comment is longer than the code because the comment is the part I will need next year:
paths = sorted(corpus_dir.glob("*.md"))
if not paths:
raise SystemExit(
f"No corpus documents found in {corpus_dir}\n"
"Grounded QA cannot run without reference material, and running\n"
"it against an empty reference block produces numbers that look\n"
"like a result."
)
It also refuses a corpus implausibly smaller than the real one, the run prints the document count and token estimate before its first call, and the scorer independently flags any QA run whose median prompt is under 400 tokens. Three checks for one bug, in the input, in the run, and in the output, because this is the bug that cost a day.
The detector that fired, and was right, and told me the wrong thing
Here is the part I find hardest to write and most worth keeping.
The abstention-collapse check from kind one did fire on the invalid run. It printed exactly this:
!! ABSTAINED on 13/15 questions overall. It is refusing, not judging.
The abstention score below is not evidence of discrimination:
a model that never answers gets it right for free.
It was correct. Something was badly wrong with that run.
Its diagnosis was completely false. The model was not refusing. It was obeying, precisely, an instruction to report when the material lacks the answer, in a situation where the material lacked everything. The fault was not in the model’s judgment, and it was not in the metric. It was in a directory that did not exist, two layers upstream.
And I acted on the diagnosis. I wrote it up: the model has a stuck needle, its perfect abstention score is an artifact of refusal, here are the implications for anyone building on a 1.5B model. Confident, specific, sourced to real output, and wrong in every particular.
A flag that tells you something is off is not a flag that tells you what. The check could only see the output. The fault was in the input. It fired for the right reason and pointed in the wrong direction, and because it fired I felt more confident in a conclusion that was false, not less.
That is the failure mode I had no defense against and still only partly do. A monitoring system that is right about the existence of a problem and wrong about its cause is more dangerous than no monitoring, because it terminates the search.
What I actually changed
For anyone who wants the diffs rather than the moral:
- Collapse checks on both classification and QA: when one answer exceeds half the set, print the concentration and declare the headline metric meaningless for that model.
- The decay check flags implausible speedups as well as slowdowns, and notes when generations are too short to time.
- Failures split into model and transport, with model failures staying in the denominator.
corpus_text()raises on an empty or implausibly small corpus.- The QA run prints its corpus size before the first call.
- The scorer flags a QA run whose prompts are too small to contain the corpus.
- The keyword check is relabelled as a weak lower bound, in the output, in the README, and in the results.
.gitignoreno longer lets*.logswallow run transcripts.vcgencmd get_throttledbefore and after every run, recorded in the run document rather than checked and forgotten. The reading reports bits set since boot, so it is perishable: power-cycle the board and the evidence is gone.
The uncomfortable part
Every one of these was found by looking at data. Not one was found by reading code, reviewing the scorer, or writing a unit test for the harness.
The vacuous fixture turned up while hand-checking a passing case. The confident-wrong flaw turned up because a score of 3 against 26 looked backwards. The abstention flaw turned up because 4 out of 4 looked too good. The keyword flaw turned up because 1 of 11 looked too bad, and I read the one. The missing corpus turned up because I was double-checking a latency figure and noticed a prompt that was too small to be possible.
Which means the only reason I found them is that I kept pulling threads after I had numbers that looked publishable. The version of this project that shipped on schedule shipped on Tuesday, with a clean verdict table, a confident conclusion about small models and grounded question answering, and a repository anyone could clone to reproduce the exact same wrong result.
Every number in it would have been reproducible. That is the thing I keep turning over. Reproducibility is table stakes and it would not have saved me, because the harness reproduced its own bug faithfully every single time.
I do not have a process that catches this reliably. What I have is a weaker and more honest claim: the checks I added are each a direct response to one specific thing that actually happened, which is the only kind of check I trust, and the number of them that I thought up in advance is zero.
So the two rows I am adding to my prediction template, to be filled in before anything runs:
- What would have to be true for these numbers to be meaningless?
- Would anything in the harness tell me if it were?
Twenty-two predictions about what the models would do. Not one about what the measuring code would do. Four of the five flaws were in the measuring code.
The harness, the fixtures, the committed predictions and the full run record, including the invalid run and what it produced, are at
github.com/kenwalger/right-size-the-model.
The findings the harness eventually produced are in the companion post.
