An interview instead of a template. The spec came out with no gaps. And the threshold in the wrong place

Part five of the series on LLM-as-a-Judge, that is, on one model grading the work of others. In part three I gave two AI agents a task described in one sentence and got back code full of decisions nobody had made. In part four I described the same task properly, to a template, and those decisions vanished from the code in eight solutions out of eight.

Except that those twenty-odd lines of template took me longer to write than the agents took to write the code. And I had a crib sheet: part three had listed every trap for me, so I knew what to ask myself. In a real task queue nobody gets a crib sheet. So I promised to check whether the questions could be asked by a model. This is that check.

Thesis

Two, because the second follows from the first.

First. The questions already exist. In part three, opus named the ambiguities in the task in every one of its eight runs — “the spec says discounts do not stack but doesn’t say which wins”. It just did so in its last message, when the code was already written, and nobody read it. If the same questions are asked before the agents start, and I answer each with one sentence, then a spec built from my answers should close the same decisions the hand-written template closed in part four.

Second. If that works, I still will not know what worked. Maybe the value sits in the asking itself, and any spec written by a model would be just as good. Or maybe in the fact that the one answering has the right to decide. To separate the two I need a control: the same interview, but with no human in it.

The study

Again I change one thing: where the task description comes from. Agents, judge, base commit, round limits stay as in part three and four. I leave the known defects of the pipeline alone too, though they itch. Neither the list of allowed commands, which stops opus from running tests, nor the CI gate that reads “don’t know” as “yes”. If I changed two things at once, I would not know which one worked.

What is new is one stage, at the very start. The judge model, the same one as always, with no tools and no access to the repository, gets a one-sentence task and the list of template fields. It is to ask only questions the code depends on: where the boundary sits, what type the data is, what the function returns, what it does on bad input, how to read a phrase that can be read two ways. Twelve questions at most.

Next to each question it writes two things. What in the code depends on the answer. And which answer it would assume if nobody answered. Remember that second field, because the whole experiment rests on it.

Then two arms, with the same sentence about a discount as in the two previous parts:

a function that calculates a discount: 10% above 500 PLN,
20% above 1000 PLN, discounts do not stack with promo codes

Arm C. The owner of the decisions answers the questions, which is me. That was the plan. And here is the first change to the protocol, recorded before the first answer was given.

I saw nine questions in the panel and did not want to answer them. Not because they were bad. They were good. But I had already made every one of those decisions once, writing the spec for part four, and this whole pipeline is supposed to work with as little human involvement as possible. Answering the same thing a second time looked like a denial of the whole idea.

So the model that helps me write this blog answered for me. The rule was one. Where my part-four spec settles the matter, the answer is my decision in its words. Where it is silent, the assistant takes the interviewer’s default or writes “you decide”, the way I do in the OPEN DECISIONS field. It was not allowed to add anything of its own, because then there would be no way to measure what the questions closed and what the assistant’s own defaults did.

The panel counts how many answers were mine, how many accepted defaults, how many “you decide”. It does not count human time, because no human was there.

Arm D. Nobody answers. The model’s default assumptions, the ones from the second field, serve as the answers. The model interviewed itself.

In both arms a spec is built from the answers to my template, with acceptance criteria the judge takes word for word as its checklist. That is exactly how it was in part four, in arm B, with the hand-written template. The spec is frozen. I cannot touch it up, because if I could, it would be arm B again.

Four runs per arm, two agents in each. Sixteen solutions.

What do I count? Above all, coverage. For the discount task I have, from the two previous parts, a list of ten decisions the one sentence leaves open. Six from part three: where exactly the threshold sits, what to do with an empty code, what to do with a bad amount or an unknown code, how to count money, what the function returns, what “do not stack” means. And three edge cases from part four: round before the threshold test or after, the list of codes passed as a plain string, True as an amount.

A question “covers” a decision if a one-sentence answer to it settles that decision. I score this at answering time, in the notes in the panel, before any code exists. Otherwise I would fit the score to the outcome, and I do not want that, even if I would not notice it.

The rest as in part four. I read the diffs and summaries and check which decisions the code made silently. I feed every solution the same three unusual inputs. I look at the judge’s verdicts and make my own decision. In arm D one extra column: where the model’s default differs from mine.

There is also a second block, because with the discount function we all know the answers by now, you too, if you have read the previous parts. So I take a fresh task, also one sentence, which nobody here has taken apart: a function that parses a duration written as text, like “2h 30m” or “45 min”, and returns the number of seconds. I was supposed to write down, before the questions, which decisions that sentence leaves open. I did not. That measurement is off. A simpler one remains: the same sentence once without an interview, once with an interview in which the model answers itself. Then each of the eight parsers gets the same twenty-one unusual inputs.

Two things changed along the way. Both touch the constants, so I have to name them.

First: after the second failed attempt at C-1 I fixed the worker’s list of allowed commands. Through all of part four opus could not run python -m pytest and handed in tests it had never run. This time, with a spec of eleven criteria, it tried twenty-two times and died on the turn limit. A constant that kills runs stops being a constant. But from that moment worker A can run tests, which it could not in the two previous parts. If it now has fewer defects, that must not be credited to the interview.

Second: every claude call in the pipeline loads my global config file. The day before, I had added there that conversation with me is in Polish. Seven of the eight discount interviews came back in Polish. Worker A signed off in Polish. The code and the specs stayed English. Nobody chose this. I am leaving it in view.

I wrote the protocol and the predictions before the first run, as before. But this time I made no commit. The files reached the repository only after the results, already scored (2e75e77), so the repository history does not vouch for the order. What remains is my word and the dates in the registry. The previous two parts had proof here. This one has a declaration. I would rather write that than pretend.

Result

The K codes in the table are the six classes of decisions from part three, in the order of the paragraph above; the E codes are the three edge cases from part four. The table keeps the codes, the text speaks plainly.

The first thesis holds. A spec built from answers closed in the code the same six classes the hand-written template closed in part four. Zero silent decisions in eight solutions out of eight. Both branches ready first time in every run. Zero repair rounds. Nobody wrote a line of spec.

Does it save the time I opened this post with? I do not know. I did not measure it, because no human sat at the panel, only a model transcribing my decisions. Honestly: I switched that measurement off myself.

The predictions that fell, I leave in view. I bet the judge would call a tie in at least three runs out of four, as in part four. It did not call one once. In three runs its two passes, once with A first, once with B first, disagreed with each other and the system returned the verdict “inconclusive”. More on that below. I also bet that a question about an empty promo code would come up in at least two interviews. It came up in none.

What the questions did not ask

Four questions came back in every one of the eight discount interviews: where exactly the threshold sits, what the function returns, how to count money, and what “do not stack” means. Exactly the ones opus listed in part three in its last message. The question about a negative or zero amount also came up every time. These are things you can see in the sentence itself.

Nobody asked about an empty promo code. And yet in arm C none of the eight solutions made that decision silently. How? It was closed by my answer to a different question, about how the code reaches the function: an optional string, which counts only if it is on the list of valid codes. No code is not on the list, so the threshold stays. All eight read it that way, and C-1 wrote it out: “a missing or unlisted code falls through to the threshold logic”.

One answer closed two decisions. That is the answer’s doing, not the question’s. And here my coverage measure showed that it counts the wrong thing. I wrote down that a question covers a decision if the answer to it settles the decision, and then I counted questions. The value sat in the answers, so coverage is understated. It is the same thesis as this whole post, only it came out of my own table. In arm D that decision does not exist at all, because there is no code parameter there. I will come back to that.

Nobody asked the three edge cases from part four. Nobody asked whether to round before the threshold test or after. Everyone asked how to round, and every one of the sixteen candidates rounds after, so an order of 999.995 zł gets 10%. Nobody asked what to do when someone passes the list of valid codes as a plain string. Grok added a type check and an exception on its own in two runs out of four, opus fell into the substring trap in four out of four. Nobody asked about True as an amount. Six interviews asked about a “non-number”, and that was enough for all eight solutions in arm C to reject it, although in two the exception is raw, from inside the library, not from validation.

Those three cases came out in part four from testing finished code. They do not come out of reading the sentence. Not for a human, not for a model.

What the interview did ask were questions none of us had asked in four posts. In eight out of eight: is the rate flat, or progressive by bracket, as in a tax table. For 1,500 zł that is a discount of 300 or of 150, and in four posts I had not noticed that my sentence does not say. In three: should the threshold be measured on the amount before the code or after. In two: is the input a single amount or a cart to be summed. One each: what if the promotion is bigger than the order; do 10% and 20% add up to 30%; which language the function should be written in.

The model that interviewed itself

Here is the heart of it. In arm D nobody answered. The interviewer’s default assumptions, the ones from the second field, were taken as the answers. On the code side the result looks identical to C: zero silent decisions, a full set of criteria, ready first time. The specs were closed. For the first time in this series the OPEN DECISIONS field was empty in four runs out of four.

Only closed not my way.

In four runs out of four an order of exactly 1,000 zł gets 10%, because the model reads “above” as strict, and in part four I decided the opposite. In three out of four the function returns the discount amount, not the amount payable, because the sentence says “calculates a discount”. In two runs out of four “do not stack” suddenly meant “the better one wins”, not “the code disables the threshold”. Same model, same prompt, same sentence, and a different default answer.

And a fourth thing, different from those three. In four runs out of four the promo code vanished from the function’s signature altogether. Twice a “code used” flag took its place, twice a ready discount amount in złoty. The function that in my version checked whether the code was on the list of valid ones, in the model’s version “does not know the code’s value and does not validate it”. Those three were wrong settlements of a decision that is in the sentence. This one is a change to the contract that nobody ordered: the sentence talks about promo codes, and the function does not accept them. Of everything that came out of D, this is the most dangerous, because in the spec it looks the tidiest. The criterion has a number, the judge ticked it, the test passes.

This is the result I ran arm D for. The interview removes silence: every decision is written down, nothing sits in a docstring. It does not remove error. A written decision is the decision of whoever answered, and if nobody answered, it is the model’s guess. The same guess as in part three, only moved from the code into the spec and given a criterion number. The judge then checked it diligently and ticked everything, because ticking does not inject missing decisions. I wrote that in part three about the checklist. Now it applies to the whole spec.

One thing from D I did not expect. Where the default answer introduced a discount amount from the code, a new decision nobody had asked about appeared at once: what if the discount is bigger than the order. In one run both solutions return a 700 zł discount on a 400 zł order. In the next, one clamps to zero, the other returns minus 300 zł, and that is exactly what the judge decided on. Closing one set of decisions opens the next, one floor down. The same observation as in part four, only now the spec itself adds the floor.

The judge that stopped agreeing with itself

The judge compares two solutions twice, once with A first, once with B first, so that it does not favour whichever one it saw at the start. In part four both passes agreed in four runs out of four, and four times it was a tie.

Here there were twelve runs, the eight discount runs from the table above and four from the second block. In six of them the passes disagreed: once A, once tie; once tie, once B. The “inconclusive” verdict is the fuse from part two. It worked exactly as it should and six times sent the decision back to a human. The human did not make it, but mechanically took branch A and entered “no preference”. That was in the protocol.

The reason is visible in the rationales. The solutions met every criterion in full, so the judge looked for differences outside the checklist. A separate module or an addition to an existing file. Fuller tests. An error message. One forgotten scratch file, which opus committed with a comment saying it had been left uncommitted, because it had no permission to delete it, so it left a note, and then added the note to the commit along with the file. In one pass the judge weighed those trifles one way, in the other another.

The better the spec, the less the judge has to do and the more noise it ranks. I wrote about that in part three, when the candidates were contaminated by cheating. Here they were not contaminated, they were simply equally good. The pairwise comparison usually cost three quarters of the judge’s output tokens and named a winner in five runs out of twelve. In one, the last of the second block, both passes agreed on a tie.

The task nobody knew

The duration parser. Without an interview the judge made itself a four-item checklist, two of the items being the examples from the sentence. Four solutions passed it in full and came apart on three of the twenty-one unusual inputs. 2.5h is an error for opus and 9,000 seconds for grok. 1w is an error for opus and a week for grok. 2h and 30m passes once and fails once. Opus, in both runs, listed at the end what it had decided on its own. Grok listed nothing. Nobody read it, as in part three. The judge was inconclusive in both runs.

With an interview in which the model answered itself, twelve questions were asked. Every one of them was one of the decisions the four parsers had made silently. Thirteen, fourteen criteria. The two solutions in each run agree on all nineteen text inputs, including the three that split the raw parsers.

The only decision nobody asked about is a non-string input: opus in both runs throws a raw AttributeError from inside .strip(). And one default answer flipped between runs again. A repeated unit, 1h 1h, sums once and is an error once. Should a week be an error, or 604,800 seconds? Nobody knows, because this task has no owner. It is written down. It is not settled. Verdicts: once B in both passes, once a tie in both. The only agreed tie in this post.

Caveats. Twelve runs, two toy tasks, one pair of models, one day. Decisions after the verdict made mechanically, so the counter of judge agreement with the human from part two is empty in this post. The target repository from arms A and B had disappeared from my disk before the first run started. I rebuilt it from the patches, byte for byte for both source files, the README is a stand-in, the base commit is different. The first attempt at C-1 failed because there was no repository. The second because grok’s token had expired and it hung at login, while opus meanwhile died on the turn limit, hence the fix to the list of allowed commands described above. Questions in Polish, because my own config file said so. The raw artefacts of all twelve runs, the protocol with its changes and the scored predictions are in the repository.

Counting. In parts three and four I gave costs in dollars. Those numbers came from the CLI’s meter, and nobody paid them, because the agents and the judge run on subscriptions, not on the API. From this part on I give tokens: output tokens separately, because that is the model’s real work, and context tokens separately. I am not correcting the dollars in the earlier posts. They stay as what they were: an estimate from a meter.

The point

The twenty-odd lines of template from part four can be replaced by eight questions and eight one-sentence answers. The code comes out just as closed. Except that those answers have to come from someone who knows what it is about. When the model answers, you get a spec without a single gap, in which the threshold is in the wrong place, the function returns the wrong thing, and the promo code has turned into a flag. All written down, all ticked, all green.

In part three I wrote that this is not a problem of context but of authority. This part adds one sentence: a machine can ask the questions, and from the sentence alone it asks them better than I do. The answers have to come from someone who made those decisions. In this experiment nobody answered live. A model transcribed my decisions from the part-four spec, and that was enough. The human part is not sitting at the panel. It is that the decisions were made somewhere, at some point; who transcribes them does not matter. That part does not shrink. It moves to the very beginning.

What next

Three things, each as a separate change.

A judge without the pairwise comparison when both are ready. Six inconclusive verdicts out of twelve, with candidates meeting the full set of criteria, is a signal that the comparison ranks noise. The next experiment: both ready, both identical on the checklist, then we pass the one cheaper in tokens, or the first, and keep the comparison for the cases where the judge’s first pass showed a difference. The “accept tie” action in the panel, announced in part three, goes in with this.

An interview that also asks about what the sentence does not show. The three edge cases from part four and the non-string input from the second block have one thing in common: they come from feeding the code bad data, not from reading the spec. A candidate to test: a second round of questions after the first diff, with the code in view, before the judge starts.

Tasks that cannot be closed. On twenty lines of code the judge had nothing to judge, in part four and here. A bigger task, where the solutions genuinely differ, will show whether the pairwise comparison is worth anything, or is just an expensive ritual from part two.