the cheapest way to read a judge

the task was easy to describe and expensive to do. take a pile of information technology act judgments and work out which way the judge was leaning.

the obvious approach is to hand every judgment to a good model and read the answer back. that works. it also costs more than the project is worth once the backfill is the whole corpus, and it is slow in a way you start to feel around the third day.

so it went in three tiers. regex first, because a surprising number of judgments announce themselves plainly in the operative paragraph, and a regular expression costs nothing at all. whatever regex could not read went to gemini flash, which is cheap and perfectly good for the middle of the distribution. only the genuinely ambiguous remainder went to claude haiku.

the backfill came out roughly ninety nine percent cheaper. accuracy barely moved, because the expensive model had never been the thing doing most of the work. it had been doing all of the work, which is a different problem wearing the same clothes.

i think about that more than i expected to. the instinct, with a good model sitting right there, is to send it everything. most of the time the better move is to send it only the part that is actually hard, and to be honest with yourself about how small that part usually is.