The easiest person to fool with an LLM benchmark is the person who trained the model. You see the numbers, you tweak the threshold, you call it a pass. So we took the option away from ourselves.
Bars before runs
Every training run on Project M&M has its pass marks written into the results file, dated, before training starts. A bar is a single number for a single exam. All of them must pass. One miss and the model is not served, and the fail is published as a fail.
Our eight bars for a reader:
| Exam | What it asks | Bar |
|---|---|---|
| Hand-written 200 | 200 analyst questions written by a person, with gold answers | ≥ 190 |
| Blind refusal | answer removed from the dossier; must say "The dossier does not say" | ≥ 0.95 |
| Indirect | answer present under an alias, case or another label | ≥ 0.65 |
| Leak | names something that is not in the dossier at all | ≤ 1.0 % |
| Held-out recall | entities the training never saw | ≥ 0.80 |
| Hard set | 100 harder questions: two-part answers, ordering, counts by date, year traps | ≥ 85 |
| Sector exam | which sectors a ransomware crew hit, and when | ≥ 0.98 |
| Vulnerability exam | CVSS score, vector, weakness, fix | ≥ 0.98 |
A rule-based judge in code scores every answer, identically for every model. No model grades another model's exam.
Amendments only before results
Plans meet reality: a GPU is out of stock, a run is slower than budgeted. We allow amendments, but only before any result exists, dated, with the reason. On run 17 we cut two epochs to one before training, because two would have broken the time cap; the bars stayed unchanged. On W1, the card we had pre-registered was briefly out of stock, so we wrote down the fallback cards, in order, before anything ran.
Cold audits and frozen data
Training data is built from the graph, never typed by a model from memory. Before a set is used:
- Two separate AI auditors read samples cold, with no context, and list every defect. Any defect class fails the set.
- We fix what they found and audit again. The last set went through five rounds, two different AI models auditing in every round.
- The passing file gets a fingerprint (md5). The run script checks it before training. The file trained is provably the file audited.
Exam subjects are banned from training data, so the model cannot have seen the answers.
Three times it caught us
The leaky first result (23 October). Our first probe set scored beautifully. Then we read it per relation and found two relation types at 98 % and 87 % because the answer string contained part of the question. We filtered every probe whose answer shared a distinctive token with its subject, and quoted only the filtered set. It cost seven points. It is the only number worth having.
The gate that measured nothing (1 November). A mid-run quality check on run 17 reported 0.0 on everything at step 206, with training loss near 0.01. Not a broken model: every prediction decoded to an empty thinking tag, and the gate cut at the first blank line. We wrote the amendment, turned the gate off, and resumed from a checkpoint before the gate's automatic kill fired.
The half-sized reading window (3 November). The live test of our best reader died on server errors. The script, as written, scored the missing file as a fail and switched production back to the CPU model. The cause was ours: the card served 8k tokens per request while the service assumed 16k. We fixed it, re-ran, and published both runs, the no-verdict one included.
Why bother
Because the result we are proudest of is a fail. The moonshot taught a model 358,000 facts and broke its ability to say "I don't know". Without bars written in advance, it would have been easy to celebrate the half of the facts it learned.
Seventeen runs. Most stayed on the shelf. Every one of them is on the record: the scoreboard, every question and miss, the live build log.