Everyone building a domain model eventually asks the same question: can the model just know our data? We asked it twice, on purpose, and published both answers.
Attempt one: from scratch, three sizes (23–25 October)
We trained our own small transformers from zero on a corpus built from our threat graph: 746,000 documents and 299 million tokens, half of it graph facts written as dated sentences. Then we asked a question most people skip: how well does a model recall facts it was trained on? Not held-out facts. Seen ones.
After two passes, our 34-million-parameter model recalled its own training facts at 45 %, barely above its held-out score. It was learning the dialect and a sense of what is plausible, and guessing. So we trained harder and ran a scaling curve, same recipe, three sizes:
| Parameters | Seen-fact recall | Held-out | Final train loss |
|---|---|---|---|
| 34M | 53.7 % | 43.7 % | 0.30 |
| 110M | 46.3 % | 39.5 % | 0.10 |
| 320M | 43.2 % | 37.8 % | 0.09 |
Bigger was worse. The 320M model had the lowest training loss and the best English, and the poorest recall of a single fact. It had learned to continue a long dossier, and the test asked for one fact from a bare name. Those are different skills.
Let the model read instead
So the facts stayed in the graph. At question time we fetch the entity's dossier and put it in front of the model. Same open 3B model, same 938 held-out questions, four options each (chance 25 %):
| Condition | Accuracy |
|---|---|
| Closed book | 32.5 % |
| Dossier, answering line removed | 45.5 % |
| Dossier in context | 76.8 % |
Seven points above chance to 77 % the moment the fact is in front of it. That is the architecture we kept: the graph holds the facts, the model only reads. (The full write-up of those first days.)
Attempt two: the moonshot (30–31 October)
A week later, with bigger models and better data, we tried once more, deliberately. One run, about $35, trained a 20-billion-parameter model to memorise 358,000 facts from the graph and answer with no dossier at all. For anything it had never been taught, it was trained to say a fixed phrase ("Nah. WTF. Nada.") instead of guessing.
The pass marks were written before the run: recall on 2,000 rephrased questions about taught facts, and refusal on 1,000 entities it was never taught.
- Taught facts: it worked. Recall went from 0.013 to 0.508. It learned half of them.
- Unknown things: it broke. The untrained model admitted ignorance 87.6 % of the time. After training, it invented an answer for 99.4 % of things it had never seen.
Verdict, as written before the run: fail.
Why this matters for threat intelligence
A confident invention is worse than silence. An analyst can work around "I don't know". An analyst cannot work around an invented ransomware victim or a CVE score that looks right. When we taught the model to answer from memory, we taught it that answering is what it does, and the "I don't know" habit was the first thing to go.
The graph also changes every hour. A fact in the weights is frozen at training time; a fact in the graph is current the moment the feed lands. Retrieval is not a workaround here. It is the design:
- Facts live in the graph, each with a date and a source.
- The model reads a dossier and quotes from it.
- Refusal is exact ("The dossier does not say") and measured on every run.
- Every answer carries a receipt: the dossier lines it used.
What we train now is the reading, not the remembering. How one sentence did more for that than seventeen fine-tunes.