Two days, a 34-million-parameter model, a 3-billion-parameter reader, under $30 of rented GPU, and the numbers we are willing to put our name to.
Why build one at all
Every security vendor now has "an AI". Almost all of them are the same few foundation models with a prompt in front, trained on the same public text: CVE descriptions, ATT&CK, vendor blogs, the open web. That text is in every model already. It is not a moat, and it is not yours.
We run a threat-intelligence platform with a graph at the centre of it: about 11.9 million nodes and 67 million relationships when you count everything, of which the part that is actually threat intelligence is a few hundred thousand entities and a few million edges with dates on them. Ransomware crews and who they claimed, when. CVEs and the products they touch. Malware families and the hashes, domains and URLs seen distributing them. Our own dark-web crawl: forum posts, leak-site claims, crew names, aliases. Our own briefs and verdicts, in our own vocabulary.
The question we set out to answer on 23 October was simple to state: can we train a model on that, on no hardware we do not already own or rent by the hour, and publish numbers honest enough to survive a sceptic?
There was a second motive. The teams who most need a model like this are the teams who cannot send their data to anyone: defence-adjacent, critical infrastructure, regulated finance, managed security providers paying per token to triage somebody else's alerts. If we could do it on our own intelligence, on a CPU box, with the evaluation in the open, we could do it on theirs, inside their walls. We are customer zero.
What "our own" means
Two things share one corpus and we built both.
Ninja Cortex proper is a decoder-only transformer written from scratch in PyTorch (RMSNorm, rotary embeddings, SwiGLU, tied embeddings), with its own byte-level BPE tokenizer trained on our text (16,384 tokens, frozen after the first corpus so every later run is comparable). Presets from 0.6M to 320M parameters. The 34M "small" is the workhorse.
The fine-tune track takes an open base, Qwen 2.5 at 1.5B and 3B, and adapts it with LoRA on the same corpus. Our data, their network.
The corpus is where the work was. Version 3, the one every later number rests on, is 746,000 documents and 299 million training tokens. Just over half of it by characters is the verbalised graph: every edge rendered as a dated sentence in a fixed dialect (On 2026-02-16, Mozi (Software) distributes http://… (Indicator, url).), grouped into 159,538 entity dossiers so a model reading about LockBit reads everything we hold on LockBit in one place. Then the dark-web layer from our crawl, our own published analyses, and public reference material (NVD, ATT&CK with its relationships, CWE, CAPEC, Sigma, Exploit-DB) so the model has the shared vocabulary. The train/validation split is grouped by entity, not by document, after we found that the obvious per-document split let the same CVE appear on both sides as NVD prose and as a graph fact.
We left out the largest thing in the graph on purpose. Ninety-seven percent of the nodes are a sentiment-event island from a news dataset. It is not threat intelligence and it never touched the corpus.
Round 1: sixteen minutes on a rented card
The first real run was small-v1: 34 million parameters, two passes over the version-1 corpus, 16 minutes on a rented RTX 5090. We built a probe set from held-out graph edges: show the model a subject, offer four candidate objects, score which continuation it finds most likely. Chance is 25 percent.
The first number looked wonderful. Then we read the per-relation table and found that two relation types scored 98 and 87 percent because the answer string contained the subject's own IP or domain. The question answered itself. We built a leak filter, threw out every probe whose answer shared a distinctive token with its subject, and quoted only the filtered set from then on. That cost us seven points and it is the only number worth having.
On the honest set, a thousand held-out edges, the 16-minute 34M model scored 41.8 percent. Qwen 2.5 1.5B, forty-four times larger, zero-shot on the same questions: 30.5 percent, five points above chance. Qwen 1.5B after a LoRA on 30 million of our proprietary tokens: 41.1 percent. The data moved the number, not the parameter count. Thirty million of our tokens lifted a 1.5B model by ten points; 153 million of them took a 34M model from nothing to the same place.
We split the metric further, because not every edge is learnable. Which product a CVE affects, which weakness it has, which sector a crew targets: those follow from other facts and a model can generalise to them. Which exact URL a family distributed on a given day is an arbitrary identifier. On the learnable relations the small model sat at 44 percent; on the arbitrary ones 37 percent, still above chance, which tells you URL and IP shapes carry some family signal. On "which month did this happen" it was close to guessing. The model had almost no sense of when, which matters for threat intelligence more than almost anything.
The uncomfortable finding
The obvious next move was to make the model remember more. Cleaner graph, dossiers, oversample the proprietary half, more epochs, bigger model. We did all of it over one night and it is the most useful thing we learned.
First we asked a question nobody had asked: how well does the model recall facts it was actually trained on? Not held-out, seen. We mined a second thousand-probe set from the training split. The 34M model after two passes recalled its own training facts at 45 percent, barely above its held-out score. It was not storing the graph. It was learning the dialect and a sense of what is plausible, and guessing.
So we hammered it. Ten passes over the verbalised graph alone took seen-fact recall from 44 to 53 percent and cost the model most of its general knowledge (16 of 20 general cyber questions down to 9). Twenty passes with a replay mix: 53.7 percent. One point per pass, saturating. Then we ran the scaling curve, same recipe, three sizes:
| Parameters | Seen-fact recall | Held-out | General (of 20) | Final train loss |
|---|---|---|---|---|
| 34M | 53.7 % | 43.7 % | 10 | 0.30 |
| 110M | 46.3 % | 39.5 % | 14 | 0.10 |
| 320M | 43.2 % | 37.8 % | 15 | 0.09 |
Bigger was worse. The 320M model had the lowest training loss of the three, the best English, and the poorest recall of a bare fact. It had learned exactly what it was trained on, which is to continue a long dossier, and the probe asked it for a single fact from a bare subject with no dossier in sight. Those are different skills, and the bigger the model, the more it leaned on the context it was used to being given.
That is the end of "make the weights the database" for this corpus, at this scale, on this budget. Not a failure of the models: a mismatch between the training objective and the test. It was also the strongest argument for what we did next, because the model's one reliable skill was reading a dossier.
Round 2: let the model read
The facts stay in the graph. At question time we fetch the entity's dossier from Neo4j, the same dated lines the corpus was built from, put it in front of the model, and ask.
Zero-shot Qwen 2.5 3B, 938 held-out probes, four-way, chance 25 percent:
| Condition | Accuracy |
|---|---|
| Closed book, bare subject | 32.5 % |
| Dossier in context, with the answering line removed | 45.5 % |
| Dossier in context | 76.8 % |
The same model goes from seven points above chance to 77 percent the moment the graph fact is in front of it. The blind row is worth reading twice: entity type and neighbourhood alone are worth 13 points, and the remaining 31 are the fact itself, which is exactly what a database lookup supplies.
Then we fine-tuned it to read our way. We generated 117,000 (dossier, question, answer) pairs from the graph: relation lists, summaries, earliest-dated questions, and 32,000 negatives where the right answer is "the dossier does not mention that". A LoRA adapter (rank 32, 1.9 percent of the weights trainable, loss on the answer tokens only, 60,000 pairs, 1 hour 42 minutes on a 5090):
| Free-text generation, 400 held-out pairs | Base model | With our adapter |
|---|---|---|
| Token F1 against the reference answer | 0.50 | 0.97 |
| Exact match | 2 % | 88.5 % |
| List item recall on relation questions | 0.73 | 0.99 |
| Questions the dossier cannot answer, correctly refused | 100 % | 100 % |
| Invented answers (a capitalised token in neither dossier nor question) | 0.7 % | 0.0 % |
The honest reading: the held-out entities were unseen but the question templates were not, so the generation jump is in-distribution. The base model already reads a dossier reasonably well; the adapter mostly teaches it to answer in our dialect, with dates and labels, and to stop. The ranking probes did not move, which is the intended outcome, since the adapter was trained to read, not to remember. The test that would matter was free-form questions a human would ask.
A hundred questions written by hand
So we wrote them. A hundred analyst questions across 25 real dossiers (crews, CVEs, packages, implants, victims, campaigns), in the phrasing people actually use: which techniques, who did they claim, when was it first seen, is it true that, which of these is not mentioned. Each has a machine-checkable expectation (strings that must appear, groups where any one counts, dates, expected refusals, expected yes), and a checker that verifies every expectation against the real dossier before the model sees a question. If the expectation is wrong, the test is wrong, and the checker caught two of ours.
Result through the production path, the 3B model with our adapter, quantised to 4 bits, running on four CPU cores: 85 out of 100, about 7 seconds a question. Dates 17 of 17. Refusals 8 of 8. Facts 46 of 55. Reasoning 14 of 17. Yes/no questions 0 of 3.
The failures split cleanly and the biggest class was ours, not the model's. The adapter had only ever seen negative yes/no pairs in training ("Did X claim Y?" where Y is absent), so it learned that the answer to a yes/no question is no, even with the line sitting second in its context. Five of the fifteen failures were that. The fix is positive pairs in the training set, and it is in the generator now. Two failures invented a CWE identifier when asked for a name. Six picked a related but wrong line (the French victims when asked for the German ones; "Windows" when asked which Windows component). Two copied a dossier line instead of answering. All of that is documented with the examples, because the misses are the part a buyer should read first.
It runs on four CPU cores
Nothing about serving this needs a GPU. The adapter was merged into the base, converted to GGUF and quantised to Q4_K_M: 1.93 GB. It runs under llama.cpp in a container pinned to four cores of a box we already own, at 22.6 generated tokens a second and 131 prompt tokens a second. A full question, a 1,900-token dossier plus a 40-token answer, is about 15 seconds of reading and 2 seconds of writing.
The path is fully live. The dossier is fetched read-only from the production graph at question time, not from an export, and merged across labels (LockBit the crew, LockBit the software, LockBit as the dark-web crews name it). Ask it which techniques LockBit uses and it returns all six technique lines the graph holds, nothing invented, in 15 seconds, with the dossier it read shown underneath the answer.
It is now behind our natural-language graph interface over a private link between the two boxes: a toggle that says "cortex · own model", an entity field, and the answer comes from our weights and our graph with no third-party API in the loop.
Fixing the graph along the way
A model that reads the graph is only as good as the graph, and the exercise turned into the most thorough audit the data has had. Over the same two days, each change dumped to disk before it was applied and read back after:
- 45,513 "Software" nodes that were actually timestamps from a free-text feed column, deleted (88 percent of that label was noise).
- 39,425 empty CVE shells described from the local NVD mirror, with 15,076 CVE-to-weakness edges merged in.
- Every CWE given its name and description (the official source was unreachable; the data was in a Python package).
- Victim sector and country backfilled across 26,000 organisations: rules first, then a language-model classifier for the 15,000 names the rules could not place, with its confidence stored next to the answer.
- Two competing victim data models unified: 16,883 historic ransomware victims that only the old model could see now carry crew, sector and country like the rest, taking the usable set from 4,970 to 21,853, with an invariant (distinct crew-victim pairs: 22,038 before and after) to prove nothing was double-counted.
- 17,728 orphan indicators linked to the malware family named in the title of the event that reported them (DarkComet alone accounted for 10,070), with the same rule added to the ingester so it does not recur.
None of that is glamorous. All of it is what the model now reads.
What it cost
Under $30 of rented GPU time across every run in this article, including the ones that failed and the scaling curve that told us to stop. The 34M model trains in 16 minutes. The adapter trained in 1 hour 42 minutes. CPU time on our own box was free. The expensive part was attention: reading the per-relation tables, building the leak filter, writing a hundred questions by hand and checking them against the source.
What it cannot do
- It does not know anything the graph does not know. If the dossier is thin, the answer is thin, and it will tell you the dossier does not say.
- It is not a general assistant. Ask it about something outside the dossier and it has nothing to read.
- The from-scratch models do not memorise facts at a useful rate at any size we can afford, and we have stopped trying. Their job is the dialect, surprise scoring (a real fact scores a perplexity of 4, "LockBit is a web browser" scores 317), ranking and embeddings.
- Yes/no questions are currently a weakness for a known reason, above.
- The hand-written set is a hundred questions. That is enough to find failure classes and not enough to quote a decimal place.
- Nothing from any of this reaches a published page without the same verdict gate every other model in our estate has to pass.
What is next
Round 3 ran the same afternoon, and it is worth reporting in full because it did what a round should: it moved the number we wanted and exposed a new mistake. We scored the adapter on 563 held-out facts asked in phrasings it never saw in training, and on 243 "blind" questions where the answering line is removed from the dossier and the only right answer is a refusal. The adapter generalises: 66.8 percent of unseen phrasings answered correctly against 45.3 percent for the bare base model, and 91 of the hand-written 100 at full precision against 86. But it refused only 0.4 percent of the blind questions. It had been trained on negatives of the planted kind only ("is X linked to Y?") and had never learned to say "the dossier does not say" to an open question. A second adapter trained with positive yes/no pairs pushed unseen phrasings to 76.6 percent and fixed the yes/no denials, and then taught us a different lesson: at 13 percent of the training set, the "Yes.
Then the thing we actually think is sellable, which is not a model. It is the pipeline: an organisation's own intelligence, their tickets, their reports, their graph, through the same corpus build, the same honest probes, the same adapter recipe, served on a CPU box inside their own walls with the evaluation attached. We have now done it once, on ours, in the open.
Every number here comes from a dated run, with the probe sets, scripts and failure tables kept alongside. The leak we found in our own first result is in there too. Written 25 October 2026; what happened next is in one sentence beat seventeen fine-tunes and on Project M&M.