Skunkworkz is the build-in-public log of a two-person cyber shop. For three weeks we have been building Ninja Cortex, a model that reads our threat-intelligence graph and answers like an analyst: from the evidence only, and "the dossier does not say" when the evidence is not there. This is the dated record, failures first where they happened.
The short version
- 17 training runs and $448 of rented GPUs (RunPod billing, 23 October to 1 November).
- The best reader we measured is untrained: Qwen3.8-27B plus one rewritten instruction, 197 of 200 on our hand-written exam, 8 of 8 pre-registered bars. Why one sentence won.
- The most useful result is a fail: teaching a 20B model 358,000 facts made it invent answers for 99.4 % of things it had never seen. Weights are not a database.
- Every bar is written down before its run. How we stop ourselves cheating.
The log
23 October. A 34-million-parameter model, built from scratch with its own tokenizer, trains in 16 minutes on a rented card. On 1,000 held-out graph questions it scores 41.8 %, where an open 1.5B model zero-shot scores 30.5 %. We find a leak in our own probe set and throw out every question that answered itself.
25 October. The architecture changes: the facts stay in the graph and a 3B model reads a dossier per question. It serves on four CPU cores at 22.6 tokens a second and scores 85 of 100 hand-written analyst questions. The full field report.
26 October. We publish the 200: every exam question, the judge rules and every miss.
27 October. Size matters: a 14B model reads at 185 of 200 on a rented card. The Skunkworkz portfolio goes up.
28–29 October. Project M&M: gpt-oss-20b runs 5 to 11. Training cuts invented words from 32 % of answers to 0.4 %, but run after run misses a bar on the refuse-or-find seesaw.
30 October. The first trained model to pass every bar of the day: runs 14 and 15 blended, 188 of 200, refusal 0.964. We order two 96 GB cards of our own.
31 October. The moonshot fails as pre-registered. A bake-off of nine untrained open models puts Qwen3.8-27B at the top at 193, ahead of our trained blend.
1 November. Run 17 (Qwen, fine-tuned) fails on the indirect exam. So does untrained Qwen with our production sentence: the sentence, not the data, was pushing it to refuse. Renting ends at $448.
2 November. W1, one rewritten sentence: 8 of 8 bars, 197 of 200.
3 November. The live gate passes, 197 and 78 of 90 dialogue turns on the serving stack, after we find and fix our own bug (the card gave each request half the reading window we assumed).
4 November. Chalk and Talk, our first livestream: four models on the front door of skunkworkz.io answering viewers' questions, and a census of the data only we hold.
5 November. Eight short clips from the stream, captioned, for anyone who missed it.
What's next: data nobody else has
Anyone can download the public feeds, and anyone can download an open model. Training a public model on public facts gives a competitor nothing they cannot copy in a weekend. The next dataset, ten times the last, comes from what only we hold:
- Time. Not just what happened, but when we saw it. Our feeds heard 142 vulnerabilities discussed before the official database published them.
- Our own sensors. A honeypot that has recorded 74,181 attacker sessions since 5 October, plus dark-web crawls and edge logs.
- Judgment. Every training row our auditors rejected, next to the fixed version and the reason.
- Fresh research. New collectors pick up security papers the day they land (261 on day one) and live Telegram channels.
One finding from the census: downloadable is not the same as trainable. Some of the most-used free threat feeds forbid AI training in their terms. Those facts can be shown to an analyst. They will never become a training row.
The two 96 GB cards are due around 16 November. The next training run has one job: beat 197 with W1, on data no one else has. Follow it on the live build log and Project M&M.