AXIOMINFERENCE
Research
Small models.
Real adaptation.
Project Kestrel
Kestrel-Nano v0.1 trained — Phase D nextA trained small-model program, not only a design.
Kestrel‑Nano v0.1 is a 157M-parameter from-scratch model trained on 1.311B tokens. The implementation, trainer, tokenizer, data pipeline, CPU smoke tests, Phase C dataset, and generalization evaluation suite are complete.
The evaluation is intentionally honest: v0.1 generates coherent text, but it remains undertrained and learned more form than content. Phase D will over-train on the completed 4.86B-token curriculum and compare directly against the v0.1 control.
Read the master plan
Why this project exists
Independence from the cloud. People in low-connectivity regions — and anyone who values privacy and ownership — deserve capable local models. Nano targets roughly 100 MB quantized; Mini roughly 300 MB. The research path is designed for modest local hardware as well as a larger training run.
Small doesn't mean shallow. Recent evidence from looped models, hybrid attention, and sparse memory research shows small models close the gap through architecture shape, data quality, and adaptive compute — not just scale.
The "frozen weights" problem. Today's models largely stop learning the moment training ends. Kestrel's memory subsystem — recurrent session state, sparse memory layers, and nightly consolidation — is a direct engineering response to that.
Hardware-grounded, by rule. The current lab GPU is an RTX 3060 12 GB. The implementation remains pure PyTorch and matmul-heavy, with local and cloud configurations measured separately instead of assuming a paper design will run.
The five pillars — v0.1
Hybrid sequence mixer
Three gated linear-attention blocks to one full-attention block — O(1) inference state per token, long context on 8 GB, a persistent "mind-state."
Looped core
The middle block group re-runs one to four times — deeper effective reasoning without more parameters, and compute that adapts to the problem.
Product-key memory layers
Large sparse key-value banks give knowledge capacity without extra FLOPs — the "plastic tissue" that makes continual learning possible.
Token efficiency
A superword BPE tokenizer plus a multi-token-prediction auxiliary head — roughly 20–30% more text per FLOP and per context window.
Layered memory & consolidation
Context flows into recurrent state, into memory slots, into nightly sparse consolidation — a model that learns from use without catastrophic forgetting.
Ground rules
Every deviation from a vanilla transformer has to beat the matched-compute baseline or it gets cut — evidence or ablation, no exceptions. Kestrel-Nano v0.1, its limitations, and the Phase C preparation work are documented openly. The next claim will only be made after Phase D is trained and compared against the v0.1 control.
View the repository