Language

Why a second engine

The core of this codebase learns locally, without backpropagation. That is its whole point — but Transformers are trained with cross-entropy and reverse-mode gradients. Rather than bend the core, the LLM project adds a separate, small autodiff engine: a tensor carrying data, gradient and a backward function, with ops for matrix multiply, causal softmax, layer norm, dropout and cross-entropy, and a topological backward pass. The trainable model on top of it is a single-block Transformer whose gradients are checked against numerical differences in the test suite.

Tokenization

Tokenizer	What it is
ByteTokenizer	Lossless byte-level with BOS/EOS/PAD/UNK — nothing unrepresentable.
BpeTokenizer	A trainable byte-level BPE: it learns its merges from a corpus and encodes greedily.
WordPieceTokenizer	Trainable greedy longest-match subword, the ##-piece scheme.

The model

GptModel embeds, runs the blocks, applies the final norm and emits logits, stepping one token at a time with a KV cache. The pieces inside are the ones the current generation of open models use:

• Attention — multi-head, grouped-query (GQA) or multi-query; rotary (RoPE) or ALiBi positions; optional QK-Norm and attention soft-capping; a flash-style online softmax.
• Latent attention (MLA) — the DeepSeek-V2/V3 trick: the cache stores a compressed latent plus a decoupled-RoPE key instead of full K/V, shrinking memory per token.
• Feed-forward — classic GELU or gated SwiGLU.
• Norms — LayerNorm or RmsNorm; optional final-logit soft-capping (Gemma 2).
• Presets — ModernPresets.Llama3Style (RMSNorm + SwiGLU + RoPE + GQA) and ModernPresets.DeepSeekStyle (adds MLA, QK-Norm and soft-caps) so a modern decoder is one constructor call.

Generation

• Sampler — greedy, temperature, top-k, top-p, plus repetition, frequency and presence penalties.
• TextGenerator — the autoregressive loop with prefill.
• BeamSearch — length-normalised, forking KV caches per hypothesis.
• BatchedGenerator — N independent sequences in lockstep.
• ContinuousBatcher — an in-process serving scheduler: admits sequences on free slots, evicts on finish.
• ChatTemplate — role-tagged prompt formatting.

Training

The autodiff path is complete: Adam and AdamW, global-norm gradient clipping, a warmup-plus-cosine learning-rate schedule, a gradient-accumulating trainer, and binary checkpoints that save and resume. Data comes as shifted next-token windows, either materialised or streamed line-by-line for corpora that do not fit in memory. Loss and perplexity are computed by the same cross-entropy op.

Infrastructure

• Mixed precision — fp16 via System.Half and bf16, stored in 16 bits, accumulated in 32.
• GEMM — SIMD and multithreaded on CPU, with a pluggable provider interface so CUDA slots in without touching model code.
• Tensor parallelism — column- and row-parallel sharded linear layers over a device mesh.
• Quantization — int8 matrices and packed int4.

The modern-techniques map

Technique	Paper	Switch
Multi-head Latent Attention	DeepSeek-V3 (arXiv:2412.19437)	AttentionKind.Latent
SwiGLU	Shazeer 2020	FeedForwardKind.SwiGlu
RMSNorm	Zhang & Sennrich 2019	NormKind.RmsNorm
RoPE	Su et al. 2021	PositionalKind.Rotary
QK-Norm	OLMo 2 / Gemma 2	UseQkNorm
Logit soft-capping	Gemma 2	AttnLogitSoftcap, FinalLogitSoftcap
Grouped-query attention	Llama 3	NumKvHeads

Two examples

// A modern decoder, DeepSeek-V3 style.
var model = new GptModel(ModernPresets.DeepSeekStyle(vocab: 260), new Random(0));
var tok   = new ByteTokenizer();
string text = new TextGenerator(model, tok)
    .Generate("hello", new SamplingOptions { MaxNewTokens = 64, TopP = 0.9f });

// Train the autodiff Transformer.
var lm  = new TinyTransformerLM(vocab: 5, dim: 8, ff: 16, new Random(0), dropout: 0.1f);
var opt = new Adam(weightDecay: 0.01f);
float loss = Trainer.Step(lm, batch, opt, learningRate: 3e-4f, clipNorm: 1f);

An honest caveat, straight from the documentation: freshly constructed GptModel weights are random — meaningful text requires the autodiff training path. The tests assert shapes, finiteness and gradient correctness, not prose quality. The point of the project is the engineering: a complete, readable, modern LLM stack where every technique is a switch you can turn off.