The theory
Predictive coding (Rao & Ballard 1999; Friston 2005; Bogacz 2017) frames perception and learning as free-energy minimisation. This is the mathematics every other project in the codebase rests on — and the reason its learning rules never need a backward pass.
The model
A hierarchical network has L layers. Each layer
l holds a latent state vector μ_l and
generates a top-down prediction of the layer below:
pred_{l-1} = f(W_l · μ_l + b_l)
where f is a non-linearity (Tanh by default) and
W_l, b_l are the generative weights and
biases. The prediction error at layer
l−1 is precision-weighted:
ε_{l-1} = Π_{l-1} · (μ_{l-1} − pred_{l-1})
Π is a scalar or diagonal precision —
the inverse variance, that is, how much the network trusts that
error. The whole network then minimises the free
energy, the sum of squared prediction errors:
F = ½ Σ_l ‖ε_l‖²In this framing the sensory input is not something to classify; it is something to explain. Perception is the process of finding the latent states that make the top-down predictions match it, and learning is the process of making the predictions better.
Two nested updates
Inference is fast and runs per example: settle the
hidden states by gradient descent on F, holding the
weights fixed:
μ_l ← μ_l − η_s · ( ε_l − W_lᵀ · δ_{l-1} + λ_act · sign(μ_l) )
δ_{l-1} = f'(pre_{l-1}) ⊙ ε_{l-1}
The input layer μ_0 is clamped to the
datum; during supervised learning the top layer is clamped to the
label. Everything in between is free to move until the network
settles.
Learning is slow and happens after inference has settled: the weights update with a local Hebbian rule — no backpropagation:
ΔW_l = η_w · δ_{l-1} ⊗ μ_lᵀ
Δb_l = η_w · δ_{l-1}Every term in that update is available locally at a layer: the product of its own prediction error and the activity of the layer above. This is the property the whole codebase exploits — the same rule trains a classifier, a memory, an agent and a cortex, because none of them needs to see beyond their own synapse.
Classification without a softmax head
There is no output layer with a softmax. To classify, the network clamps the input, tries each candidate label at the top, and keeps the one that settles to the lowest free energy:
c* = argmin_c F( x, one-hot(c) )
The class that needs the least correction to explain the input wins.
This argmin F principle reappears everywhere in the
solution: the mixture-of-experts gates on free energy, the cortex
readout uses it, and the LLM's conventional cross-entropy training is
a deliberate departure from it.
Variants implemented in the core
| Idea | Reference | Where |
|---|---|---|
| Hierarchical predictive coding | Rao & Ballard 1999 | Network |
| Free-energy principle | Friston 2005 | Layer.LocalEnergy |
| Diagonal precision, L2 decay | Huang & Rao 2011 | UseDiagonalPrecision, WeightDecay |
| Feedforward Gist warm-start | Haider et al. 2024 | UseFeedforwardGistInit |
| L1 activity sparsity | Keller & Mrsic-Flogel 2018 | ActivityRegularization |
| Temporal continuity | Friston 2008 | TemporalNetwork |
Each of these is a configuration switch on the same engine, not a fork: a diagonal precision changes how errors are weighted, the gist warm-start seeds the hidden states with a single top-down pass before the iterative loop, and the temporal variants inject the previous timestep's latent as a prior.
Why locality matters
Because learning is local and energy-based, the same engine generalises cleanly to modular hierarchies, temporal cascades, attractor memories, active inference and mixtures of experts — all of which appear in the Expanded and Cortex projects without ever introducing global backpropagation. It is also the property that makes the system biologically plausible: no neuron anywhere needs to know about the loss function of the whole network.
References
- Rao, R. & Ballard, D. (1999). Predictive coding in the visual cortex. Nature Neuroscience.
- Friston, K. (2005). A theory of cortical responses. Philosophical Transactions of the Royal Society B.
- Bogacz, R. (2017). A tutorial on the free-energy framework for modelling perception and learning. Journal of Mathematical Psychology.
- Whittington, J. & Bogacz, R. (2017). An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation.
- Huang, Y. & Rao, R. (2011). Predictive coding under the free-energy principle. Phil. Trans. R. Soc. B.
- Haider, P. et al. (2024). Feedforward Gist initialisation for predictive coding.
- Keller, G. & Mrsic-Flogel, T. (2018). Predictive coding: a computational and biological perspective. Neuron.