Hebb
A memory is held not in one neuron but in a coalition of them.
E ← γE + η(Q⊗V)
Hebbian state matrix. Memory is written by co-activation and decayed by time — not held in weights that only backprop can move.
The idea
A memory is a coalition, not a cell.
Donald Hebb, 1949: neurons that fire together wire together — into assemblies that can re-ignite from a fragment of the original input.
-
1949
-
2012
Optogenetics
Tag the cells active while a mouse learns to fear a box. Shine light on just those cells in a safe box. The mouse freezes. The assembly was the memory.
-
Today
The unit that never died
A fatiguing leaky integrate-and-fire neuron. It spikes, it leaks, it tires. Learning is local and Hebbian — no labels, no global error, no backward pass.
Assemblies form, overlap, compete, and sustain recurrent activity after the stimulus is gone. A weight matrix doesn't do that.
partial cue
assembly re-ignites
fatigue + sustain
What I've been building
BDH — a language model with brain mechanics underneath.
The Dragon Hatchling architecture (arXiv 2509.26507), implemented from scratch and extended: transformer-scale training without the transformer's attention table.
Q(KᵀV), not (QKᵀ)V
Linear attention. O(N) instead of O(N²). No N×N attention matrix is ever built.
λ = 0.95 / 0.99 / 0.995
Multi-scale memory. Three decay rates, roughly short-term plasticity, early LTP, late LTP.
~5% of neurons active
Sparse by construction. ReLU in the FFN keeps cortex-like sparsity without a sparsity penalty.
From-scratch implementation plus extensions: BDH-RD recurrent depth, BBPE tokenization, teacher distillation, and benchmark tooling.
What it actually managed
Ten million parameters, trained from scratch.
10.23M
parameters
24×
500-token retention vs single-scale baseline (15% vs 0.6%)
~5%
active neurons
2.75×
faster training with BBPE (27.5K vs 10K tok/s)
95%
stable runs after stabilization (vs 40% baseline)
Figures are project-reported from the BDH training runs and docs (README / BDH_SUMMARY). Shown as results so far, not as claims of beating transformers.
The open question
Did the neurobiological programme lose on merit — or on compute?
I can't answer that tonight. What I can say is that the simple version of the claim — that local, label-free, Hebbian learning cannot scale — is now testable on hardware the 1990s never had. BDH is one such test. It is not a win over gradient descent. It is also not a failure of the idea.
What the Hebbian line still knows that gradient descent doesn't
- Memory you can read — the state matrix is an object, not a hidden activation.
- Retention without a KV cache — context held in a fixed O(d²) state.
- Learning with no backward pass — weights move only when pre- and post-synaptic cells agree.
The question is whether that trade changes at scale. That's what I'm working on next.