In May 2025, two Stanford researchers published a result that deserves more attention than it got. Francois Chaubard and Mykel Kochenderfer trained recurrent neural networks to over a billion parameters without backpropagation through time (arXiv:2505.17852). Forward passes only. Perturb the weights, measure the loss, step. Their central-difference estimator matched or beat backprop convergence in their experiments, sometimes by an order of magnitude, while using a fraction of the memory. A 4.3 billion parameter model trained on a single mid-range GPU that standard backprop could not push past 270 million on.
The result is impressive. The premise behind it matters more.
The Premise Matters More Than the Result
The premise is this: transformers pay for their power with unsustainable economics. Attention means every generated token looks back at every prior token, so compute and memory grow with context. A recurrent model instead compresses everything it has seen into a fixed-size state, and updates that state once per observation. Constant memory. Constant cost per step. By the paper's own estimate, that difference is worth roughly 14,000 times in cost per generated token at scale.
Recurrence did not lose the last decade because it was the wrong idea. It lost because the training algorithm, backpropagation through time, has to store the entire history of activations, and that memory wall arrives long before the architecture's limits do.
So their move is: keep gradient descent, but estimate the gradients from forward passes alone, and the wall disappears.
We work on the same premise, and we take one more step in the same direction.
One More Step: Remove the Gradients Entirely
If the thing you want is a fixed-size state that summarizes everything relevant about what came before, there is a mathematical object that is exactly that, by construction: the causal state. Two histories belong to the same causal state precisely when they predict the same future. The machine built from those states, the epsilon-machine in Crutchfield's computational mechanics, is the minimal sufficient model of the process that produced the signal. This is proven theory with four decades of literature behind it, not a new architecture we invented.
And here is the part that changes the economics again: you do not train it. You induce it. The causal states are reconstructed directly from the statistics of the raw signal. There is no loss landscape, no descent, no gradients to estimate, cheaply or otherwise. Learning is not an optimization problem you run on a GPU farm before deployment. It is a measurement you perform on the stream, where the stream lives.
What That Buys on Real Machines
The model's size is set by the intrinsic state complexity of the process being modeled, not by a parameter count chosen in advance. Simple process, small machine.
The state update is one transition per observation. That is what lets it run at telemetry rate on a CPU core, on the device, offline. No accelerator, no connection to anywhere.
Learning continues in place. New signal refines the states on the machine that is observing it, because induction is cheap enough to live next to the sensor.
Scope, Stated Precisely
We should be precise about scope, because precision is the point. This is not a language model and we do not claim parity on language benchmarks. The regime where causal-state induction wins is tracking the state of a physical process from its own signal: telemetry, protocol bytes, audio, video streams.
In our controlled validation work, the same induction chain that learns part-of-speech categories from text separated authentic from manipulated video at AUC 1.000 when pointed at the right atom of change, and recovered the true two-state structure of a drone control protocol from raw bytes. Small corpora, controlled experiments, stated as such. The pattern across all of them is the same: the structure is in the signal, and the mathematics recovers it without a single gradient.
Two Routes, One Destination
Chaubard and Kochenderfer make stateful models trainable at scale by removing backprop's memory from training. We make stateful models learnable at the edge by removing optimization from learning. Both routes follow from taking one uncomfortable fact seriously: the economics of attention do not survive contact with machines that must think at stream rate, on the power budget of the device they live in.
The next decade of on-device intelligence belongs to models cheap enough to live with the signal. The interesting argument is no longer whether state beats attention there. It is how the state should be built: estimated by perturbation, or induced from the causal structure that was in the signal all along.
We build the second answer.
References and Further Reading
The Paper
- Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization. Chaubard & Kochenderfer, Stanford (2025). CD-RGE training of LSTMs and DNCs to 4.3B parameters, forward passes only; code open-sourced.
- Multivariate Stochastic Approximation Using a Simultaneous Perturbation Gradient Approximation. Spall, IEEE Transactions on Automatic Control 37 (1992). The SPSA lineage the paper builds on.
The Mathematics: Causal States
- Inferring Statistical Complexity. Crutchfield & Young, Physical Review Letters 63 (1989). The founding paper: causal states and ε-machines.
- Computational Mechanics: Pattern and Prediction, Structure and Simplicity. Shalizi & Crutchfield, Journal of Statistical Physics 104 (2001). Causal states as the unique minimal sufficient statistic of a process.
This piece continues the argument from "The Physical-World Turn" and "The Missing Epidermis": the industry captures physical signal to train models later; we build the layer that understands the signal at the source, while the machine runs. At Senua AI that layer already flies drones and drives industrial machines.