Sakana AI Proposes Brain-Inspired Alternative for Neural Network Training

Sakana AI Proposes Brain-Inspired Alternative for Neural Network Training

All modern neural networks we use are trained via backpropagation. This algorithm, invented in the 1970s, is still used everywhere.

However, there are several criticisms of it. The main issue lies in its structure. Backprop works strictly in three sequential phases: first, the signal passes forward through the network to produce an output; then, the error signal travels backward from the output to the input; and only then are the weights updated. Biologically, this is highly implausible: there is no mechanism that forces our neurons to wait for an error signal to arrive from somewhere through all other neurons before they can update.

Sakana AI is known for drawing inspiration from neuroscience. They propose looking towards an approach called predictive coding (a term from research on the visual cortex structure).

Usually, when we train a neural network, only its weights are optimized. In predictive coding, we optimize both the weights and the activations of each layer: each layer tries to predict the activity of the next layer, and only the residual that it couldn't explain is passed upwards. The idea is that the system must find a local equilibrium between the prediction error from below and from above, and then the network's outputs will accurately predict the target variable.

This approach wasn't invented by Sakana AI; it's quite well-known. But unfortunately, it didn't work: in deep networks, the signal simply attenuates, and nothing learns because the network doesn't know where to move the weights.

Sakana added just one small tweak — and the method started training networks up to 1000 layers effectively. The addition of a Lagrange multiplier (an old Soviet-era mathematical concept) helped. As early as 1988, Yann LeCun showed that if you formulate a neural network as a constrained optimization problem (as predictive coding does) and assign a Lagrange multiplier λ to each constraint, then at equilibrium, λ is exactly equal to the backprop gradient. This is a mathematical fact that Sakana decided to leverage.

In other words, while vanilla predictive coding provides an approximate signal, Sakana's approach — PC-ALM — converges exactly to the backprop gradient on linear networks using only connections with neighboring layers, without a single global pass.

On various benchmarks, PC-ALM almost matches backprop (a difference of ~2 percentage points in accuracy on MNIST). And yes, there is no advantage in accuracy or training speed over backprop. It will likely win (eventually) only on neuromorphic hardware in the future. But we may have taken a step closer to recreating the learning process of a living brain. Plus, it's just elegant ☕️

Read the original publication