GoKawiilAugmented Lagrangian Predictive Coding
Training 1000-layer networks without backpropagation
We introduce PC-ALM, a local alternative to backpropagation. PC-ALM trains residual MLPs up to 1000 layers, nearly matching backprop's performance despite using only layer-local dynamics. PC-ALM equips each layer with a feedback control dynamical system that distributes and propagates supervision credit throughout a network.
Resources Paper
Code Authors Jeffrey Seely Sakana AI Julian Gould Sakana AI Published
Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly [1, 2] . How the brain solves the multilayer credit assignment problem without explicit use of backprop remains one of the fundamental unsolved problems in neuroscience (though not without progress [3, 4, 5] ).
There are several reasons the brain can't implement exact backpropagation. One is “phase locking” [6, 2] . Backpropagation runs in three phases, in strict order: 1) a forward pass, then 2) a backward pass, then 3) a weight update. A weight update is locked until the forward and backward passes have completed—a neuron in an early layer must hold its activation and wait for the error signal to arrive. The brain has no known mechanism that could enforce such strict timing coordination across an entire network [1] .
In this post, we introduce PC-ALM (Augmented Lagrangian Predictive Coding), a method for training networks that replaces the forward and backward passes of backprop with layer-local dynamical systems. Each layer is coupled only to its neighbors. Instead of forward-then-backward, we run each layer forward in time. When run to convergence, the dynamics of the whole system distribute supervision credit signals quickly and accurately across the entire network.
PC-ALM is an extension of standard predictive coding (PC) [7, 8, 9, 10] . PC uses diffusive (i.e. energy-based or "heat flow") coupling between layers. Compared to PC, PC-ALM introduces dual neurons (Lagrange multipliers) per layer, making each layer's local recurrence a PI feedback controller. In the limiting case of linear networks, the dual neurons converge to the exact backprop credit signals, despite using only local computation.
We compare PC-ALM to PC and backprop in a suite of experiments. Local training methods such as PC have historically been difficult to scale. Following the PC literature, we use simple tasks (Fashion-MNIST, CIFAR-10, etc.) and networks such as residual MLPs.
... continue reading