At variance with previous beliefs, we show an alternative to backpropagation, devoid of weight transport, can scale to modern deep learning tasks and architectures, up to Transformers.

Image classification shouldn't be the litmus test of a training method. We broaden the usual perspective of evaluating alternative to backpropagations on simple tasks like MNIST and CIFAR-10 with a study of unprecented scale:

We ensure learning is happening using a unique breadth of controls. Rather than relying on possibly misleading accuracy evaluations, we perform in depth benchmarks and verifications:

accuracy
evaluations

shallow
comparisons

alignment measurements

embeddings visualizations

We focus our study on Direct Feedback Alignment. DFA provides a compromise between biological realism and practical considerations, thanks to its compelling characteristics:

synaptic asymmetry

parallelizable backward

single operation

Direct Feedback Alignment, briefly:

Direct Feedback Alignment scales to modern deep learning tasks and architectures. We show alternative training methods are not limited to toy-tasks and can scale to state-of-the-art deep learning:

Neural Radiance Fields

Hybrid recommender systems

Graph convolutional networks

Transformers

Our study shows that tackling challenging real-world tasks beyond backpropagation is possible. We hope it will inspire and motive further research on alternative methods.

Cite us: