Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures

NeurIPS 2020

François Boniface 1

📓

Paper

🐙

Paper code

🔀

DFA implementation

Check out our virtual poster!

Attending NeurIPS? 💬 with us in poster session 6 on Thursday, Dec 10th @ 6-8pm CET.

At variance with previous beliefs, we show an alternative to backpropagation, devoid of weight transport, can scale to modern deep learning tasks and architectures, up to Transformers.

Image classification shouldn't be the litmus test of a training method. We broaden the usual perspective of evaluating alternative to backpropagations on simple tasks like MNIST and CIFAR-10 with a study of unprecented scale:

4 domains

neural view synthesis, recommender systems, geometric learning, natural language processing.

9 tasks

simulated/real 3D scenes, Criteo dataset, citation networks, language generation.

11 architectures

NeRF, hybrid factorization machines, graph convolutions & attention, Transformers.

We ensure learning is happening using a unique breadth of controls. Rather than relying on possibly misleading accuracy evaluations, we perform in depth benchmarks and verifications:

Alyssa Long

accuracy
evaluations

Alyssa Long

shallow
comparisons

Alyssa Long

alignment measurements

Alyssa Long

embeddings visualizations

We focus our study on Direct Feedback Alignment. DFA provides a compromise between biological realism and practical considerations, thanks to its compelling characteristics:

Alyssa Long

synaptic asymmetry

Alyssa Long

parallelizable backward

Alyssa Long

single operation

Direct Feedback Alignment, briefly:

Alyssa Long

Direct Feedback Alignment scales to modern deep learning tasks and architectures. We show alternative training methods are not limited to toy-tasks and can scale to state-of-the-art deep learning:

Alyssa Long

Neural Radiance Fields

Alyssa Long

Hybrid recommender systems

Alyssa Long

Graph convolutional networks

Alyssa Long

Transformers

Our study shows that tackling challenging real-world tasks beyond backpropagation is possible. We hope it will inspire and motive further research on alternative methods.

Cite us:

@article{launay2020direct,
title={Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures},
author={Launay, Julien and Poli, Iacopo and Boniface, Fran{\c{c}}ois and Krzakala, Florent},
journal={Advances in Neural Information Processing Systems},
volume={33},
year={2020}}

A LightOn AI Research paper

LightOn is a Paris-based AI hardware start-up building photonic co-processors for the most demanding applications of machine learning.

LightOn Cloud More cool research