3 ms·
> Transformers (GPT-3, Copilot ,) are built upon an expensive self-attention network. Instead or also FFT looks to be 7x GPU-cheaper. "Google Replaces BERT Sel
by westurner 3y ago
> Transformers (GPT-3, Copilot ,) are built upon an expensive self-attention network. Instead or also FFT looks to be 7x GPU-cheaper.
"Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs" (2021) https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-19/ https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba...
The next step of Conway's Game can be calculated with FFT and also 2D Convolution.
Convolution >
Visual explanation:
https://en.wikipedia.org/wiki/Convolution https://en.wikipedia.org/wiki/Convolution
- westurner 3y ago"Convolutions / Why X+Y in probability is a beautiful mess" by 3blue1brown https://youtu.be/IaSGqQa5O-M https://youtu.be/IaSGqQa5O-M