2 ms·
He isn't wrong, but with current GPU hardware back propagation is actually the superior algorithm compared to his latest forward forward algorithm. Swapping the
by imtringued 3y ago
He isn't wrong, but with current GPU hardware back propagation is actually the superior algorithm compared to his latest forward forward algorithm. Swapping the training algorithm doesn't get around the fact that you need to perform lots and lots of forward passes and those need an insane amount of memory bandwidth. Back propagation isn't a significant bottleneck in computers and also not in terms of memory capacity. So the only benefit of the biologically plausible forward forward algorithm is that you could run it on a digital or analog NPU, but with Ryzen AI coming in Strix Point, every high end laptop is going to have 50 TOPS of AI compute and 200GB/s of memory bandwidth. Nothing except large datacenter GPUs or a hypothetical 5090 with 32 GB would be competitive against that anytime soon. Anything smaller and you will have to buy several 3090 on eBay for like $900 a pop. Analog circuits are far off for now.