3 ms·
Imagine if you could take Llama 3.1 405B and break it down to a tree of logical gates, optimizing out all the things like multiplies by 0 in one of the bits, et
by mikewarot 2y ago
Imagine if you could take Llama 3.1 405B and break it down to a tree of logical gates, optimizing out all the things like multiplies by 0 in one of the bits, etc... then load it into a massive FPGA like chip that had no von Neumann bottleneck, was just pure compute without memory access latency with a conservative 1 Ghz clock rate.
Such a system would be limited by the latency across the reported 126 layers worth of math involved, before it could generate the next token, which might be as much as 100 uSec. So it would be 10x faster, but you could have thousands of other independent streams pipelined through in parallel because you'd get a token per clock cycle out the end.
In summary, 1 Gigatoken/second, divided into 100,000 separate users each getting 10k tokens/second.
This is the future I want to build.
- seangrogg 2y agoI'm actively trying to learn how to do exactly this, though I'm just getting started with FPGAs now so probably a very long range goal.
- ryao 2y agoThere is not enough memory attached to FPGAs to do this. Some FOGAs come with 16GB of HBM attached, but that is not enough and the bandwidth provided is not as high as it is on GPUs. You would need to work out how to connect enough memory chips simultaneously to get high bandwidth and enough capacity in order for a FPGA solution to be performance competitive with a GPU solution.
- mikewarot 2y agoInstead of separate memory/compute, I propose to fuse them.
- jacobgorm 2y agoSee Convolutional Differentiable Logic Gate Networks https://arxiv.org/abs/2411.04732 https://arxiv.org/abs/2411.04732 , which is a small step in that direction.