6 ms·
Transformers scale poorly vs. context window size and parameter count. Which means really impressive when those N’s are small! I’m but a pundit in this area s
by cadamsdotcom 4mo ago
Transformers scale poorly vs. context window size and parameter count.
Which means really impressive when those N’s are small!
I’m but a pundit in this area so don’t know much. But one wonders if there’s a future in burning larger models to FPGAs - whether big enough FPGAs exist (or can be built), and whether locating specialized compute right with the memory it needs can speed things up.
Likely would need a lot of algorithm parallelism work that’d translate back to CPUs/GPUs.
- T-A 4mo agoRelated: https://www.spheron.network/blog/etched-ai-sohu-vs-nvidia-transformer-asic-inference/ https://www.spheron.network/blog/etched-ai-sohu-vs-nvidia-tr...
- naikrovek 4mo agoHuge FPGAs don’t really exist, but you can couple many together with high-speed interconnects. They will never be as fast as an NPU designed to run large models, though. GPUs are extremely general purpose in comparison, and FPGAs are about as general purpose as one can get.