7 ms·
The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not red
by bongobingo1 3y ago
The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
- airgapstopgap 3y agoMulti-Query Attention, used here, should make 40B inference viable on systems where even 33B LLaMA with Multi-Head is basically unusable, so sometimes improvements still come from software optimization (it's no free lunch though).
- BonoboIO 3y agoWill be interesting to see if someone comes up with an ASIC or FPGA.
- groby_b 3y agoWhy would you think there's an ASIC/FPGA design that significantly improves over GPUs specifically targeted at running large models already? Where's the win? The fundamental limit for hardware acceleration are number of gates you can squeeze on a die, right now. (Or, alternatively. memory bandwidth)
- code_witch_sam 3y agoanalog chips like what MythicAI is developing seem like the next obvious leap forward for deploying inferences broadly. ASIC/FPGA wouldn't be much different than a GPU. ASIC seems like a brittle solution https://www.youtube.com/watch?v=GVsUOuSjvcg https://www.youtube.com/watch?v=GVsUOuSjvcg
- wrs 3y agoThe minimum size of a model with equivalent performance is unknown (like so much else about LLMs), so requirements could be reduced.
- Tepix 3y agoI think there is still lots room for improvement to reduce hardware requirements, such as 3-bit quantization or pruning weights from sparse models. But if you're willing to spend $1500 on two used RTX 3090, it's the sweet spot in terms of the ability to run large models right now. Everything beyond that is much more expensive.