3 ms·
With all such optimizations, isn't llama too slow to be run on a CPU with a reasonable amount of parameters?
by DeathArrow 2y ago
With all such optimizations, isn't llama too slow to be run on a CPU with a reasonable amount of parameters?
- nxpnsv 2y agoI don't see how that conclusion follows from this optimization.
- baq 2y agoModels get better for equal parameters, CPUs get faster and good folks at intel, amd and arm are probably working very hard to catch up with apple silicon in terms of memory architecture. I can see this be very relevant in a couple of years.
- jart 2y agoI'm gpu poor so I use cpu to run large models. For example, llamafile running mixtral 8x22b on my threadripper can chew through long legal documents and give me advice in a few minutes, using f16 weights and kahan summation. Show me someone who has that many graphics cards.
- kkzz99 2y agoWhat are your tokens/s on that setup?
- bsenftner 2y agoAre you "gpu poor" by choice? I would have thought you, of all people, would be "gpu wealthy" like few can imagine.
- jart 2y agoI'm a hobbyist developer. I haven't been employed in six years. In the past few months I've been fortunate enough to find work as a TVC for Mozilla, which has been wonderful. It's a good lifestyle that affords me a great deal of freedom and autonomy, but certainly not a lot of GPUs. I have an RTX 4090, a few XTX cards, and I rent stuff from Vast occasionally, but that's about it.
- baobabKoodaa 2y agoI love how RTX 4090 is now "GPU poor"
- jart 2y agoIt can't run most of the open source models being released today. It's not useful if your job is to keep up with LLM releases.