4 ms·
I'm gpu poor so I use cpu to run large models. For example, llamafile running mixtral 8x22b on my threadripper can chew through long legal documents and give me
by jart 2y ago
I'm gpu poor so I use cpu to run large models. For example, llamafile running mixtral 8x22b on my threadripper can chew through long legal documents and give me advice in a few minutes, using f16 weights and kahan summation. Show me someone who has that many graphics cards.
- kkzz99 2y agoWhat are your tokens/s on that setup?
- bsenftner 2y agoAre you "gpu poor" by choice? I would have thought you, of all people, would be "gpu wealthy" like few can imagine.
- jart 2y agoI'm a hobbyist developer. I haven't been employed in six years. In the past few months I've been fortunate enough to find work as a TVC for Mozilla, which has been wonderful. It's a good lifestyle that affords me a great deal of freedom and autonomy, but certainly not a lot of GPUs. I have an RTX 4090, a few XTX cards, and I rent stuff from Vast occasionally, but that's about it.
- baobabKoodaa 2y agoI love how RTX 4090 is now "GPU poor"
- jart 2y agoIt can't run most of the open source models being released today. It's not useful if your job is to keep up with LLM releases.