4 ms·
I'm no LLM evangelist, far from it, but I expect models of similar quality to the current bleeding-edge, will be freely runnable on consumer hardware within 3 y
by wubrr 1y ago
I'm no LLM evangelist, far from it, but I expect models of similar quality to the current bleeding-edge, will be freely runnable on consumer hardware within 3 years. Future bleeding-edge models may well be more expensive than current ones, who knows.
- ls612 1y agoHow do the best models that can run on say a single 4090 today compare to GPT 3.5?
- electroglyph 1y agoQwen 2.5 32B which is an older model at this point clearly outperforms it: https://llm-stats.com/models/compare/gpt-3.5-turbo-0125-vs-qwen-2.5-32b-instruct https://llm-stats.com/models/compare/gpt-3.5-turbo-0125-vs-q...
- ls612 1y agoEven when quantized down to 4 bits to fit on a 4090?
- FuckButtons 1y agoNot in my experience, running qwen3:32b is good, but it’s not as coherent or useful as 3.5 at a 4bit quant. But the gap is a lot narrower than llama 70b.
- TeMPOraL 1y agoFor the purpose of keeping the costs of LLM-dependent services down, you don't need to run bleeding-edge models on single consumer GPUs. Even if it takes a hundred GPUs, it still means people can start businesses around hosting those models, and compete with the large vendors.