2 ms·
> it would be infeasible to run a general GPT-like chatbot on consumer PC hardware for a good while. It's lower quality and slower than cloud, but I can run o
by yjftsjthsd-h 3y ago
> it would be infeasible to run a general GPT-like chatbot on consumer PC hardware for a good while.
It's lower quality and slower than cloud, but I can run ollama on my aging laptops today; I wouldn't be surprised if the addition of hardware acceleration allowed new PCs to run a chatbot locally.
- runjake 3y agoYou can run Ollama on your aging laptop today, but at how many tokens per second and at how many parameters? Most consumer PC hardware isn't going to have a very capable GPU. But, I certainly anticipate "AI accelerators" in consumer PCs, in the short term, ala AI PC NPUs. FWIW, I basically leave Ollama running on my M3 Max and 24/7 and use it regularly, and it's ridiculously fast, but: M3 Max with lots of RAM and a crazy memory busy.