3 ms·
You can run a 7B model on CPU relatively quickly. If you want to go faster, the best value in public clouds may be a rented Mac mini.
by speedgoose 3y ago
You can run a 7B model on CPU relatively quickly. If you want to go faster, the best value in public clouds may be a rented Mac mini.
- objektif 3y agoDo you have any resources to read on how to host LLMs in general? I am looking for scaleable ways to host our own models. Thanks.
- speedgoose 3y agoSorry I haven’t followed the latest developments to run at scale since the summer. I don’t have concurrent users so llama.cpp or diffusers are good enough for me.