3 ms·
What are you doing that requires local over calling an API provider? If you're developing an AI app that makes lots of calls and is designed for a local GPU it
by kristianp 2y ago
What are you doing that requires local over calling an API provider? If you're developing an AI app that makes lots of calls and is designed for a local GPU it makes some sense perhaps?
- KronisLV 2y agoOn prem alternatives to ChatGPT and the likes of GitHub Copilot, where you cannot legally send data to cloud services. Underneath everything, ollama can do a lot of the heavy lifting, but it still needs to run somewhere. For decent chat models you probably want a server with either one beefy GPU or a few regular consumer ones (it actually seems to split the load just fine, at least when it comes to Nvidia hardware), whereas for smaller models like autocomplete (where 3B or 1.5B models are enough) you can choose whether to use the same server or run locally.