3 ms·
On prem alternatives to ChatGPT and the likes of GitHub Copilot, where you cannot legally send data to cloud services. Underneath everything, ollama can do a l
by KronisLV 2y ago
On prem alternatives to ChatGPT and the likes of GitHub Copilot, where you cannot legally send data to cloud services.
Underneath everything, ollama can do a lot of the heavy lifting, but it still needs to run somewhere. For decent chat models you probably want a server with either one beefy GPU or a few regular consumer ones (it actually seems to split the load just fine, at least when it comes to Nvidia hardware), whereas for smaller models like autocomplete (where 3B or 1.5B models are enough) you can choose whether to use the same server or run locally.