3 ms·
You could always run your own server locally if you have a decent gpu. Some of the smaller LLMs are getting pretty good.
by redman25 1y ago
You could always run your own server locally if you have a decent gpu. Some of the smaller LLMs are getting pretty good.
- n_ary 1y agoCorrect. My dusty Intel Nuc is able to run a decent 3B model(thanks to ollama) with fans spinning but does not affect any other running applications. It ks very useful for local hobby projects. Visible lags and freezes begin if I start a 5B+ model locally.
- theshrike79 1y agoAlso M-series Macs have an insane price/performance/electricity consumption ratio in LLM use-cases. Any M-series Mac Mini can run a pretty good local model with usable speed. The high-end models easily compete with dedicated GPUs.
- stunnAR 1y agoYes - of course. That's been my experience with "ultimate" privacy.