4 ms·
How do you do it? Do you host on your hardware or do you use cloud-based providers for open models?
by haolez 1y ago
How do you do it? Do you host on your hardware or do you use cloud-based providers for open models?
- egypturnash 1y agoYou have a very curious definition of "local" if that includes "cloud-based providers".
- jasonjmcghee 1y ago> the models I do use remain static and predictable Some people overload "local" a bit to mean you are hosting the model - whether it's on your computer, on your rack, or on your hetzner instance etc. But I think parent is referring to the open/static aspect of the models. If it's hosted by a generic model provider that is serving many users in parallel to reduce the end cost to the user, it's also theoretically a static version of the model... But I could see ad-supported fine tunes being a real problem.
- deleted 1y ago[deleted]
- throwanem 1y agoI am bewildered to hear the sense of "local," in which I and engineers in my experience have for thirty years referred to things which are not remotely hosted, referred to as an "overload."
- haolez 1y agoMy definition of local is the same as yours. It's just that my mind got focused on the open models part and I forgot that you specifically mentioned a "local" setup. My bad
- jasonjmcghee 1y agoI thought it was friendlier than "abuse"
- throwanem 1y agoLess absurd, anyway. In what sense does one "abuse" a word to use it in its dictionary definition? You appear to be a good example of where this benighted excuse for an industry is headed. Explain it to me.
- waynecochran 1y agoI assume this means, for example, using their own AWS / EC2 instances to store and process -- not "local" geographically, but "local" personally.
- throwanem 1y agoIt's a two-year-old Mac Studio in the other room. Where are you coming by this novel sense of 'local?'
- politelemon 1y agoLocally it's pretty simple to run models on GPUs, even low powered ones. Have a look at gpt4all as a starting point but there are plenty of offerings in this space.
- yzjumper 1y agoNot the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access. Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API. An M4 Pro would do much better do to the increase in RAM and GPU size.
- mansilladev 1y agoIf you have Docker (Desktop) installed, with just a couple of clicks, you can get a local model going on your computer. llama3.2 (3B), llama3.3 (70B), deepseek-r1, and about a dozen others.
- c0nducktr 1y agoDo you know if there's a site which will help me pick a model to run locally, based on my system specs and needs? This has been the hardest thing for me to learn and since everything's evolving so quick, what's recommended one week might not be the next.