4 ms·

This has been coming for a long time and it is why I use local models only. I'm willing to give up capabilities in exchange for being able to trust that whateve

This has been coming for a long time and it is why I use local models only. I'm willing to give up capabilities in exchange for being able to trust that whatever biases may exist in the models I do use remain static and predictable.

atentaten 1y ago
What does your hardware setup look like?
- yzjumper 1y ago
  Not the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access. Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API.
haolez 1y ago
How do you do it? Do you host on your hardware or do you use cloud-based providers for open models?
- egypturnash 1y ago
  You have a very curious definition of "local" if that includes "cloud-based providers".
  - jasonjmcghee 1y ago
    > the models I do use remain static and predictable Some people overload "local" a bit to mean you are hosting the model - whether it's on your computer, on your rack, or on your hetzner instance etc. But I think parent is referring to the open/static aspect of the models. If it's hosted by a generic model provider that is serving many users in parallel to reduce the end cost to the user, it's also theoretically a static version of the model... But I could see ad-supported fine tunes being a real problem.
    - deleted 1y ago
      [deleted]
    - throwanem 1y ago
      I am bewildered to hear the sense of "local," in which I and engineers in my experience have for thirty years referred to things which are not remotely hosted, referred to as an "overload."
      - haolez 1y ago
        My definition of local is the same as yours. It's just that my mind got focused on the open models part and I forgot that you specifically mentioned a "local" setup. My bad
      - jasonjmcghee 1y ago
        I thought it was friendlier than "abuse"
        throwanem 1y ago
        Less absurd, anyway. In what sense does one "abuse" a word to use it in its dictionary definition? You appear to be a good example of where this benighted excuse for an industry is headed. Explain it to me.
  - waynecochran 1y ago
    I assume this means, for example, using their own AWS / EC2 instances to store and process -- not "local" geographically, but "local" personally.
    - throwanem 1y ago
      It's a two-year-old Mac Studio in the other room. Where are you coming by this novel sense of 'local?'
- politelemon 1y ago
  Locally it's pretty simple to run models on GPUs, even low powered ones. Have a look at gpt4all as a starting point but there are plenty of offerings in this space.
- yzjumper 1y ago
  Not the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access. Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API. An M4 Pro would do much better do to the increase in RAM and GPU size.
- mansilladev 1y ago
  If you have Docker (Desktop) installed, with just a couple of clicks, you can get a local model going on your computer. llama3.2 (3B), llama3.3 (70B), deepseek-r1, and about a dozen others.
  - c0nducktr 1y ago
    Do you know if there's a site which will help me pick a model to run locally, based on my system specs and needs? This has been the hardest thing for me to learn and since everything's evolving so quick, what's recommended one week might not be the next.
unshavedyak 1y ago
I agree with this, though i'm still using Claude atm. I figure if we're aware of the downsides you pointed at then we can skip the fast changing landscape of self hosting. It keeps getting cheaper and cheaper to self host, so i'm not sure at what point it makes sense to invest. For me the switching point will probably be when they (AI companies) start the big rug pull. By then my hope is self hosting will be cheaper, better, easier, etc.
- throwanem 1y ago
  Better not to form the habit, I thought. I'm sure I miss out on some things that way, but that is the lesser risk. I do use the Gemini assistant that came with this Android, in the same cases and with the same caveats with which I use Siri's fallback web search. As a synthesist of web search results, an LLM isn't half bad, when it doesn't come as a surprise to be hearing from one at least.
- Kon-Peki 1y ago
  Perhaps you should evaluate in terms of the price premium for speed. Sometimes you buy milk at the 7-eleven instead of the grocery store. It costs more, but is worth it for the convenience in the situation you are currently in. Most of the time it is not. You can buy a used recent PC for a hundred or two, cram it full of memory, and then run a very advanced model. Slowly. But if you are planning to run an agent while you sleep and then review the work in the morning, do you really care if the run time is 4 hours instead of 40 seconds? Most of the time, no. Sometimes, yes.
  - throwanem 1y ago
    The difference is not nearly so stark.
likium 1y ago
We only have access to local models because they're subsidized too. There's nothing to prevent companies or state actors from paying/funding for increased probability and inherent bias. Also local models are close in capabilities now but who knows in a few years what that'll look like.
- throwanem 1y ago
  Eh. Files on my hard disks change when I say. And getting hooked on ChatGPT is like - is exactly like - getting addicted to money. If I benefit less from what I do use, I'll accept the trade of never having that rug yanked out from under me. It looks to me like raw model capabilities are topping out anyway; the engineering around them looks like making more difference in the back half of the decade, and I see nothing essential about that requiring nuclear-powered datacenter pyramids or whatever.
- acoard 1y ago
  > We only have access to local models because they're subsidized too. Yes, and the flow of future models may dry up, but the current local models we'll have forever.

atentaten 1y ago

What does your hardware setup look like?

yzjumper 1y ago
Not the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access. Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API.

haolez 1y ago

How do you do it? Do you host on your hardware or do you use cloud-based providers for open models?

egypturnash 1y ago
You have a very curious definition of "local" if that includes "cloud-based providers".
- jasonjmcghee 1y ago
  > the models I do use remain static and predictable Some people overload "local" a bit to mean you are hosting the model - whether it's on your computer, on your rack, or on your hetzner instance etc. But I think parent is referring to the open/static aspect of the models. If it's hosted by a generic model provider that is serving many users in parallel to reduce the end cost to the user, it's also theoretically a static version of the model... But I could see ad-supported fine tunes being a real problem.
  - deleted 1y ago
    [deleted]
  - throwanem 1y ago
    I am bewildered to hear the sense of "local," in which I and engineers in my experience have for thirty years referred to things which are not remotely hosted, referred to as an "overload."
    - haolez 1y ago
      My definition of local is the same as yours. It's just that my mind got focused on the open models part and I forgot that you specifically mentioned a "local" setup. My bad
    - jasonjmcghee 1y ago
      I thought it was friendlier than "abuse"
      - throwanem 1y ago
        Less absurd, anyway. In what sense does one "abuse" a word to use it in its dictionary definition? You appear to be a good example of where this benighted excuse for an industry is headed. Explain it to me.
- waynecochran 1y ago
  I assume this means, for example, using their own AWS / EC2 instances to store and process -- not "local" geographically, but "local" personally.
  - throwanem 1y ago
    It's a two-year-old Mac Studio in the other room. Where are you coming by this novel sense of 'local?'
politelemon 1y ago
Locally it's pretty simple to run models on GPUs, even low powered ones. Have a look at gpt4all as a starting point but there are plenty of offerings in this space.
yzjumper 1y ago
Not the user above, but I am using the iOS app PrivateLLM when I need offline access or use uncensored models. I use kappa-3-phi-abliterated, models under 6B usually work without crashing. Using Ollama on my Mac Mini 24GB base processor (M4 not M4 pro), I am able to run 7B models. On the mac I am able to set up API access. Funny enough the mac has almost the same processor as my iPhone 16 Pro, so its just a RAM constraint, and of course PrivateLLM does not let you host an API. An M4 Pro would do much better do to the increase in RAM and GPU size.
mansilladev 1y ago
If you have Docker (Desktop) installed, with just a couple of clicks, you can get a local model going on your computer. llama3.2 (3B), llama3.3 (70B), deepseek-r1, and about a dozen others.
- c0nducktr 1y ago
  Do you know if there's a site which will help me pick a model to run locally, based on my system specs and needs? This has been the hardest thing for me to learn and since everything's evolving so quick, what's recommended one week might not be the next.

unshavedyak 1y ago

I agree with this, though i'm still using Claude atm. I figure if we're aware of the downsides you pointed at then we can skip the fast changing landscape of self hosting. It keeps getting cheaper and cheaper to self host, so i'm not sure at what point it makes sense to invest. For me the switching point will probably be when they (AI companies) start the big rug pull. By then my hope is self hosting will be cheaper, better, easier, etc.

throwanem 1y ago
Better not to form the habit, I thought. I'm sure I miss out on some things that way, but that is the lesser risk. I do use the Gemini assistant that came with this Android, in the same cases and with the same caveats with which I use Siri's fallback web search. As a synthesist of web search results, an LLM isn't half bad, when it doesn't come as a surprise to be hearing from one at least.
Kon-Peki 1y ago
Perhaps you should evaluate in terms of the price premium for speed. Sometimes you buy milk at the 7-eleven instead of the grocery store. It costs more, but is worth it for the convenience in the situation you are currently in. Most of the time it is not. You can buy a used recent PC for a hundred or two, cram it full of memory, and then run a very advanced model. Slowly. But if you are planning to run an agent while you sleep and then review the work in the morning, do you really care if the run time is 4 hours instead of 40 seconds? Most of the time, no. Sometimes, yes.
- throwanem 1y ago
  The difference is not nearly so stark.

likium 1y ago

We only have access to local models because they're subsidized too. There's nothing to prevent companies or state actors from paying/funding for increased probability and inherent bias. Also local models are close in capabilities now but who knows in a few years what that'll look like.

throwanem 1y ago
Eh. Files on my hard disks change when I say. And getting hooked on ChatGPT is like - is exactly like - getting addicted to money. If I benefit less from what I do use, I'll accept the trade of never having that rug yanked out from under me. It looks to me like raw model capabilities are topping out anyway; the engineering around them looks like making more difference in the back half of the decade, and I see nothing essential about that requiring nuclear-powered datacenter pyramids or whatever.
acoard 1y ago
> We only have access to local models because they're subsidized too. Yes, and the flow of future models may dry up, but the current local models we'll have forever.