3 ms·

Github Copilot had an outage for me this morning. It was kind of shocking. I now believe this metric. :-) I'll be looking into ways of running a local LLM for

by lysace 2y ago

Github Copilot had an outage for me this morning. It was kind of shocking. I now believe this metric. :-) I'll be looking into ways of running a local LLM for this purpose (code assistance in VS Code). I'm already really impressed with various quite large models running on my 32 GB Mac Studio M2 Max via Ollama. It feels like having a locally running chatgpt.

kulahan 2y ago
I'm very happy to hear this; maybe it's finally time to buy a ton of ram for my PC! A local, private LLM would be great. I'd try talking to it about stuff I don't feel comfortable being on OpenAI's servers.
- lysace 2y ago
  Getting lots of ram will let you run large models on the CPU, but it will be so slow. The Apple Silicon Macs have this shared memory between CPU and GPU that let's the (relatively underpowered GPU, compared to a decent Nvidia GPU) run these models at decent speeds, compared with a CPU, when using llama.cpp. This should all get dramatically better/faster/cheaper within a few years, I suspect. Capitalism will figure this one out.
  - kulahan 2y ago
    Interesting, so this is a Mac-specific solution? That's pretty cool. I assume, then, that the primary goal would be to drop in the beefiest GPU possible when on windows/linux?
    - lysace 2y ago
      With Windows/Linux I think the issue is that NVidia is artificially limiting the amount of onboard RAM (they want to sell those devices for 10x more to openai, etc) and that AMD for whatever reason can't get their shit together. I'm sure that there are other much more knowledgeable people here though, on this topic.
      - rustcleaner 2y ago
        This is why the DMCA must be repealed.
    - evilduck 2y ago
      There's nothing Mac specific about running LLMs locally, they just happen to be a convenient way to get a ton of VRAM in a single small power efficient package. In Windows and Linux, yes you'll want at least 12GB of VRAM to have much of any utility but the beefiest consumer GPUs are still topping out at 24GB which is still pretty limiting.
evoke4908 2y ago
Ollama, docker and "open webui". It immediately works out of the box and that's it. I've been using local LLMs on my laptop for a while, it's pretty nice. The only thing you really need to worry about is VRAM. Make sure your GPU has enough memory to run your model and that's pretty much it. Also "open webui" is the worst project name I've ever seen.

by lysace 2y ago

kulahan 2y ago

I'm very happy to hear this; maybe it's finally time to buy a ton of ram for my PC! A local, private LLM would be great. I'd try talking to it about stuff I don't feel comfortable being on OpenAI's servers.

lysace 2y ago
Getting lots of ram will let you run large models on the CPU, but it will be so slow. The Apple Silicon Macs have this shared memory between CPU and GPU that let's the (relatively underpowered GPU, compared to a decent Nvidia GPU) run these models at decent speeds, compared with a CPU, when using llama.cpp. This should all get dramatically better/faster/cheaper within a few years, I suspect. Capitalism will figure this one out.
- kulahan 2y ago
  Interesting, so this is a Mac-specific solution? That's pretty cool. I assume, then, that the primary goal would be to drop in the beefiest GPU possible when on windows/linux?
  - lysace 2y ago
    With Windows/Linux I think the issue is that NVidia is artificially limiting the amount of onboard RAM (they want to sell those devices for 10x more to openai, etc) and that AMD for whatever reason can't get their shit together. I'm sure that there are other much more knowledgeable people here though, on this topic.
    - rustcleaner 2y ago
      This is why the DMCA must be repealed.
  - evilduck 2y ago
    There's nothing Mac specific about running LLMs locally, they just happen to be a convenient way to get a ton of VRAM in a single small power efficient package. In Windows and Linux, yes you'll want at least 12GB of VRAM to have much of any utility but the beefiest consumer GPUs are still topping out at 24GB which is still pretty limiting.

evoke4908 2y ago

Ollama, docker and "open webui". It immediately works out of the box and that's it. I've been using local LLMs on my laptop for a while, it's pretty nice. The only thing you really need to worry about is VRAM. Make sure your GPU has enough memory to run your model and that's pretty much it. Also "open webui" is the worst project name I've ever seen.