6 ms·

> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've

> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.

pennaMan 3y ago
I can run 4bit on a beat up 1070 ti. GP talks about higher precision models
- sp332 3y ago
  You wouldn’t be able to fit the whole model into 8GB VRAM. It’s faster than not using a GPU at all, but most of it would still be computed on the CPU.
  - baq 3y ago
    IME ollama ran mixtral on a 1070 fast enough.
    - dimask 3y ago
      Though it most probably does not run in on the 1070 but rather on the cpu. It cannot fit on a 1070, it is not about speed, a 1070 cannot run it period.
      - Dkuku 3y ago
        In llama.cpp You can offload some of the layers to gpu with -ngl X. Where x is the number of layers
mkesper 3y ago
Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).
- jacooper 3y ago
  Wait ROCm support Ryzen APUs and still doesn't support dedicatedly GPUs like the 6700XT?!
  - jacooper 3y ago
    Supports* dedicated*
    - trissi1996 3y ago
      While not being officially supported, rocm runs just fine on my 6700XT, i just have to set an env var(export HSA_OVERRIDE_GFX_VERSION=10.3.0)
      - jacooper 3y ago
        Really? Does everything run? Even AI stuff? Do you have any links where I can read more about that?
        trissi1996 3y ago
        Everything I've tried to get running, worked quite smoothly. Although I only tried LLMs via llama-cpp and stable diffusion via ComfyUI. I don't see any reason why other AI stuff wouldn't work as long as it supports rocm. Also I only tried it on linux, AFAIK windows is a lot more difficult to get running, if it works at all... With llama-cpp, I successfully tried various LLMs(e.g. LLAMA 13B, Mixtral etc) with very solid performance. Even for models that don't fit in VRAM completely, performance can be surprisingly solid, as long as you compile with AVX extensions. (and your CPU supports those) Stable Diffusion via ComfyUI also works very well. However, be aware of VRAM limitations with the larger SDXL variants, especially when running a heavy desktop environment. Regarding setup guides/links, there isn't a good centralized resource sadly, so some tinkering is needed. Unlike some of those CUDA 1-click solutions, ROCm requires more manual setup, especially for the models only unofficially supported. Here are a couple of links that might be helpful: https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setup_for_using_rocm_with_rx_6700xt_gpu_on/ https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setu... https://old.reddit.com/r/StableDiffusion/comments/ww436j/howto_stable_diffusion_on_an_amd_gpu/ https://old.reddit.com/r/StableDiffusion/comments/ww436j/how... https://rentry.org/eq3hg https://rentry.org/eq3hg In general the r/localllama & r/StableDiffusion subreddits are good places to search for info.
asimpleusecase 3y ago
Did you do anything special to make that work? Is it useful? Or just a toy?
- EarthLaunch 3y ago
  I've been using it for 'easy' queries like syntax/parameter questions, in place of ChatGPT 4. It's great for that. I am using a ~48GB version.
- windexh8er 3y ago
  I have a 14" MBP with an M1 Max and 64GB. The M3 won't really make a difference, but the RAM, since unified, is huge. I can run most models on this machine with realtime performance compared to a Ryzen 7735HS and 64GB (DDR5). Now I'm not saying the Ryzen setup should be good, but the M1 architecture just makes it a much better option. I could add an eGPU to the Ryzen system and it could likely do better, but would also exceed the price point and portability.
  - paulmd 3y ago
    it's not just that it's huge and unified - ryzen APUs obviously can have 2x32GB SODIMMs put in them and they support unified memory too. the difference is the bandwidth and the computational power of the APU. M1 Max is roughly similar to a PS5 in terms of overall system design (shader configuration and bandwidth) plus has dedicated AI inference units already (which won't be added to consoles until PS5 Pro launches with RDNA 3.5). It is far more bandwidth than you can get out of a socketed-memory laptop system. https://twitter.com/Locuza_/status/1450271726827413508 https://twitter.com/Locuza_/status/1450271726827413508 To support that level of performance in a socketed-memory system you will need an extra layer of caching added to the processor to supplement the bandwidth - and maybe still need to go to quad-channel. Those products are Strix and Strix Halo and should be hitting the market over the next year or two but the reality is that the M1 Max was an absurdly powerful laptop, far more potent than even the first-gen 5nm laptops for x86 let alone the other junk you could buy in 2020. This is the problem with the discourse around apple silicon for the last few years: yeah, they're expensive, but even a loaded-out x86 laptop doesn't get you the same capabilities. Even if the x86 is competitive in some particular benchmark on iso-node you are probably spending more power to do it, and the x86 product comes years after the apple product, and still has a much weaker gpu and less bandwidth (which doesn't just matter for GPU, it matters for compiling and JIT too). It is incredibly silly to look back on the discourse in 2020-2023 around apple silicon, a lot of reviewers made extremely silly claims about how "even 7nm x86 processors were already competitive with apple silicon" and as the ecosystems have matured it is obvious that even 5nm processors are not quite competitive yet. And they dumped on the SPEC tests and Geekbench that measured this properly, in favor of dumb things like cinebench R23 and so on (it's always cinebench used for this dumb shit tbh, CB R13/R15 were hugely misleading at the zen1 launch too). Let alone things like, you know, compiling or JVM/node workloads...) (similarly, gotta love the vibe a few years ago of: "threadripper vs mac pro" - did you know that a 64C threadripper with 256GB RAM is actually cheaper than a mac pro loaded out with 2TB!? waow, who knew systems with an order of magnitude less capacity would be cheaper!? https://youtu.be/BH291DQRIOg https://youtu.be/BH291DQRIOg )

pennaMan 3y ago

I can run 4bit on a beat up 1070 ti. GP talks about higher precision models

sp332 3y ago
You wouldn’t be able to fit the whole model into 8GB VRAM. It’s faster than not using a GPU at all, but most of it would still be computed on the CPU.
- baq 3y ago
  IME ollama ran mixtral on a 1070 fast enough.
  - dimask 3y ago
    Though it most probably does not run in on the 1070 but rather on the cpu. It cannot fit on a 1070, it is not about speed, a 1070 cannot run it period.
    - Dkuku 3y ago
      In llama.cpp You can offload some of the layers to gpu with -ngl X. Where x is the number of layers

mkesper 3y ago

Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).

jacooper 3y ago
Wait ROCm support Ryzen APUs and still doesn't support dedicatedly GPUs like the 6700XT?!
- jacooper 3y ago
  Supports* dedicated*
  - trissi1996 3y ago
    While not being officially supported, rocm runs just fine on my 6700XT, i just have to set an env var(export HSA_OVERRIDE_GFX_VERSION=10.3.0)
    - jacooper 3y ago
      Really? Does everything run? Even AI stuff? Do you have any links where I can read more about that?
      - trissi1996 3y ago
        Everything I've tried to get running, worked quite smoothly. Although I only tried LLMs via llama-cpp and stable diffusion via ComfyUI. I don't see any reason why other AI stuff wouldn't work as long as it supports rocm. Also I only tried it on linux, AFAIK windows is a lot more difficult to get running, if it works at all... With llama-cpp, I successfully tried various LLMs(e.g. LLAMA 13B, Mixtral etc) with very solid performance. Even for models that don't fit in VRAM completely, performance can be surprisingly solid, as long as you compile with AVX extensions. (and your CPU supports those) Stable Diffusion via ComfyUI also works very well. However, be aware of VRAM limitations with the larger SDXL variants, especially when running a heavy desktop environment. Regarding setup guides/links, there isn't a good centralized resource sadly, so some tinkering is needed. Unlike some of those CUDA 1-click solutions, ROCm requires more manual setup, especially for the models only unofficially supported. Here are a couple of links that might be helpful: https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setup_for_using_rocm_with_rx_6700xt_gpu_on/ https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setu... https://old.reddit.com/r/StableDiffusion/comments/ww436j/howto_stable_diffusion_on_an_amd_gpu/ https://old.reddit.com/r/StableDiffusion/comments/ww436j/how... https://rentry.org/eq3hg https://rentry.org/eq3hg In general the r/localllama & r/StableDiffusion subreddits are good places to search for info.

asimpleusecase 3y ago

Did you do anything special to make that work? Is it useful? Or just a toy?

EarthLaunch 3y ago
I've been using it for 'easy' queries like syntax/parameter questions, in place of ChatGPT 4. It's great for that. I am using a ~48GB version.
windexh8er 3y ago
I have a 14" MBP with an M1 Max and 64GB. The M3 won't really make a difference, but the RAM, since unified, is huge. I can run most models on this machine with realtime performance compared to a Ryzen 7735HS and 64GB (DDR5). Now I'm not saying the Ryzen setup should be good, but the M1 architecture just makes it a much better option. I could add an eGPU to the Ryzen system and it could likely do better, but would also exceed the price point and portability.
- paulmd 3y ago
  it's not just that it's huge and unified - ryzen APUs obviously can have 2x32GB SODIMMs put in them and they support unified memory too. the difference is the bandwidth and the computational power of the APU. M1 Max is roughly similar to a PS5 in terms of overall system design (shader configuration and bandwidth) plus has dedicated AI inference units already (which won't be added to consoles until PS5 Pro launches with RDNA 3.5). It is far more bandwidth than you can get out of a socketed-memory laptop system. https://twitter.com/Locuza_/status/1450271726827413508 https://twitter.com/Locuza_/status/1450271726827413508 To support that level of performance in a socketed-memory system you will need an extra layer of caching added to the processor to supplement the bandwidth - and maybe still need to go to quad-channel. Those products are Strix and Strix Halo and should be hitting the market over the next year or two but the reality is that the M1 Max was an absurdly powerful laptop, far more potent than even the first-gen 5nm laptops for x86 let alone the other junk you could buy in 2020. This is the problem with the discourse around apple silicon for the last few years: yeah, they're expensive, but even a loaded-out x86 laptop doesn't get you the same capabilities. Even if the x86 is competitive in some particular benchmark on iso-node you are probably spending more power to do it, and the x86 product comes years after the apple product, and still has a much weaker gpu and less bandwidth (which doesn't just matter for GPU, it matters for compiling and JIT too). It is incredibly silly to look back on the discourse in 2020-2023 around apple silicon, a lot of reviewers made extremely silly claims about how "even 7nm x86 processors were already competitive with apple silicon" and as the ecosystems have matured it is obvious that even 5nm processors are not quite competitive yet. And they dumped on the SPEC tests and Geekbench that measured this properly, in favor of dumb things like cinebench R23 and so on (it's always cinebench used for this dumb shit tbh, CB R13/R15 were hugely misleading at the zen1 launch too). Let alone things like, you know, compiling or JVM/node workloads...) (similarly, gotta love the vibe a few years ago of: "threadripper vs mac pro" - did you know that a 64C threadripper with 256GB RAM is actually cheaper than a mac pro loaded out with 2TB!? waow, who knew systems with an order of magnitude less capacity would be cheaper!? https://youtu.be/BH291DQRIOg https://youtu.be/BH291DQRIOg )