3 ms·

Great to find this narrow focused thing: > We support the following backends: Metal is our primary target. Starting from MacBooks with 96GB of RAM. NV

Great to find this narrow focused thing: > We support the following backends: Metal is our primary target. Starting from MacBooks with 96GB of RAM. NVIDIA CUDA with special care for the DGX Spark. AMD ROCm is only supported in the rocm branch. It is kept separate from main since I (antirez) don't have direct hardware access, so the community rebases the branch as needed. > This project would not exist without llama.cpp and GGML, make sure to read the acknowledgements section, a big thank you to Georgi Gerganov and all the other contributors. Edit: aww, doesn't seem to support offloading to system RAM[0] (yet) [0] https://github.com/antirez/ds4/issues/108 https://github.com/antirez/ds4/issues/108 Guess I'll have to keep watching the llama.cpp issue[1] [1] https://github.com/ggml-org/llama.cpp/issues/22319 https://github.com/ggml-org/llama.cpp/issues/22319

keyle 5mo ago
If only you could still buy Mac's with that much RAM
- shric 5mo ago
  You can buy 128GB M5 MacBook Pros? Configured one just now, delivers in 2 weeks
  - keyle 5mo ago
    Interesting there were news last week or so of apple removing Mac minis options.
    - littlecranky67 5mo ago
      They removed the baseline 8GB RAM/256GBstorage model. My bet is with increased RAM prices the markup on the lower end is not enough to still make a profit
      - a1o 5mo ago
        baseline was 16GB RAM
zimmerfrei 5mo ago
> AMD ROCm is only supported in the rocm branch. Has anybody tried it? There is a lot of emphasis on MacBook Pro in this thread, but I would like to use it with an AMD Halo Strix with 128GB of unified RAM.
- jaimeyap 5mo ago
  I just got the rocm branch compiled and running. Starting with one of the common strix halo rocm toolboxes, just needed to install a few more dependencies to get the repo to build. So far just tried the q2-imatrix model and I'm seeing ~7.32tok/s with a locally bound claude code session. It's pretty unusably slow for agentic coding like this - with it being tens of minutes per round of thinking. But it does seem to be working. Suspiciously amdgpu_top is only showing ~16GB of memory being used. Not sure if this is somehow misreading that.

keyle 5mo ago

If only you could still buy Mac's with that much RAM

shric 5mo ago
You can buy 128GB M5 MacBook Pros? Configured one just now, delivers in 2 weeks
- keyle 5mo ago
  Interesting there were news last week or so of apple removing Mac minis options.
  - littlecranky67 5mo ago
    They removed the baseline 8GB RAM/256GBstorage model. My bet is with increased RAM prices the markup on the lower end is not enough to still make a profit
    - a1o 5mo ago
      baseline was 16GB RAM

zimmerfrei 5mo ago

> AMD ROCm is only supported in the rocm branch. Has anybody tried it? There is a lot of emphasis on MacBook Pro in this thread, but I would like to use it with an AMD Halo Strix with 128GB of unified RAM.

jaimeyap 5mo ago
I just got the rocm branch compiled and running. Starting with one of the common strix halo rocm toolboxes, just needed to install a few more dependencies to get the repo to build. So far just tried the q2-imatrix model and I'm seeing ~7.32tok/s with a locally bound claude code session. It's pretty unusably slow for agentic coding like this - with it being tens of minutes per round of thinking. But it does seem to be working. Suspiciously amdgpu_top is only showing ~16GB of memory being used. Not sure if this is somehow misreading that.