3 ms·

Is shared ram any useful? For instance a mini PC with an AMD 7940HS chip and 64GB of ddr5 ram costs about 800€. At less than half the price of just a GPU, I am

Is shared ram any useful? For instance a mini PC with an AMD 7940HS chip and 64GB of ddr5 ram costs about 800€. At less than half the price of just a GPU, I am not expecting any great results, but is it usable?

pella 3y ago
as I know - only 16GB VRAM allowed in 7940HS ( via BIOS ) so probably you can expect similar results: https://old.reddit.com/r/Amd/comments/15t0lsm/i_turned_a_95_amd_apu_into_a_16gb_vram_gpu_and_it/ https://old.reddit.com/r/Amd/comments/15t0lsm/i_turned_a_95_... HN https://news.ycombinator.com/item?id=37162762 https://news.ycombinator.com/item?id=37162762
maerF0x0 3y ago
I was curious about this topic too because M3 macs have "Unified memory" which is shared amongst their CPU/GPUs. Anyone have a link or explanation of how this works?
- treesciencebot 3y ago
  One of the main bottlenecks for inference is memory bandwith (esp when dealing with huge models, like SD/SDXL) and for that, nothing I know of comes close to matching memory speeds on Apple Silicon (up to 400GB/s).
  - svnt 3y ago
    You can run very large models on a Macbook M2 with 96 GB. They run 1/3 to 1/4 slower in tokens/s than the faster hardware, but they fit in memory. (400 GB/s is a lot in the form factor but the 4090 and equivalent have 1 TB/s, and H100s several times that) Edit: Here someone asked the same question: https://www.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_40906000_vs_m2_max_with_96gb_unified_memory/ https://www.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_409...
    - samspenc 3y ago
      Thanks, I've been trying to find this information - Mac shared memory vs Nvidia VRAM performance differences - for the longest time and your answer and the Reddit link were both super helpful!
      - svnt 3y ago
        You’re welcome — it’s too late for me to edit my wording but hopefully you understood what I meant by “1/3 to 1/4 slower.” That is ambiguous, instead I should have said that models that fit in memory on both take 3-4 times as long on the M2 as they do on the 4090.

pella 3y ago

as I know - only 16GB VRAM allowed in 7940HS ( via BIOS ) so probably you can expect similar results: https://old.reddit.com/r/Amd/comments/15t0lsm/i_turned_a_95_amd_apu_into_a_16gb_vram_gpu_and_it/ https://old.reddit.com/r/Amd/comments/15t0lsm/i_turned_a_95_... HN https://news.ycombinator.com/item?id=37162762 https://news.ycombinator.com/item?id=37162762

maerF0x0 3y ago

I was curious about this topic too because M3 macs have "Unified memory" which is shared amongst their CPU/GPUs. Anyone have a link or explanation of how this works?

treesciencebot 3y ago
One of the main bottlenecks for inference is memory bandwith (esp when dealing with huge models, like SD/SDXL) and for that, nothing I know of comes close to matching memory speeds on Apple Silicon (up to 400GB/s).
- svnt 3y ago
  You can run very large models on a Macbook M2 with 96 GB. They run 1/3 to 1/4 slower in tokens/s than the faster hardware, but they fit in memory. (400 GB/s is a lot in the form factor but the 4090 and equivalent have 1 TB/s, and H100s several times that) Edit: Here someone asked the same question: https://www.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_40906000_vs_m2_max_with_96gb_unified_memory/ https://www.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_409...
  - samspenc 3y ago
    Thanks, I've been trying to find this information - Mac shared memory vs Nvidia VRAM performance differences - for the longest time and your answer and the Reddit link were both super helpful!
    - svnt 3y ago
      You’re welcome — it’s too late for me to edit my wording but hopefully you understood what I meant by “1/3 to 1/4 slower.” That is ambiguous, instead I should have said that models that fit in memory on both take 3-4 times as long on the M2 as they do on the 4090.