3 ms·
One of the main bottlenecks for inference is memory bandwith (esp when dealing with huge models, like SD/SDXL) and for that, nothing I know of comes close to ma
by treesciencebot 3y ago
One of the main bottlenecks for inference is memory bandwith (esp when dealing with huge models, like SD/SDXL) and for that, nothing I know of comes close to matching memory speeds on Apple Silicon (up to 400GB/s).
- svnt 3y agoYou can run very large models on a Macbook M2 with 96 GB. They run 1/3 to 1/4 slower in tokens/s than the faster hardware, but they fit in memory. (400 GB/s is a lot in the form factor but the 4090 and equivalent have 1 TB/s, and H100s several times that) Edit: Here someone asked the same question: https://www.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_40906000_vs_m2_max_with_96gb_unified_memory/ https://www.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_409...
- samspenc 3y agoThanks, I've been trying to find this information - Mac shared memory vs Nvidia VRAM performance differences - for the longest time and your answer and the Reddit link were both super helpful!
- svnt 3y agoYou’re welcome — it’s too late for me to edit my wording but hopefully you understood what I meant by “1/3 to 1/4 slower.” That is ambiguous, instead I should have said that models that fit in memory on both take 3-4 times as long on the M2 as they do on the 4090.