3 ms·

It’s not the vram number that’s the problem on windows side. It’s the throughput. For whatever reason they’re a good 5-10x lower. Even if it’s a large amount

by Havoc 2y ago

It’s not the vram number that’s the problem on windows side. It’s the throughput. For whatever reason they’re a good 5-10x lower. Even if it’s a large amount and gpu accessible if the throughput sucks that badly then 70B models aren’t usable What puzzles me is that the even the newer AMD APUs have this issue. The upcoming strix halo is rumoured to be around 140 ish which puts it at very bottom end of Apple devices

hnuser123456 2y ago
Apple silicon RAM is basically all VRAM, with 150-400 GB/s bandwidth. Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth. Nvidia cards are around 1 TB/s, but models with 40+GB of VRAM are more expensive than macs with equivalent RAM capacity. I would expect that an Nvidia card on Windows would be around twice as fast as apple silicon if you have a model with 40+GB, and from some quick research this seems to be true. edit: "Any Macbook 64GB can run it at about 6-7 Tok/s using Q4 quantization. 2x3090 can do it much faster at about 18 tok/s" https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment/m135clq/ https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment...
- Havoc 2y ago
  > Apple silicon RAM is basically all VRAM The actual ram is lpddr5 - miles slower than what you’d find in even very old dedicated GPUs It’s as you say though - for some tasks like LLMs that doesn’t matter if you’ve got enough bandwidth to said slow ram…ends up being almost the same thing as having actual fast ram. It’s a neat trick & little bit frustrated that nobody is copying it. Cause using dirt cheap lpddr5 (about a dollar per gig) but paying Apple’s comical ram upgrade prices feels like a cruel joke.
  - hnuser123456 2y ago
    Ah, so do they use 2 channels on standard / 4 on pro / 8 on max? Rumors are there may be a X3D Threadripper in the works, would be nice to not have to choose between RAM bandwidth and IPC again.

by Havoc 2y ago

hnuser123456 2y ago

Apple silicon RAM is basically all VRAM, with 150-400 GB/s bandwidth. Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth. Nvidia cards are around 1 TB/s, but models with 40+GB of VRAM are more expensive than macs with equivalent RAM capacity. I would expect that an Nvidia card on Windows would be around twice as fast as apple silicon if you have a model with 40+GB, and from some quick research this seems to be true. edit: "Any Macbook 64GB can run it at about 6-7 Tok/s using Q4 quantization. 2x3090 can do it much faster at about 18 tok/s" https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment/m135clq/ https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment...

Havoc 2y ago
> Apple silicon RAM is basically all VRAM The actual ram is lpddr5 - miles slower than what you’d find in even very old dedicated GPUs It’s as you say though - for some tasks like LLMs that doesn’t matter if you’ve got enough bandwidth to said slow ram…ends up being almost the same thing as having actual fast ram. It’s a neat trick & little bit frustrated that nobody is copying it. Cause using dirt cheap lpddr5 (about a dollar per gig) but paying Apple’s comical ram upgrade prices feels like a cruel joke.
- hnuser123456 2y ago
  Ah, so do they use 2 channels on standard / 4 on pro / 8 on max? Rumors are there may be a X3D Threadripper in the works, would be nice to not have to choose between RAM bandwidth and IPC again.