3 ms·
Apple silicon RAM is basically all VRAM, with 150-400 GB/s bandwidth. Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth. Nvidia cards
by hnuser123456 2y ago
Apple silicon RAM is basically all VRAM, with 150-400 GB/s bandwidth.
Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth.
Nvidia cards are around 1 TB/s, but models with 40+GB of VRAM are more expensive than macs with equivalent RAM capacity.
I would expect that an Nvidia card on Windows would be around twice as fast as apple silicon if you have a model with 40+GB, and from some quick research this seems to be true.
edit:
"Any Macbook 64GB can run it at about 6-7 Tok/s using Q4 quantization. 2x3090 can do it much faster at about 18 tok/s"
https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment/m135clq/ https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment...
- Havoc 2y ago> Apple silicon RAM is basically all VRAM The actual ram is lpddr5 - miles slower than what you’d find in even very old dedicated GPUs It’s as you say though - for some tasks like LLMs that doesn’t matter if you’ve got enough bandwidth to said slow ram…ends up being almost the same thing as having actual fast ram. It’s a neat trick & little bit frustrated that nobody is copying it. Cause using dirt cheap lpddr5 (about a dollar per gig) but paying Apple’s comical ram upgrade prices feels like a cruel joke.
- hnuser123456 2y agoAh, so do they use 2 channels on standard / 4 on pro / 8 on max? Rumors are there may be a X3D Threadripper in the works, would be nice to not have to choose between RAM bandwidth and IPC again.