4 ms·
Apple kinda does something like this with their GPU. They put it on the same chip as their CPU, added RAM very close to it with a lot of bandwidth and then use
by hmottestad 3y ago
Apple kinda does something like this with their GPU. They put it on the same chip as their CPU, added RAM very close to it with a lot of bandwidth and then use the RAM interchangeably between the CPU and GPU. Downside is that you can’t upgrade the RAM since it’s all stuck together in the same package as the CPU/GPU. It does allow you to get a CPU/GPU with 192GB of memory for pennies compared to something similar from Nvidia, but definitely not going to be as fast.
I don’t think the GPU can address all the memory though, I know that the 128GB is limited to 96GB that can be shared with the GPU.
- hmottestad 3y agoYou can get two NVIDIA A40 cards with 48GB of memory each, second hand on eBay for 12 000 USD. You can connect them with NVLink to get 96GB effectively. Or you could get a Mac Studio for 4 800 USD.
- codedokode 3y agoAMD (and Intel) has CPUs with integrated GPUs that use system memory. Cannot one use them as a reasonably priced alternative? I think that buying 64 Gb of ordinary memory is much cheaper than buying Apple's proprietary memory.
- codedokode 3y agoNot only Apple does that. Intel and AMD have CPUs with integrated GPUs that do not have their own VRAM and use system memory. Why cannot they be used for LLMs? They don't cost an arm and a leg.
- jamesTee49 3y ago[dead]
- hmottestad 3y agoIntel Iris Xe can use half of the available memory. I don’t think it will help much though, the GPU is not very powerful and the bandwidth is around 90 GB/s. The M2 Ultra in the Mac Studio has 800 GB/s. The Nvidia A40 has 696 GB/s while the Nvidia H100 SMX has 3.35 TB/s, which is probably the best you could get today. https://www.intel.com/content/www/us/en/support/articles/000020962/graphics.html https://www.intel.com/content/www/us/en/support/articles/000...