4 ms·

Very hopeful for unified/configurable memory. Why are gpu vendors deciding how much ram the gpu gets? This feels so 1980s. GenAI workloads have radically varia

by lumost 3y ago

Very hopeful for unified/configurable memory. Why are gpu vendors deciding how much ram the gpu gets? This feels so 1980s. GenAI workloads have radically variable memory footprints, some folks want 256GB per card so they have a spitting shot at trying something out - and other folks want 8GB per card to maximize throughput. To add complication, I really don’t want to deal with copying 256GB models between cpu/card. We should have unified memory.

brucethemoose2 3y ago
CXL is that, but a compromise. Practically, that means an APU. Intel/AMD are reportedly coming out with wide M1-Pro like chips.
- bick_nyers 3y ago
  At the end of the day it's limited by the PCIE lanes it is attached to, no different from a 2nd GPU. Just saw one yesterday, 128GB PCIE 5.0 x8 which is a maximum of 32 GB/s in and out, where the 4090 has 1 TB/s memory bandwidth. IMO for inference, dual socket EPYC (24 channel DDR5) is the way to go and CXL can theoretically allow you to bump up the bandwidth in that situation (assuming that you can optimize the software properly). Already in llama.cpp there seems to be some issues using 2P servers regarding numa.
smoldesu 3y ago
> Why are gpu vendors deciding how much ram the gpu gets? ...because they make them? You can go buy 256gb systems if you want, at extreme cost. You can also use an 8gb card for whatever you want too. What are they doing wrong? > To add complication, I really don’t want to deal with copying 256GB models between cpu/card. For deployment, it doesn't really matter. You just memory-map it to the CPU or keep it hot-loaded in GPU memory. With a unified memory model you're still limited by your disk speed, so the performance difference probably wouldn't come out great either way.
- lumost 3y ago
  Intel makes microchips, but they do not control how much memory I put in them. There are no 256 GB GPUs available today outside of multi-GPU setups. Would love to see giant NUMA GPU boards.
  - smoldesu 3y ago
    > Would love to see giant NUMA GPU boards. To my knowledge, this is pretty much how InfiniBand works on datacenter-scale systems like the GH200. The incentive and upsell for bringing this to consumer systems seems weak to me.
mschuster91 3y ago
> To add complication, I really don’t want to deal with copying 256GB models between cpu/card. We should have unified memory. The problem is latency. Apple can do what they do in terms of performance because they place the RAM extremely close to the SoC and don't have sockets that present their own RF interference challenges. If you want to run modular, unified RAM, it gets really messy really fast. [1] https://www.ifixit.com/Guide/Macbook+Air+(M2+2022)+Logic+Board+and+Chip+Identification/151816#s316003 https://www.ifixit.com/Guide/Macbook+Air+(M2+2022)+Logic+Boa...

by lumost 3y ago

brucethemoose2 3y ago

CXL is that, but a compromise. Practically, that means an APU. Intel/AMD are reportedly coming out with wide M1-Pro like chips.

bick_nyers 3y ago
At the end of the day it's limited by the PCIE lanes it is attached to, no different from a 2nd GPU. Just saw one yesterday, 128GB PCIE 5.0 x8 which is a maximum of 32 GB/s in and out, where the 4090 has 1 TB/s memory bandwidth. IMO for inference, dual socket EPYC (24 channel DDR5) is the way to go and CXL can theoretically allow you to bump up the bandwidth in that situation (assuming that you can optimize the software properly). Already in llama.cpp there seems to be some issues using 2P servers regarding numa.

smoldesu 3y ago

> Why are gpu vendors deciding how much ram the gpu gets? ...because they make them? You can go buy 256gb systems if you want, at extreme cost. You can also use an 8gb card for whatever you want too. What are they doing wrong? > To add complication, I really don’t want to deal with copying 256GB models between cpu/card. For deployment, it doesn't really matter. You just memory-map it to the CPU or keep it hot-loaded in GPU memory. With a unified memory model you're still limited by your disk speed, so the performance difference probably wouldn't come out great either way.

lumost 3y ago
Intel makes microchips, but they do not control how much memory I put in them. There are no 256 GB GPUs available today outside of multi-GPU setups. Would love to see giant NUMA GPU boards.
- smoldesu 3y ago
  > Would love to see giant NUMA GPU boards. To my knowledge, this is pretty much how InfiniBand works on datacenter-scale systems like the GH200. The incentive and upsell for bringing this to consumer systems seems weak to me.

mschuster91 3y ago

> To add complication, I really don’t want to deal with copying 256GB models between cpu/card. We should have unified memory. The problem is latency. Apple can do what they do in terms of performance because they place the RAM extremely close to the SoC and don't have sockets that present their own RF interference challenges. If you want to run modular, unified RAM, it gets really messy really fast. [1] https://www.ifixit.com/Guide/Macbook+Air+(M2+2022)+Logic+Board+and+Chip+Identification/151816#s316003 https://www.ifixit.com/Guide/Macbook+Air+(M2+2022)+Logic+Boa...