3 ms·

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

LatticeAnimal 3y ago
> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it? I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat). But there is also no reason for any of the GPU manufacturers to do this. A skew with twice as much memory can go for a lot more than the difference in memory cost alone
- SunlitCat 3y ago
  And especially doing "interesting" combinations of gpu and memory. Like lower end gpu with 16 GB of VRAM, but offering just 8 / 12 GB of VRAM in the middle class and then again 16 GB in the upper class of gpu selection.
- ItsBob 3y ago
  I don't disagree but (I know nothing about this btw...) would it not benefit in terms of, say, a L3 cache kind of thing? Imagine you could stick 2 x 64GB DDR5 DIMMS on the GPU in sockets, would that not be faster to access than the motherboard DIMMS? It won't be as fast as on-die memory of course but could it not act like a sort of halfway house?
jsheard 3y ago
Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting around it, the bandwidth that GPUs crave just isn't compatible with modularity. Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.
- wtallis 3y ago
  FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.
  - osigurdson 3y ago
    I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).
    - MobiusHorizons 3y ago
      Gpu ram is typically gddr6 or gddr6x which is a different standard to the chips used in ddr5 for example. GPUs have terrible latency to ram, but enormous throughput, and I assume the chips are internally optimized for that. Many aspects of a design change when you choose different latency or clockspeed targets translating into different power / area calculations.
- tverbeure 3y ago
  For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.
  - dsvf 3y ago
    [dead]
- viraptor 3y ago
  That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial. It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-your-2070/ https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)
tverbeure 3y ago
Check out PCB back drilling. It's a process where you remove a few hundred microns from the vias that are used to connect GDDR RAMs to the GPUs, to avoid reflections due to the impedance mismatch that's caused by the stub. When you have a pulse coded signal traveling at close to 10GHz, everything becomes an antenna. The technical problem is that you can't do this with a flimsy connector like the ones used for DIMMs. The reason GDDR can have a bandwidth per pin that is 4 times higher than regular DDR is because they are soldered down on the PCB.
cwegener 3y ago
[dead]
hahnchen 3y ago
> It almost hurts, to read that PyTorch is faster. Why?
theGeatZhopa 3y ago
NVIDIA hates that trick.

LatticeAnimal 3y ago

> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it? I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat). But there is also no reason for any of the GPU manufacturers to do this. A skew with twice as much memory can go for a lot more than the difference in memory cost alone

SunlitCat 3y ago
And especially doing "interesting" combinations of gpu and memory. Like lower end gpu with 16 GB of VRAM, but offering just 8 / 12 GB of VRAM in the middle class and then again 16 GB in the upper class of gpu selection.
ItsBob 3y ago
I don't disagree but (I know nothing about this btw...) would it not benefit in terms of, say, a L3 cache kind of thing? Imagine you could stick 2 x 64GB DDR5 DIMMS on the GPU in sockets, would that not be faster to access than the motherboard DIMMS? It won't be as fast as on-die memory of course but could it not act like a sort of halfway house?

jsheard 3y ago

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting around it, the bandwidth that GPUs crave just isn't compatible with modularity. Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.

wtallis 3y ago
FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.
- osigurdson 3y ago
  I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).
  - MobiusHorizons 3y ago
    Gpu ram is typically gddr6 or gddr6x which is a different standard to the chips used in ddr5 for example. GPUs have terrible latency to ram, but enormous throughput, and I assume the chips are internally optimized for that. Many aspects of a design change when you choose different latency or clockspeed targets translating into different power / area calculations.
tverbeure 3y ago
For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.
- dsvf 3y ago
  [dead]
viraptor 3y ago
That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial. It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-your-2070/ https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)

tverbeure 3y ago

Check out PCB back drilling. It's a process where you remove a few hundred microns from the vias that are used to connect GDDR RAMs to the GPUs, to avoid reflections due to the impedance mismatch that's caused by the stub. When you have a pulse coded signal traveling at close to 10GHz, everything becomes an antenna. The technical problem is that you can't do this with a flimsy connector like the ones used for DIMMs. The reason GDDR can have a bandwidth per pin that is 4 times higher than regular DDR is because they are soldered down on the PCB.

cwegener 3y ago

[dead]

hahnchen 3y ago

> It almost hurts, to read that PyTorch is faster. Why?

theGeatZhopa 3y ago

NVIDIA hates that trick.