3 ms·

The other thing that's changed from the 'every optimization is premature' era is that shrinking CPUs don't result in big gains in frequency anymore -- Moore's l

The other thing that's changed from the 'every optimization is premature' era is that shrinking CPUs don't result in big gains in frequency anymore -- Moore's law isn't going to make your python run at C speed no matter how long you wait for better hardware.

0x000xca0xfe 4y ago
And the speed of light ensures that memory latencies won't get much better until CPUs are small cubes made of SRAM.
- actionfromafar 4y ago
  Ehrm. Small spheres of SRAM, if I may.
  - speed_spread 4y ago
    That's one way of dealing with corner cases.
- amluto 4y ago
  Come again? Modern servers seem to have about 100ns latency to main memory. The speed of light (actually electrical signals) delay is maybe 1-2ns.
  - 0x000xca0xfe 4y ago
    Thanks for the response, this made me dig around some more in this fascinating topic. I found multiple papers that also simply use 100ns for RAM latency (e.g. this one[1]) but in reality it is a lot more complicated than one number. E.g. the time to first word for DDR3 is just above 6ns[2]. I'm not a CPU engineer but it looks like CPUs don't necessarily wait for the entire cache line and can utilize the first word straight away. Also, the signal needs to travel both ways and the traces are not straight lines. I'm not an electronics engineer either so the best number for the maximum DRAM trace lengths I could find are 12-30cm, which would be 1-2ns indeed. This means that there is some room for improvement (in theory). But even with ideal RAM it is physically impossible to beat register access at 0.17ns (i9-13900KS @ 6GHz) or even L1D cache at 0.7ns[3]. What is causing the extra 4-5ns latency of the RAM? Would it be physically possible to remove it? [1] https://www.usenix.org/system/files/conference/hotcloud18/hotcloud18-paper-thomas.pdf https://www.usenix.org/system/files/conference/hotcloud18/ho... [2] https://en.wikipedia.org/wiki/CAS_latency https://en.wikipedia.org/wiki/CAS_latency [3] https://chipsandcheese.com/2022/11/08/amds-zen-4-part-2-memory-subsystem-and-conclusion/ https://chipsandcheese.com/2022/11/08/amds-zen-4-part-2-memo...
    - pclmulqdq 4y ago
      The extra delay comes from resistance and capacitance in the circuit, which makes electrical signals slow down (this is the ELI5 version). DRAM cells and long wires in the DRAM array are big capacitors and they are driven by circuits with fairly weak output (the resistance). Device physics makes these resistances and capacitances extremely hard to reduce. That is why on-device latency for DRAM has held fairly steady at ~5 ns for a very long time.
    - ummonk 4y ago
      Motherboard manufacturers certainly have to consider DRAM trace lengths when supporting modern DDR5 clock speeds, which are now 6,000 Mhz for top of the line hardware. The CAS latency for RAM is considerably higher and has been decoupled from this - as clock speeds go up CL multipliers go up and the CAS latency ends up being roughly the same - ~10 ns for the best RAM. I'm not sure why CAS latency can't be driven lower. Row selection (tRCD) could be limited by the physics of charging up sense amplifiers, but I don't see why the logic for column address shouldn't be able to run faster. It's probably just something that hasn't been an area of optimization for the simple reason that there are many other much larger sources of latency. And on that note, a CPU won't just jump to accessing some random word from RAM and copying it into a register. It will first look through L1, then L2, and then L3 before seeking the data from RAM. It will then copy an entire cache line into all three caches and load the target word into a register. This entire process results in ~100 ns latency.
- ummonk 4y ago
  Imagine having machine learning model weights just be encoded in blocks of SRAM right on the CPU die.
  - alberth 4y ago
    That already exists. https://youtu.be/GVsUOuSjvcg https://youtu.be/GVsUOuSjvcg
  - astrange 4y ago
    Conversely, you can start putting ALUs and matrix multiplies in the DRAM.
    - sidpatil 4y ago
      This is what I thought they meant when they mentioned "in-memory compute".
cma 4y ago
That and memory latency has improved much slower than everything else, so pointer chasing implicit throughout languages like Python is just horrendously slow. SRAM for bigger cache isn't scaling down anymore either in the last several process nodes.
pjmlp 4y ago
Not at all, but shipping a JIT compiler in the box would make it usable for plenty of workloads, even if C would still be faster.

0x000xca0xfe 4y ago

And the speed of light ensures that memory latencies won't get much better until CPUs are small cubes made of SRAM.

actionfromafar 4y ago
Ehrm. Small spheres of SRAM, if I may.
- speed_spread 4y ago
  That's one way of dealing with corner cases.
amluto 4y ago
Come again? Modern servers seem to have about 100ns latency to main memory. The speed of light (actually electrical signals) delay is maybe 1-2ns.
- 0x000xca0xfe 4y ago
  Thanks for the response, this made me dig around some more in this fascinating topic. I found multiple papers that also simply use 100ns for RAM latency (e.g. this one[1]) but in reality it is a lot more complicated than one number. E.g. the time to first word for DDR3 is just above 6ns[2]. I'm not a CPU engineer but it looks like CPUs don't necessarily wait for the entire cache line and can utilize the first word straight away. Also, the signal needs to travel both ways and the traces are not straight lines. I'm not an electronics engineer either so the best number for the maximum DRAM trace lengths I could find are 12-30cm, which would be 1-2ns indeed. This means that there is some room for improvement (in theory). But even with ideal RAM it is physically impossible to beat register access at 0.17ns (i9-13900KS @ 6GHz) or even L1D cache at 0.7ns[3]. What is causing the extra 4-5ns latency of the RAM? Would it be physically possible to remove it? [1] https://www.usenix.org/system/files/conference/hotcloud18/hotcloud18-paper-thomas.pdf https://www.usenix.org/system/files/conference/hotcloud18/ho... [2] https://en.wikipedia.org/wiki/CAS_latency https://en.wikipedia.org/wiki/CAS_latency [3] https://chipsandcheese.com/2022/11/08/amds-zen-4-part-2-memory-subsystem-and-conclusion/ https://chipsandcheese.com/2022/11/08/amds-zen-4-part-2-memo...
  - pclmulqdq 4y ago
    The extra delay comes from resistance and capacitance in the circuit, which makes electrical signals slow down (this is the ELI5 version). DRAM cells and long wires in the DRAM array are big capacitors and they are driven by circuits with fairly weak output (the resistance). Device physics makes these resistances and capacitances extremely hard to reduce. That is why on-device latency for DRAM has held fairly steady at ~5 ns for a very long time.
  - ummonk 4y ago
    Motherboard manufacturers certainly have to consider DRAM trace lengths when supporting modern DDR5 clock speeds, which are now 6,000 Mhz for top of the line hardware. The CAS latency for RAM is considerably higher and has been decoupled from this - as clock speeds go up CL multipliers go up and the CAS latency ends up being roughly the same - ~10 ns for the best RAM. I'm not sure why CAS latency can't be driven lower. Row selection (tRCD) could be limited by the physics of charging up sense amplifiers, but I don't see why the logic for column address shouldn't be able to run faster. It's probably just something that hasn't been an area of optimization for the simple reason that there are many other much larger sources of latency. And on that note, a CPU won't just jump to accessing some random word from RAM and copying it into a register. It will first look through L1, then L2, and then L3 before seeking the data from RAM. It will then copy an entire cache line into all three caches and load the target word into a register. This entire process results in ~100 ns latency.
ummonk 4y ago
Imagine having machine learning model weights just be encoded in blocks of SRAM right on the CPU die.
- alberth 4y ago
  That already exists. https://youtu.be/GVsUOuSjvcg https://youtu.be/GVsUOuSjvcg
- astrange 4y ago
  Conversely, you can start putting ALUs and matrix multiplies in the DRAM.
  - sidpatil 4y ago
    This is what I thought they meant when they mentioned "in-memory compute".

cma 4y ago

That and memory latency has improved much slower than everything else, so pointer chasing implicit throughout languages like Python is just horrendously slow. SRAM for bigger cache isn't scaling down anymore either in the last several process nodes.

pjmlp 4y ago

Not at all, but shipping a JIT compiler in the box would make it usable for plenty of workloads, even if C would still be faster.