3 ms·

> Have we really progressed memory speeds to the point that reading memory is faster than execution speed on the processor? That just feels wrong. Sort of. Ra

> Have we really progressed memory speeds to the point that reading memory is faster than execution speed on the processor? That just feels wrong. Sort of. Random main memory access speeds aren't there, but L1 is; as long as your memory accesses are sequential, proper coding can stream sequential main memory into L1 and keep the CPU working. But it's effectively a lost art - the vast majority of programs spend a significant part of their time stalling on memory access.

taeric 8y ago
That doesn't follow. Yes, the cache is fast enough, but assuming you really are using every cycle, eventually the CPU will catch up, no? My metaphor is a tub. If the drain is faster than the faucet, filling the tub before opening the drain will still see an empty tub. Only way you can prevent that is if the faucet is the same speed as the drain. Right? Now, again, the cache can help you see more time where you are getting max utilization. Hard to believe it would be a full workload, though.
- beagle3 8y ago
  Main memory has enough bandwidth but not enough latency. If your code is written to account for that, then - yes, your CPU will be your bottleneck. If it isn’t then your CPU will be waiting for memory. DDR4 can do >20GB/s but the latency is >10ns. If you prefetch enough cycles in advance, data will be in L1 by the time you need it. If you don’t, it’s won’t.
  - taeric 8y ago
    And that is still surprising to me. Quickly checking the internet, I see [1] which lists the throughput of an i7 is about 25 GB/s. That said, I see the newer specs report a GT/s, which is new to me. Looks like those are much lower. Though, I suspect I'm just not looking at the right specs. Regardless, I've learned that RAM has gotten much faster than I would expect. Cool to see. [1] https://ark.intel.com/products/83503/Intel-Core-i7-4980HQ-Processor-6M-Cache-up-to-4_00-GHz https://ark.intel.com/products/83503/Intel-Core-i7-4980HQ-Pr...
    - beagle3 8y ago
      The 25GB/s corresponds to the fastest speed a DDR4 can send memory. That said, any nontrivial computation is unlikely to keep up with that speed (e.g. 64-bit data will be ~3GD/s, so whatever computation you do has to fit in amortized 1 cycle on a 3Ghz clock, or the CPU is the bottleneck) GD=GigaDatum
      - taeric 8y ago
        I'm curious how much of a contributing factor this is to why many deep learning pipelines use smaller floats. Specifically, I "knew" that the throughput some math operations was actually greater than 1. That said, I have not kept up with which operations. I would not have expected the floating point units to be the fast ones. Again, thanks for sticking with me in this thread. I learned a bit. Hoping to come up with ways to play with this knowledge. :)
    - srean 8y ago
      Curious whether memory controllers can stripe bytes across slots much like RAID can do for disks.
      - mmt 8y ago
        I don't think it's bytewise (just like nobody does RAID3), but, yes, they can and do. The keyword you're looking for is "interleaving". I think it's also not even by DIMM slot but by channel, so if the channels are unevenly populated (e.g. 2 slots filled on one channel but only 1 on another), interleaving is disabled.
- mmt 8y ago
  To stretch the tub metaphor to include latency, you have to imagine someone turning the faucet off and on (or plugging and unplugging the drain) depending on how full the tub is.. and, with memory, there may be one spigot (the bandwidth), but a tremendous number of valve knobs (addresses), such that the someone has to decide the correct knob to turn before the tub empties out completely. If the wrong valve is opened, no water comes out (the wrong data is loaded into cache, which is the same as no data being loaded), so merely guessing is no good.

taeric 8y ago

That doesn't follow. Yes, the cache is fast enough, but assuming you really are using every cycle, eventually the CPU will catch up, no? My metaphor is a tub. If the drain is faster than the faucet, filling the tub before opening the drain will still see an empty tub. Only way you can prevent that is if the faucet is the same speed as the drain. Right? Now, again, the cache can help you see more time where you are getting max utilization. Hard to believe it would be a full workload, though.

beagle3 8y ago
Main memory has enough bandwidth but not enough latency. If your code is written to account for that, then - yes, your CPU will be your bottleneck. If it isn’t then your CPU will be waiting for memory. DDR4 can do >20GB/s but the latency is >10ns. If you prefetch enough cycles in advance, data will be in L1 by the time you need it. If you don’t, it’s won’t.
- taeric 8y ago
  And that is still surprising to me. Quickly checking the internet, I see [1] which lists the throughput of an i7 is about 25 GB/s. That said, I see the newer specs report a GT/s, which is new to me. Looks like those are much lower. Though, I suspect I'm just not looking at the right specs. Regardless, I've learned that RAM has gotten much faster than I would expect. Cool to see. [1] https://ark.intel.com/products/83503/Intel-Core-i7-4980HQ-Processor-6M-Cache-up-to-4_00-GHz https://ark.intel.com/products/83503/Intel-Core-i7-4980HQ-Pr...
  - beagle3 8y ago
    The 25GB/s corresponds to the fastest speed a DDR4 can send memory. That said, any nontrivial computation is unlikely to keep up with that speed (e.g. 64-bit data will be ~3GD/s, so whatever computation you do has to fit in amortized 1 cycle on a 3Ghz clock, or the CPU is the bottleneck) GD=GigaDatum
    - taeric 8y ago
      I'm curious how much of a contributing factor this is to why many deep learning pipelines use smaller floats. Specifically, I "knew" that the throughput some math operations was actually greater than 1. That said, I have not kept up with which operations. I would not have expected the floating point units to be the fast ones. Again, thanks for sticking with me in this thread. I learned a bit. Hoping to come up with ways to play with this knowledge. :)
  - srean 8y ago
    Curious whether memory controllers can stripe bytes across slots much like RAID can do for disks.
    - mmt 8y ago
      I don't think it's bytewise (just like nobody does RAID3), but, yes, they can and do. The keyword you're looking for is "interleaving". I think it's also not even by DIMM slot but by channel, so if the channels are unevenly populated (e.g. 2 slots filled on one channel but only 1 on another), interleaving is disabled.
mmt 8y ago
To stretch the tub metaphor to include latency, you have to imagine someone turning the faucet off and on (or plugging and unplugging the drain) depending on how full the tub is.. and, with memory, there may be one spigot (the bandwidth), but a tremendous number of valve knobs (addresses), such that the someone has to decide the correct knob to turn before the tub empties out completely. If the wrong valve is opened, no water comes out (the wrong data is loaded into cache, which is the same as no data being loaded), so merely guessing is no good.