15 ms·

An instructive thing here is that a lot of stuff has not improved since ~2004 or so, and working around those things that have not improved (memory latency from

An instructive thing here is that a lot of stuff has not improved since ~2004 or so, and working around those things that have not improved (memory latency from ram all the way down to l1 cache really) requires fine control of memory layout and minimizing cache pollution, which is difficult to do with all of our popular garbage collected languages, even harder with languages that don't offer memory layout controls, and jits and interpreters add further difficulty. To get the most out of modern hardware you need to: * minimize memory usage/hopping to fully leverage the CPU caches * control data layout in memory to leverage the good throughput you can get when you access data sequentially * be able to fully utilize multiple cores without too much overhead and with minimal risk of error For programs to run faster on new hardware, you need to be able to do at least some of those things.

greggsy 5y ago
It’s interesting that L2 cache has basically been steady at 2MB/core since 2004 aswell. It hasn’t changed speed in that time, but is still an order of magnitude faster than memory across that whole timeframe. Does this suggest that the memory speed bottleneck means that there simply hasn’t been a need to increase availability of that faster cache?
- ip26 5y ago
  Every level of cache strikes a balance between latency & capacity. Bigger caches have higher latency; it's a fundamental property of caches. What you can conclude is that 0.5MB-2MB and 12-15 cycles of latency has been a steady sweet spot for L2 size for twenty years. Sidebar: it was a property of caches. 3D assembly may upend the local optima.
- gameswithgo 5y ago
  the bigger the cache the longer it takes to address it, and kinda fundamental physics prevents it being faster
- throwawaylinux 5y ago
  Bigger caches could help but as a rule of thumb cache hit rate increases approximately with the square root of cache size, so it diminishes. Then the bigger you make a cache, the slower it tends to be so at some point you could make your system slower by making your cache bigger and slower.
- formerly_proven 5y ago
  Some of these numbers are clearly wrong. Some of the old latency numbers seem somewhat optimistic (e.g. 100 ns main memory ref in 1999), some of the newer ones are pessimistic (e.g. 100 ns main memory ref in 2020). The bandwidth for disks is clearly wrong, as it claims ~1.2 GB/s for a hard drive in 2020. The seek time is also wrong. It crossed 10 ms in 2000 and has reduced to 5 ms in 2010 and is 2 ms for 2020. Seems like linear interpolation to me. It's also unclear what the SSD data is supposed to mean before ~2008 as they were not really a commercial product before then. Also, for 2020 the SSD transfer rate is given as over 20 GB/s. Main memory bandwidth is given as 300+ GB/s. Cache performance has increased massively. Especially bandwidth, not reflected in a latency chart. Bandwidth and latency are of course related; just transferring a cache line over a PC66 memory bus takes a lot longer than 100 ns. The same transfer on DDR5 takes a nanosecond or so, which leaves almost all of the latency budget for existential latency. edit: https://github.com/colin-scott/interactive_latencies/blob/master/interactive_latency.html https://github.com/colin-scott/interactive_latencies/blob/ma... The data on this page is simply extrapolated using formulas and guesses.
  - dekhn 5y ago
    The oldest latency numbers were based on actual hardware Google had on hand at the time. Most of them came from microbenchmarks checked into google3. Occasionally an engineer would announce they had found an old parameter tuned for pentium and cranking it up got another 1-2% performance gain on important metrics. Many of the newer numbers could be based on existing google hardware; for example, Google deployed SSDs in 2008 (custom designed and manufactured even before then) because hard drive latency wasn't getting any better. That opened up a bunch of new opportunities. I worked on a project that wanted to store a bunch of data, Jeff literally came to me with a code that he said "compressed the data enough to justify storing it all on flash, which should help query latency" (this led to a patent!).
TillE 5y ago
It's pretty remarkable that, for efficient data processing, it's super super important to care about memory layout / cache locality in intimate detail, and this will probably be true until something fundamental changes about our computing model. Yet somehow this is fairly obscure knowledge unless you're into serious game programming or a similar field.
- efuquen 5y ago
  > Yet somehow this is fairly obscure knowledge unless you're into serious game programming or a similar field. Because the impact in optimizing hardware like that can be not so important in many applications. Getting the absolute most out of your hardware is very clearly important in game programming, but web apps where scale being served is not huge (vast majority)? Not so much. And in this context developer time is more valuable when you can throw hardware at the problem for less. Traditional game programming you had to run on the hardware people used to play, you are constrained by the client's abilities. Cloud gaming might(?) be changing some of that, but GPUs are super expensive too compared to the rest of the computing hardware. Even in that case the amounts of data you are pushing you need to be efficient within the context of the GPU, my feeling is it's not easily horizontally scaled.
  - Gigachad 5y ago
    TBH I don't think cloud gaming is a long term solution. It might be a medium term solution for people with cheap laptops but eventually the chip in cheap laptops will be able to produce photo realistic graphics and there will be no point going any further than that
    - themacguffinman 5y ago
      Photo realistic graphics ought to be enough for anybody? This seems unlikely, there's so many aspects to graphical immersion that there's still plenty of room for improvement and AAA games will find them. Photo realistic graphics is a rather vague target, it depends on what and how much you're rendering. Then you need to consider that demand grows with supply, with eg. stuff like higher resolutions, even higher refresh rates.

greggsy 5y ago

It’s interesting that L2 cache has basically been steady at 2MB/core since 2004 aswell. It hasn’t changed speed in that time, but is still an order of magnitude faster than memory across that whole timeframe. Does this suggest that the memory speed bottleneck means that there simply hasn’t been a need to increase availability of that faster cache?

ip26 5y ago
Every level of cache strikes a balance between latency & capacity. Bigger caches have higher latency; it's a fundamental property of caches. What you can conclude is that 0.5MB-2MB and 12-15 cycles of latency has been a steady sweet spot for L2 size for twenty years. Sidebar: it was a property of caches. 3D assembly may upend the local optima.
gameswithgo 5y ago
the bigger the cache the longer it takes to address it, and kinda fundamental physics prevents it being faster
throwawaylinux 5y ago
Bigger caches could help but as a rule of thumb cache hit rate increases approximately with the square root of cache size, so it diminishes. Then the bigger you make a cache, the slower it tends to be so at some point you could make your system slower by making your cache bigger and slower.
formerly_proven 5y ago
Some of these numbers are clearly wrong. Some of the old latency numbers seem somewhat optimistic (e.g. 100 ns main memory ref in 1999), some of the newer ones are pessimistic (e.g. 100 ns main memory ref in 2020). The bandwidth for disks is clearly wrong, as it claims ~1.2 GB/s for a hard drive in 2020. The seek time is also wrong. It crossed 10 ms in 2000 and has reduced to 5 ms in 2010 and is 2 ms for 2020. Seems like linear interpolation to me. It's also unclear what the SSD data is supposed to mean before ~2008 as they were not really a commercial product before then. Also, for 2020 the SSD transfer rate is given as over 20 GB/s. Main memory bandwidth is given as 300+ GB/s. Cache performance has increased massively. Especially bandwidth, not reflected in a latency chart. Bandwidth and latency are of course related; just transferring a cache line over a PC66 memory bus takes a lot longer than 100 ns. The same transfer on DDR5 takes a nanosecond or so, which leaves almost all of the latency budget for existential latency. edit: https://github.com/colin-scott/interactive_latencies/blob/master/interactive_latency.html https://github.com/colin-scott/interactive_latencies/blob/ma... The data on this page is simply extrapolated using formulas and guesses.
- dekhn 5y ago
  The oldest latency numbers were based on actual hardware Google had on hand at the time. Most of them came from microbenchmarks checked into google3. Occasionally an engineer would announce they had found an old parameter tuned for pentium and cranking it up got another 1-2% performance gain on important metrics. Many of the newer numbers could be based on existing google hardware; for example, Google deployed SSDs in 2008 (custom designed and manufactured even before then) because hard drive latency wasn't getting any better. That opened up a bunch of new opportunities. I worked on a project that wanted to store a bunch of data, Jeff literally came to me with a code that he said "compressed the data enough to justify storing it all on flash, which should help query latency" (this led to a patent!).

TillE 5y ago

It's pretty remarkable that, for efficient data processing, it's super super important to care about memory layout / cache locality in intimate detail, and this will probably be true until something fundamental changes about our computing model. Yet somehow this is fairly obscure knowledge unless you're into serious game programming or a similar field.

efuquen 5y ago
> Yet somehow this is fairly obscure knowledge unless you're into serious game programming or a similar field. Because the impact in optimizing hardware like that can be not so important in many applications. Getting the absolute most out of your hardware is very clearly important in game programming, but web apps where scale being served is not huge (vast majority)? Not so much. And in this context developer time is more valuable when you can throw hardware at the problem for less. Traditional game programming you had to run on the hardware people used to play, you are constrained by the client's abilities. Cloud gaming might(?) be changing some of that, but GPUs are super expensive too compared to the rest of the computing hardware. Even in that case the amounts of data you are pushing you need to be efficient within the context of the GPU, my feeling is it's not easily horizontally scaled.
- Gigachad 5y ago
  TBH I don't think cloud gaming is a long term solution. It might be a medium term solution for people with cheap laptops but eventually the chip in cheap laptops will be able to produce photo realistic graphics and there will be no point going any further than that
  - themacguffinman 5y ago
    Photo realistic graphics ought to be enough for anybody? This seems unlikely, there's so many aspects to graphical immersion that there's still plenty of room for improvement and AAA games will find them. Photo realistic graphics is a rather vague target, it depends on what and how much you're rendering. Then you need to consider that demand grows with supply, with eg. stuff like higher resolutions, even higher refresh rates.