4 ms·

This is just an aside but almost every post about writing high-performance code stresses the importance of data locality, that related data must be packed toget

This is just an aside but almost every post about writing high-performance code stresses the importance of data locality, that related data must be packed together in memory, and a cache miss that goes to main memory costs thousands of instructions worth of executed code. This implies that languages with tons of pointer indirection like Java will never be as fast as well-written C++ code. I wonder how the continuous growth of caches in the recent past, as well as this approach will affect this performance requirement.

bsenftner 5y ago
Talk to game developers - game code lives or dies with it's ability to persist on cache.
gameswithgo 5y ago
Have they grown? I don’t recall that l1 has grown much, and the bigger ones likely have more latency. AMD has some big l3 lately but their l3 has a bit more latency for it these big caches will help indirect code be a few % less bad but it will still be better by a lot to keep things in l1. now maybe one day with some unforeseen developments, linked lists will be cool again.
- torginus 5y ago
  L1, not so much, but comparing my old Haswell and my new Zen 2 proc, the L2 size has doubled, and the L3 has has increased from 6MB to a whopping 32MB
  - gameswithgo 5y ago
    right but I bet the ratios of cpu latency to l2 and l3 latency have both gotten worse with that growth.
    - torginus 5y ago
      I checked, and it's about the same according to these: https://www.7-cpu.com/cpu/Haswell.html https://www.7-cpu.com/cpu/Haswell.html https://www.7-cpu.com/cpu/Zen.html https://www.7-cpu.com/cpu/Zen.html I think latency has a lot to do with the speed of light - since both of these CPUs run at similar frequencies, you can cover more SRAM cells worth of distance in the one with a denser process geometry, hence you can have access to a larger cache with the same latency.
      - dragontamer 5y ago
        The speed of light is the absolute limit but isn't really a useful measure at any chip-level design. IIRC, its more about capacitance and leakage current. The less capacitance (and 3d finfets / other innovations reduce capacitance), the fewer electrons it will take to "turn on" or "turn off" a transistor. That's why modern process designers are trying more and more exotic shapes to reduce capacitance and reduce the number of electrons that need to be moved anywhere for the "on-off" toggle to happen. Leakage current is the amount of electrons that "leak" even in the offstate (which all CMOS designs are statically in the off-state. Even at 5V / on, there's a complementary transistor that's in the off-state somewhere that theoretically prevents current from flowing). As such, the two sources of current (and therefore power usage) are capacitance (the number of electrons you need to move to turn a transistor from on to off, or vice versa). Or leakage (the number of electrons that flow despite the transistor being nominally off). --------- The lower the leakage, the less power is used. The lower the capacitance, the less power is used AND the quicker the transistor turns on / off (because fewer electrons have to move, so the switch goes from .25 nanoseconds to .1 nanoseconds or whatever).
pulse7 5y ago
Data locality is just one of the things needed for writing high-performance code. Another thing needed is to have more development time for performance optimizations. C++ developers spend significant amount of time for manual-memory-management related bugs. Developers in languages without manual-memory-management can spend this time for performance optimizations. And regarding "Java will never be as fast as well-written C++ code." => since Java is getting inline/value/data classes [1] is will also benefit from data locality. [1] https://openjdk.java.net/jeps/169 https://openjdk.java.net/jeps/169
megameter 5y ago
The general rule of thumb in software performance is that the larger the scale of the codebase is, the more the "hotspots" will matter. The CPU word size, number of registers, memory sizes and cache all matter to this since they determine the "base unit size" that the hotspot code could address without dropping down a layer. Pointer-chasing code can be slow, but if the entire structure fits in cache, it won't be that slow. It gets slow mostly because the CPU can't predict what's needed next and keep the pipeline fed. And if you were to have a pointer structure that also existed in memory mostly-linearly, it would most likely have performance characteristics similar to a flat array. At scale, the data format really is the bottleneck to performance - shaving off a few bytes with better packing can make the difference, and in this respect, yes, Java is going to impose some limits via lack of control. That said, as a society we still process petabytes of JSON daily. We aren't constantly doing high-performance things - inputting and editing data and making it legible to humans are tasks that have the natural result of putting some "slack" and redundancy into data. Games tend to get results fast by "cheating" and turning the full-fat editable data into a lean, single-purpose rendering structure.

bsenftner 5y ago

Talk to game developers - game code lives or dies with it's ability to persist on cache.

gameswithgo 5y ago

Have they grown? I don’t recall that l1 has grown much, and the bigger ones likely have more latency. AMD has some big l3 lately but their l3 has a bit more latency for it these big caches will help indirect code be a few % less bad but it will still be better by a lot to keep things in l1. now maybe one day with some unforeseen developments, linked lists will be cool again.

torginus 5y ago
L1, not so much, but comparing my old Haswell and my new Zen 2 proc, the L2 size has doubled, and the L3 has has increased from 6MB to a whopping 32MB
- gameswithgo 5y ago
  right but I bet the ratios of cpu latency to l2 and l3 latency have both gotten worse with that growth.
  - torginus 5y ago
    I checked, and it's about the same according to these: https://www.7-cpu.com/cpu/Haswell.html https://www.7-cpu.com/cpu/Haswell.html https://www.7-cpu.com/cpu/Zen.html https://www.7-cpu.com/cpu/Zen.html I think latency has a lot to do with the speed of light - since both of these CPUs run at similar frequencies, you can cover more SRAM cells worth of distance in the one with a denser process geometry, hence you can have access to a larger cache with the same latency.
    - dragontamer 5y ago
      The speed of light is the absolute limit but isn't really a useful measure at any chip-level design. IIRC, its more about capacitance and leakage current. The less capacitance (and 3d finfets / other innovations reduce capacitance), the fewer electrons it will take to "turn on" or "turn off" a transistor. That's why modern process designers are trying more and more exotic shapes to reduce capacitance and reduce the number of electrons that need to be moved anywhere for the "on-off" toggle to happen. Leakage current is the amount of electrons that "leak" even in the offstate (which all CMOS designs are statically in the off-state. Even at 5V / on, there's a complementary transistor that's in the off-state somewhere that theoretically prevents current from flowing). As such, the two sources of current (and therefore power usage) are capacitance (the number of electrons you need to move to turn a transistor from on to off, or vice versa). Or leakage (the number of electrons that flow despite the transistor being nominally off). --------- The lower the leakage, the less power is used. The lower the capacitance, the less power is used AND the quicker the transistor turns on / off (because fewer electrons have to move, so the switch goes from .25 nanoseconds to .1 nanoseconds or whatever).

pulse7 5y ago

Data locality is just one of the things needed for writing high-performance code. Another thing needed is to have more development time for performance optimizations. C++ developers spend significant amount of time for manual-memory-management related bugs. Developers in languages without manual-memory-management can spend this time for performance optimizations. And regarding "Java will never be as fast as well-written C++ code." => since Java is getting inline/value/data classes [1] is will also benefit from data locality. [1] https://openjdk.java.net/jeps/169 https://openjdk.java.net/jeps/169

megameter 5y ago

The general rule of thumb in software performance is that the larger the scale of the codebase is, the more the "hotspots" will matter. The CPU word size, number of registers, memory sizes and cache all matter to this since they determine the "base unit size" that the hotspot code could address without dropping down a layer. Pointer-chasing code can be slow, but if the entire structure fits in cache, it won't be that slow. It gets slow mostly because the CPU can't predict what's needed next and keep the pipeline fed. And if you were to have a pointer structure that also existed in memory mostly-linearly, it would most likely have performance characteristics similar to a flat array. At scale, the data format really is the bottleneck to performance - shaving off a few bytes with better packing can make the difference, and in this respect, yes, Java is going to impose some limits via lack of control. That said, as a society we still process petabytes of JSON daily. We aren't constantly doing high-performance things - inputting and editing data and making it legible to humans are tasks that have the natural result of putting some "slack" and redundancy into data. Games tend to get results fast by "cheating" and turning the full-fat editable data into a lean, single-purpose rendering structure.