3 ms·

Data flow, data flow, data flow. It's all about structuring your data such that don't don't cache miss on your fetches and you don't stall your pipelines by do

Data flow, data flow, data flow. It's all about structuring your data such that don't don't cache miss on your fetches and you don't stall your pipelines by doing operations that flush/occupy the same pipe. You can easily see 10-50x performance boosts on the same dataset with the right approach. Look up Data Oriented Design, Mike Acton talks about it a ton.

Asooka 9y ago
To expand slightly, a single read from memory that's not cached and actually has to go to RAM can take 100 cycles. That's 100 cycles during which your CPU does nothing but wait for electrical signals to travel to and from RAM. Well written data processing programs (game engines, physics simulations &c) strive to structure their data so they have as few of those as possible. If you just read "Design Patterns" and wrote a beautiful 10-level deep class hierarchy in Java, where every object access can be behind something like 3 levels of indirection via pointers. This is not a critique against either Design Patterns or Java, both of them are important and useful. And really, quite often you don't care about waiting 100 cycles, you have ten million times 100 cycles in every second. It's when you start crunching numbers in large quantities that it can get problematic. Modern CPUs are very fast, but only if you feed them data in the right order. Otherwise you're basically running on a machine from 20 years ago, just with 32GB of RAM. Best program design right now is to yes, do write your deep class hierarchies and whatnot, but also know how to identify the parts of your program doing heavy computation and how to restructure those so you can unleash the full power of a modern processor.
- SomeHacker44 9y ago
  To clarify: "That's 100 cycles during which your CPU does nothing but wait..." is true only in the absolute worst case. In the likely case, your superscalar, out-of-order processor is working on the next several instructions, possibly including some speculative instructions after branches, and simultaneous-multithreading is letting another thread use the now-idle resources. Furthermore, since it's superscalar there are tons of compute resources (integer ALUs, FP ALUs, vector ALUs, to mention a few) which the CPU scheduler will try to keep busy as close to 100% of the time as possible. That said, in any performance critical application, you pay close attention to your memory hierarchy and you design and compile your application in a way to respect that as well as take advantage of the underlying hardware architecture and resources to the fullest.
  - vvanders 9y ago
    > and simultaneous-multithreading is letting another thread use the now-idle resources. This can actually be worse due to context switch overhead and caches you were using getting evicted by the now running thread hitting the same issue. If you want to do this right you should be organizing your data so that the prefetcher and take advantage of it. All of those speculative things still have a cost.

Asooka 9y ago

To expand slightly, a single read from memory that's not cached and actually has to go to RAM can take 100 cycles. That's 100 cycles during which your CPU does nothing but wait for electrical signals to travel to and from RAM. Well written data processing programs (game engines, physics simulations &c) strive to structure their data so they have as few of those as possible. If you just read "Design Patterns" and wrote a beautiful 10-level deep class hierarchy in Java, where every object access can be behind something like 3 levels of indirection via pointers. This is not a critique against either Design Patterns or Java, both of them are important and useful. And really, quite often you don't care about waiting 100 cycles, you have ten million times 100 cycles in every second. It's when you start crunching numbers in large quantities that it can get problematic. Modern CPUs are very fast, but only if you feed them data in the right order. Otherwise you're basically running on a machine from 20 years ago, just with 32GB of RAM. Best program design right now is to yes, do write your deep class hierarchies and whatnot, but also know how to identify the parts of your program doing heavy computation and how to restructure those so you can unleash the full power of a modern processor.

SomeHacker44 9y ago
To clarify: "That's 100 cycles during which your CPU does nothing but wait..." is true only in the absolute worst case. In the likely case, your superscalar, out-of-order processor is working on the next several instructions, possibly including some speculative instructions after branches, and simultaneous-multithreading is letting another thread use the now-idle resources. Furthermore, since it's superscalar there are tons of compute resources (integer ALUs, FP ALUs, vector ALUs, to mention a few) which the CPU scheduler will try to keep busy as close to 100% of the time as possible. That said, in any performance critical application, you pay close attention to your memory hierarchy and you design and compile your application in a way to respect that as well as take advantage of the underlying hardware architecture and resources to the fullest.
- vvanders 9y ago
  > and simultaneous-multithreading is letting another thread use the now-idle resources. This can actually be worse due to context switch overhead and caches you were using getting evicted by the now running thread hitting the same issue. If you want to do this right you should be organizing your data so that the prefetcher and take advantage of it. All of those speculative things still have a cost.