5 ms·

> In order to get any useful results I had to run my Advent of Code solution 1000 times in a loop Yeah that's my general problem with all these flamegraphs and

> In order to get any useful results I had to run my Advent of Code solution 1000 times in a loop Yeah that's my general problem with all these flamegraphs and other time based tools. There's a bunch of noise! I'd image for something with deterministic GC (or hell no-gc) you should be able to get a "instruction count" based approach that'd be much more deterministic as to what version of the code is fastest (for that workflow).

foldr 4y ago
One nice think about Go’s built in benchmarking is that it automatically takes care or running benchmarks repeatedly until runtime is stable and then averaging the time over a number of hot runs.
dahfizz 4y ago
An instruction account would be a decent first order approximation, but I don't think it will be very useful for optimization. The bottleneck of modern hardware (generally) is memory. You can get huge speedups by tweaking the way your program structures and operates on data to make it more cache friendly. This won't really affect instruction count but could make your program run 2x faster.
- hinkley 4y ago
  I have been wondering for a while if the solution to the memory bottleneck is to stop treating cache memory as an abstraction and make it directly addressable. Removing L1 cache is probably impossible, but I suspect removing L2 and L3 caches and replacing them with working memory might fare better. The latest iteration of this thought process was wondering what would happen if you exposed microcode for memory management to the kernel, and it ran a moral equivalent of eBNF directly in a beefed up MMU. Legacy code and code that doesn't deign to do its own management would elect to use routines that maintain the cache abstraction. The kernel could also segment the caches for different processes, reducing the surface area for cache-related bugs.
- lesuorac 4y ago
  Somebody else pointed out that I probably want to use Valgrind. It seems to have the ability to count instruction counts as well as cache misses [1] [2]. [1]: https://web.stanford.edu/class/archive/cs/cs107/cs107.1202/resources/callgrind https://web.stanford.edu/class/archive/cs/cs107/cs107.1202/r... [2]: http://www.codeofview.com/fix-rs/2017/01/24/how-to-optimize-rust-programs-on-linux/ http://www.codeofview.com/fix-rs/2017/01/24/how-to-optimize-...
  - dahfizz 4y ago
    Yup! valgrind is an amazing tool. Just keep in mind that the program itself will be slower when run under valgrind. Its great for debugging, but your benchmarks should be run raw.
    - dmit 4y ago
      And when all else fails, you reach for Intel VTune. :) Which has provided a free community version for a couple years now, I might add.
pornel 4y ago
Rust has libraries like https://lib.rs/criterion https://lib.rs/criterion that help running code 1000 times in a loop, with proper timing, elimination of outliers, etc.
kaba0 4y ago
Valgrind (and “friends”) are exactly this, they can measure cache misses, branch mispredictions, etc.
hinkley 4y ago
Something about Flamegraphs has been bugging me for a while and as more languages are adopting async/await semantics is coming to a head: Flamegraphs work well for synchronous, sequential code. I don't think it's an accident that a lot of people first encountered them for Javascript, in the era immediately before Promises became a widely known technique. They actually worked then, but now they're quickly becoming more trouble than they're worth. A long time ago when people still tried to charge for profilers, I remember one whose primary display was a DAG, not unlike the way some microservices are visualized today. Only the simplest cases of cause and effect can be adequately displayed as a flamegraph. For anything else it's, as you say, all noise.
- Arnavion 4y ago
  Your asynchronous code is made up of synchronous segments between the yield points. The flamegraph measures those just fine.
  - hinkley 4y ago
    It tells you something happened, it doesn't tell you why. The value of the flamegraph over the previous iteration was that it more directly showed the chain of events. So no, it doesn't measure it 'just fine'. It's accomplishing absolutely nothing.
    - Arnavion 4y ago
      >The value of the flamegraph over the previous iteration was that it more directly showed the chain of events. Yes, and it continues to show that. A perf trace for: async fn foo() -> i32 { bar().await } async fn bar() -> i32 { baz().await; whatever() } ... that happens to be captured while `whatever` is executing sees a callstack that is: #1 whatever() (somewhere inside it) #2 bar::poll() (at the cpu_intensive_i32 call) #3 foo::poll() (at the bar.await line) #4 runtime executor ... (internals we don't care about) The flamegraph records the CPU usage accordingly. >So no, it doesn't measure it 'just fine'. It's accomplishing absolutely nothing. How about you actually try to use it instead of assuming?

foldr 4y ago

One nice think about Go’s built in benchmarking is that it automatically takes care or running benchmarks repeatedly until runtime is stable and then averaging the time over a number of hot runs.

dahfizz 4y ago

An instruction account would be a decent first order approximation, but I don't think it will be very useful for optimization. The bottleneck of modern hardware (generally) is memory. You can get huge speedups by tweaking the way your program structures and operates on data to make it more cache friendly. This won't really affect instruction count but could make your program run 2x faster.

hinkley 4y ago
I have been wondering for a while if the solution to the memory bottleneck is to stop treating cache memory as an abstraction and make it directly addressable. Removing L1 cache is probably impossible, but I suspect removing L2 and L3 caches and replacing them with working memory might fare better. The latest iteration of this thought process was wondering what would happen if you exposed microcode for memory management to the kernel, and it ran a moral equivalent of eBNF directly in a beefed up MMU. Legacy code and code that doesn't deign to do its own management would elect to use routines that maintain the cache abstraction. The kernel could also segment the caches for different processes, reducing the surface area for cache-related bugs.
lesuorac 4y ago
Somebody else pointed out that I probably want to use Valgrind. It seems to have the ability to count instruction counts as well as cache misses [1] [2]. [1]: https://web.stanford.edu/class/archive/cs/cs107/cs107.1202/resources/callgrind https://web.stanford.edu/class/archive/cs/cs107/cs107.1202/r... [2]: http://www.codeofview.com/fix-rs/2017/01/24/how-to-optimize-rust-programs-on-linux/ http://www.codeofview.com/fix-rs/2017/01/24/how-to-optimize-...
- dahfizz 4y ago
  Yup! valgrind is an amazing tool. Just keep in mind that the program itself will be slower when run under valgrind. Its great for debugging, but your benchmarks should be run raw.
  - dmit 4y ago
    And when all else fails, you reach for Intel VTune. :) Which has provided a free community version for a couple years now, I might add.

pornel 4y ago

Rust has libraries like https://lib.rs/criterion https://lib.rs/criterion that help running code 1000 times in a loop, with proper timing, elimination of outliers, etc.

kaba0 4y ago

Valgrind (and “friends”) are exactly this, they can measure cache misses, branch mispredictions, etc.

hinkley 4y ago

Something about Flamegraphs has been bugging me for a while and as more languages are adopting async/await semantics is coming to a head: Flamegraphs work well for synchronous, sequential code. I don't think it's an accident that a lot of people first encountered them for Javascript, in the era immediately before Promises became a widely known technique. They actually worked then, but now they're quickly becoming more trouble than they're worth. A long time ago when people still tried to charge for profilers, I remember one whose primary display was a DAG, not unlike the way some microservices are visualized today. Only the simplest cases of cause and effect can be adequately displayed as a flamegraph. For anything else it's, as you say, all noise.

Arnavion 4y ago
Your asynchronous code is made up of synchronous segments between the yield points. The flamegraph measures those just fine.
- hinkley 4y ago
  It tells you something happened, it doesn't tell you why. The value of the flamegraph over the previous iteration was that it more directly showed the chain of events. So no, it doesn't measure it 'just fine'. It's accomplishing absolutely nothing.
  - Arnavion 4y ago
    >The value of the flamegraph over the previous iteration was that it more directly showed the chain of events. Yes, and it continues to show that. A perf trace for: async fn foo() -> i32 { bar().await } async fn bar() -> i32 { baz().await; whatever() } ... that happens to be captured while `whatever` is executing sees a callstack that is: #1 whatever() (somewhere inside it) #2 bar::poll() (at the cpu_intensive_i32 call) #3 foo::poll() (at the bar.await line) #4 runtime executor ... (internals we don't care about) The flamegraph records the CPU usage accordingly. >So no, it doesn't measure it 'just fine'. It's accomplishing absolutely nothing. How about you actually try to use it instead of assuming?