3 ms·

Coming from a Rust game development background, something that's noticeable to me is that sampling based profiling seeks to solve a different problem than the t

by jms55 6y ago

Coming from a Rust game development background, something that's noticeable to me is that sampling based profiling seeks to solve a different problem than the types of questions games need to answer. With sampling you can say "I saw function() 100/150 samples (10ms interval between samples), spend your time looking at that!". That doesn't really work for games, where your work is on the scale of 0.1ms, and you might call the same function 100's of times in 16ms. Calling update_object() 1000 times an interval isn't actually a problem, when it's only taking 5% of your overall frame budget (on average). Games also have the constraint of not wanting your profiling to throw off the game speed itself, it needs to run very fast. With games, what you really want to know are: 1. Which frames are taking over 16ms? (On the current machine; what's fast enough on one computer isn't on weaker machines. Sometimes you want to look at every frame, and not just the slow ones.) 2. What functions are causing them to go over? 3. Are those functions _usually_ that slow? How fast do they usually run, was this just an uncommon edge case? Mean/median/90th percentile performance? With typical applications, instead you want to know * Of the many diverging branches and tasks my application can do, which areas occur the most often, and are therefore worth spending time optimizing? Sampling based profiling obviously works better for applications like this, while non-sampling profiling works better for games. Sadly, there doesn't seem to be very many GUIs/APIs for working with the latter, especially with linux compatibility. I'm hoping to remedy this, as soon as I get better at GTK. EDIT: An example of why sampling based profilers (in this case, perf) aren't great for games. Although sandbox::sandbox::Sandbox::temperature_update takes up a decent amount of overall time, it's hard to tell whether that matters per frame or not. https://profiler.firefox.com/public/c4y22fect9qg9mfm0wv0pjweacpvpqz204y2by8/calltree/?globalTrackOrder=0&localTrackOrderByPid=560138-0-1~&thread=0&timelineType=stack&v=5 https://profiler.firefox.com/public/c4y22fect9qg9mfm0wv0pjwe...

chairmanwow1 6y ago
Great comment. Thank you for sharing
vlovich123 6y ago
Have you tried a flameshot graph? It’ll give you a nicer visualization of the relative amount of cpu something took. Sampling profilers are but one thing in the toolbox. On Windows you can use WPA for extremely detailed tracing and can link CPU work to GPu work (the tooling can be awkward as the GPU one hasn’t seen love in a while). Google is tackling this problem for Android via Perfetto although the tooling isn’t as mature and the GPU stuff will probably stay Android unless Nvidia/AMD extend it for the desktop (seeing as how stadia is being deprioritized).
- jms55 6y ago
  I can't find any references to flameshot graphs, do you mean flamegraphs? They have the same issue, they tend to show the whole program, not per frame. Perfetto does look interesting though, thank you for the reference.
killingtime74 6y ago
May I ask what games use Rust?
- jms55 6y ago
  The most notable game is Veloren https://veloren.net https://veloren.net I made a game https://github.com/JMS55/sandbox https://github.com/JMS55/sandbox See what other people are up to here https://rust-gamedev.github.io/posts/ https://rust-gamedev.github.io/posts/
Dylan16807 6y ago
> That doesn't really work for games, where your work is on the scale of 0.1ms, and you might call the same function 100's of times in 16ms. Calling update_object() 1000 times an interval isn't actually a problem, when it's only taking 5% of your overall frame budget (on average). Games also have the constraint of not wanting your profiling to throw off the game speed itself, it needs to run very fast. I'm confused, because the thing sampling measures is how much of your frame budget goes to each function. And sampling works equally well regardless of how fast your functions are. But I do agree that looking for variance and focusing on frames that take too long are important.
- jms55 6y ago
  Say you sample every 10ms. This frame, and every frame for the sake of example, looks like this: * f1() - 1ms * f2() - 8ms * f3() - 4ms * wait_for_next_frame() - 3ms After 10ms, you'll sample and see f3() is running. Now you seem to think, oh, I should optimize f3(). But f2() is clearly the main cost of the frame. You can totally increase the frequency, so now you sample every 0.1ms or something, and that's probably fine. But you probably want to just instrument your code and get more accurate results at that point. My main point is that out of the box typical sampling profilers A: Don't sample fast enough B: Don't display the data split up by frame.
  - Dylan16807 6y ago
    That's why you take thousands of samples. Most of them will hit f2 if you do it right. If your sampler's timer is running perfectly in sync with your framerate, that's a problem, but it's not a "not enough samples" problem. Fix it by taking samples every 10.05-10.07 milliseconds instead.

by jms55 6y ago

chairmanwow1 6y ago

Great comment. Thank you for sharing

vlovich123 6y ago

Have you tried a flameshot graph? It’ll give you a nicer visualization of the relative amount of cpu something took. Sampling profilers are but one thing in the toolbox. On Windows you can use WPA for extremely detailed tracing and can link CPU work to GPu work (the tooling can be awkward as the GPU one hasn’t seen love in a while). Google is tackling this problem for Android via Perfetto although the tooling isn’t as mature and the GPU stuff will probably stay Android unless Nvidia/AMD extend it for the desktop (seeing as how stadia is being deprioritized).

jms55 6y ago
I can't find any references to flameshot graphs, do you mean flamegraphs? They have the same issue, they tend to show the whole program, not per frame. Perfetto does look interesting though, thank you for the reference.

killingtime74 6y ago

May I ask what games use Rust?

jms55 6y ago
The most notable game is Veloren https://veloren.net https://veloren.net I made a game https://github.com/JMS55/sandbox https://github.com/JMS55/sandbox See what other people are up to here https://rust-gamedev.github.io/posts/ https://rust-gamedev.github.io/posts/

Dylan16807 6y ago

> That doesn't really work for games, where your work is on the scale of 0.1ms, and you might call the same function 100's of times in 16ms. Calling update_object() 1000 times an interval isn't actually a problem, when it's only taking 5% of your overall frame budget (on average). Games also have the constraint of not wanting your profiling to throw off the game speed itself, it needs to run very fast. I'm confused, because the thing sampling measures is how much of your frame budget goes to each function. And sampling works equally well regardless of how fast your functions are. But I do agree that looking for variance and focusing on frames that take too long are important.

jms55 6y ago
Say you sample every 10ms. This frame, and every frame for the sake of example, looks like this: * f1() - 1ms * f2() - 8ms * f3() - 4ms * wait_for_next_frame() - 3ms After 10ms, you'll sample and see f3() is running. Now you seem to think, oh, I should optimize f3(). But f2() is clearly the main cost of the frame. You can totally increase the frequency, so now you sample every 0.1ms or something, and that's probably fine. But you probably want to just instrument your code and get more accurate results at that point. My main point is that out of the box typical sampling profilers A: Don't sample fast enough B: Don't display the data split up by frame.
- Dylan16807 6y ago
  That's why you take thousands of samples. Most of them will hit f2 if you do it right. If your sampler's timer is running perfectly in sync with your framerate, that's a problem, but it's not a "not enough samples" problem. Fix it by taking samples every 10.05-10.07 milliseconds instead.