2 ms·

> While if you’re almost rock solid at 120FPS a dozen 70ms stalls it’s really obvious that something is wrong but the calculated 1% low’s may actually look bett

> While if you’re almost rock solid at 120FPS a dozen 70ms stalls it’s really obvious that something is wrong but the calculated 1% low’s may actually look better. I think this is an instance of the coordinated omission problem: you cannot simply measure the latency of the frames that were completed, but instead have to consider the frames that should have been delivered during a stall. Looking at frame time percentiles means a stall only penalizes your metric with one bad frame, when it should be penalized for missing several frames and showing the user an increasingly stale image for several average-frametimes.

sudosysgen 3y ago
This isn't actually true. Looking at aveeages will in fact reflect correctly on average frame staleness over a sampling period. A single frame that's 5x too long or 4 frames that are 2x too long have the same impact on the average delta of frame staleness on the total sum. Think of it like Little's law in queing theory , it's better to simply just look at different percentiles, like 99 or 99.9th. The average framerate correctly indicates the average staleness.
- wtallis 3y ago
  > A single frame that's 5x too long or 4 frames that are 2x too long have the same impact on the average delta of frame staleness on the total sum. It's not clear to me: are you saying those two scenarios should be quantified as equally bad? Because it seems pretty obvious to me that the 5x outlier is qualitatively much worse.
  - sudosysgen 3y ago
    The fact that the 5x outlier may be qualitatively worse is something that you pin down using percentiles, which is what everyone does, either 1% or 0.1% low. The argument about time-weighted frame staleness is not mathematically valid.
    - Retric 3y ago
      There’s nothing inherently mathematically special about 1% or 0.1%, you can also use 0.0042% or 0.00069% which highlights this kind of extreme outliers more. Benchmarking sites use 1% or 0.1% because they better highlight the commodity PC hardware rather than the games bugs and architecture.
      - sudosysgen 3y ago
        That's not true. You can't use 0.0042% because then you don't have enough frames to be statistically significant without running for a very long time. It's also true that a frame pacing issue that is so rare that it isn't even impacting the 0.1% frame time isn't going to be noticeable, really. These benchmarks are controlled for game bugs and architecture by simply averaging over ~40-50 games, on repeatable loops.

sudosysgen 3y ago

This isn't actually true. Looking at aveeages will in fact reflect correctly on average frame staleness over a sampling period. A single frame that's 5x too long or 4 frames that are 2x too long have the same impact on the average delta of frame staleness on the total sum. Think of it like Little's law in queing theory , it's better to simply just look at different percentiles, like 99 or 99.9th. The average framerate correctly indicates the average staleness.

wtallis 3y ago
> A single frame that's 5x too long or 4 frames that are 2x too long have the same impact on the average delta of frame staleness on the total sum. It's not clear to me: are you saying those two scenarios should be quantified as equally bad? Because it seems pretty obvious to me that the 5x outlier is qualitatively much worse.
- sudosysgen 3y ago
  The fact that the 5x outlier may be qualitatively worse is something that you pin down using percentiles, which is what everyone does, either 1% or 0.1% low. The argument about time-weighted frame staleness is not mathematically valid.
  - Retric 3y ago
    There’s nothing inherently mathematically special about 1% or 0.1%, you can also use 0.0042% or 0.00069% which highlights this kind of extreme outliers more. Benchmarking sites use 1% or 0.1% because they better highlight the commodity PC hardware rather than the games bugs and architecture.
    - sudosysgen 3y ago
      That's not true. You can't use 0.0042% because then you don't have enough frames to be statistically significant without running for a very long time. It's also true that a frame pacing issue that is so rare that it isn't even impacting the 0.1% frame time isn't going to be noticeable, really. These benchmarks are controlled for game bugs and architecture by simply averaging over ~40-50 games, on repeatable loops.