6 ms·

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in pe

by lysp 8y ago

> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances. Source code for their AI engine: if (app == benchmark) { increasePerformance(); }

vvanders 8y ago
As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also lined up with our normal workloads so we had a decent idea of where we expected the whole system to land.
- MBCook 8y ago
  Desktops have gone through it too: quack.exe and the HardOCP exposé. It’s been a long time since I thought of that.
  - dmix 8y ago
    Backstory I found, I wasn't familiar with this story (2001). https://www.hardocp.com/article/2001/10/23/optimizing_or_cheating_radeon_8500_drivers/ https://www.hardocp.com/article/2001/10/23/optimizing_or_che... TLDR: ATI did the same thing with Quake 3 Arena which was commonly used as a benchmark game for testing new video cards of that era and which many game news sites were using.
    - kllrnohj 8y ago
      That's not an entirely accurate TLDR here as what ATI did back then is now considered standard practice for GPU drivers and is welcome. It is a bit different than the mobile space as GPU drivers are doing game-specific optimizations rather than benchmark-specific ones. The end user experience is a net win for the major titles, and as long as there's no visual cost it's considered fair game. It did go too far, even to the extreme of Nvidia hardcoding in their GPU driver the exact camera path that 3dmark took ( https://www.extremetech.com/computing/54154-driver-irregularities-may-inflate-nvidia-benchmarks https://www.extremetech.com/computing/54154-driver-irregular... ), which is more like what the mobile space is grappling with now. That was solved through a combination of scolding and by just changing the benchmarks to be what people actually do. You don't see 3dmark show up in GPU reviews much, you instead see performance results in a handful of major games. If mobile follows suit and shifts to doing things like benchmarking the average/90th/99th percentile of a game of Fortnite or something then these cheating practices become less feasible as a whole.
      - MBCook 8y ago
        > what ATI did back then is now considered standard practice for GPU drivers and is welcome. Yep. That was the problem at he time. There were no shaders/etc so you’d just taking a demanding game and see how good it ran. No one optimized specific games in their drivers. So what ATI did made it appear the GPU was stronger on average that it was. If it was disclosed it would have been a feature and might have been welcomed. But it was secret. I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates? I forgot about the nVidia “micro optimization for a specific benchmark” one. No good excuse there because no one actually plays it.
        kllrnohj 8y ago
        > So what ATI did made it appear the GPU was stronger on average that it was. Well, not really. It really did run Quake 3 at the speed the benchmarks were reporting. The problem was it got that speed at the expense of a drop in image quality, and a somewhat substantial one at that. But if you launched the game and just played it you did get the exact same performance that the benchmarks were reporting. > I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates? Desktop GPU optimizations or mobile ones? Desktop GPU optimizations these days are not considered cheating because of 2 key points: 1) it improves the actual game experience (this is why reviews focus on game performance now) and 2) it doesn't impact image quality (periodically you'll see image quality comparisons to ensure both sides are still being honest) The mobile things that Huawei are doing though are cheating because it does not translate into improvements in actual user experience or gameplay. It's tuning the thermal & power consumption to hit the throttling point at the exact duration of the benchmark. This doesn't translate in any way to any real-world improvements. Thus, cheating.
  - ekianjo 8y ago
    for benchmarks to work against such practices they need to be changed every single time, in a non predictable way.
  - bri3d 8y ago
    Shader replacement was extremely popular for years - see AMD's infamous Doom3 shader replacement: https://www.anandtech.com/show/1488/6 https://www.anandtech.com/show/1488/6 . NVidia drivers did the same thing although they were (generally) a bit smarter about it, relying on shader fingerprinting rather than application fingerprinting to prevent embarrassing "rename the EXE" expose articles. They were notoriously caught using a benchmark's camera path as a fingerprint: https://www.extremetech.com/computing/54154-driver-irregularities-may-inflate-nvidia-benchmarks https://www.extremetech.com/computing/54154-driver-irregular... . GPU drivers are still loaded with such replacements, either to work around bugs or to optimize games where a developer partnership was impossible or otherwise rejected. I think that with the shift to lower-level GPU APIs this practice is beginning to fade, but it's basically been par for the course since programmable shaders became a thing.
    - hjfa0j 8y ago
      There's absolutely nothing wrong with shader replacement. Quite the opposite, as it shows the OEM cares about performance. They have two options: convince the game developer to use a better optimised shader or just replace it themselves. The latter is obviously better.
      - bri3d 8y ago
        The latter is obviously better? Shader replacement is fairly fragile and circumvents whatever QA process the game developers had in place - if the shader could be incorporated into the actual product I would think it would be better than shipping thousands of patches. However, I agree that shader replacement is great for the consumer especially in situations where developers have agreed to vendor lock-in/specialization agreements. It's only questionable to me in the sense of benchmarking applications.
    - lisk1 8y ago
      Doom 3 was selling title for Nvidia back in the days. Carmack and Co. didnt wanted to do any optimizations for ATI cards because it wont favor Nvidia cards and they had a contract with them back then.
- Jerry2 8y ago
  >we saw every single vendor do this Except Apple.
  - vvanders 8y ago
    So PowerVR is the only GPU design I don't have direct experience with in the mobile space, never worked with an SoC that fielded it. However given that Apple isn't the only one who licenses their designs I'd put a large sum of money on them doing the same thing.
  - StudentStuff 8y ago
    Apple plays this game, but in a different way. They know that apps like Geekbench test for only a short duration which won't necessarily hit their thermal throttling if the phone is cool enough. Detecting 100% load for a few dozen seconds and giving it full performance isn't hard, that is what Huawei and other vendors should have been doing. Issues like the Intel i9 performance going significantly downward with consistent usage are often missed by benchmarking apps: https://arstechnica.com/gadgets/2018/07/we-tested-throttling-on-the-macbook-pro-now-apple-says-it-has-a-firmware-fix/ https://arstechnica.com/gadgets/2018/07/we-tested-throttling...
    - jakevn 8y ago
      Are you implying that short burst performance is for gaming benchmarks?
      - StudentStuff 8y ago
        Nope, not implying that at all. For some games that may make sense, but I think that'd fall under the sustained heavy CPU/GPU usage category for most games. Same for compiling large apps, interactive HTML5 PWAs and such.
        dmoy 8y ago
        I am now very confused about which usage of the term "gaming" y'all are referring to.
userbinator 8y ago
What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.
- Shank 8y ago
  No mobile device on the market runs at the same clock speed for any length of time. They’re always scaling down the clock speed for a variety of reasons (usually battery life and thermal management). If you ran at max speed all the time, you’d have trash battery life and probably cause overheating.
  - vvanders 8y ago
    Each device has a pretty unique thermal profile(tablets dissipate heat better than small phones, more metal = better profile, etc) so these things tend to get tuned on a per-device basis. We had one device I was helping bring up where you could run the GPU full-tilt for ~60s. After that the device would reboot and fail to come up until it had cooled enough. Was a fun process characterizing that device and our workloads in 60s bursts. Couple weeks later they had it tuned in where you could run it 24/7 as the thermal limits kicked in pretty aggressively.
- JoshTriplett 8y ago
  > If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life Battery life is not the only concern; a device that sits in your hand or your pocket needs to carefully manage heat.
- anyfoo 8y ago
  If you enjoy a phone that runs hot, but only for a few minutes, depending on whether an empty battery or thermal protection kicks in first, sure. Power management and thermal budgets are a very real thing. The times where all device’s components would run at full speed are (thankfully) over for quite some time. 20 years ago PCs had multiple incredibly loud constantly running fans, that I think hardly anyone would put up with anymore (and would obviously not work for phones anyway). Today’s phones are extremely more powerful than those PCs in many ways, and yet consume a fraction of energy per time unit. Does not work without complex power management.
remlov 8y ago
Reminds me of the time way back where Futuremark confirmed Nvidia was cheating on synthetic benchmarks by doing something similar except in reverse where they would degrade texture quality and use other tricks to artificially inflate scores in 3dmark. I'm certain ATI was caught doing so as well at some point.
- MBCook 8y ago
  Yep. I believe ATI was the one where if you renamed quake.exe to quack.exe it suddenly performed significantly worse.
  - bri3d 8y ago
    Shader replacement and fingerprinting is par for the course in the GPU industry. The ethics are arguable when it comes to applications that aren't benchmarks - is it really wrong for the vendor to apply post-facto optimization to give end-users a better experience? Certainly, though, in the case of benchmarks it's a slimy practice.
    - MBCook 8y ago
      It is now, and because it improves things for gamers it’s hard to argue with. Quack.exe was for Quake 3, Before shaders even existed. At that time drivers didn’t have special paths for certain games, which is what made it a scandal. If ATI had said they were doing it, perhaps it would have been fine. But they hid it and looked like they were doing something underhanded (true or not).
samsonradu 8y ago
Looks a lot like the Volkswagen’s emission software /s

by lysp 8y ago

vvanders 8y ago

As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this. They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps. We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also lined up with our normal workloads so we had a decent idea of where we expected the whole system to land.

MBCook 8y ago
Desktops have gone through it too: quack.exe and the HardOCP exposé. It’s been a long time since I thought of that.
- dmix 8y ago
  Backstory I found, I wasn't familiar with this story (2001). https://www.hardocp.com/article/2001/10/23/optimizing_or_cheating_radeon_8500_drivers/ https://www.hardocp.com/article/2001/10/23/optimizing_or_che... TLDR: ATI did the same thing with Quake 3 Arena which was commonly used as a benchmark game for testing new video cards of that era and which many game news sites were using.
  - kllrnohj 8y ago
    That's not an entirely accurate TLDR here as what ATI did back then is now considered standard practice for GPU drivers and is welcome. It is a bit different than the mobile space as GPU drivers are doing game-specific optimizations rather than benchmark-specific ones. The end user experience is a net win for the major titles, and as long as there's no visual cost it's considered fair game. It did go too far, even to the extreme of Nvidia hardcoding in their GPU driver the exact camera path that 3dmark took ( https://www.extremetech.com/computing/54154-driver-irregularities-may-inflate-nvidia-benchmarks https://www.extremetech.com/computing/54154-driver-irregular... ), which is more like what the mobile space is grappling with now. That was solved through a combination of scolding and by just changing the benchmarks to be what people actually do. You don't see 3dmark show up in GPU reviews much, you instead see performance results in a handful of major games. If mobile follows suit and shifts to doing things like benchmarking the average/90th/99th percentile of a game of Fortnite or something then these cheating practices become less feasible as a whole.
    - MBCook 8y ago
      > what ATI did back then is now considered standard practice for GPU drivers and is welcome. Yep. That was the problem at he time. There were no shaders/etc so you’d just taking a demanding game and see how good it ran. No one optimized specific games in their drivers. So what ATI did made it appear the GPU was stronger on average that it was. If it was disclosed it would have been a feature and might have been welcomed. But it was secret. I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates? I forgot about the nVidia “micro optimization for a specific benchmark” one. No good excuse there because no one actually plays it.
      - kllrnohj 8y ago
        > So what ATI did made it appear the GPU was stronger on average that it was. Well, not really. It really did run Quake 3 at the speed the benchmarks were reporting. The problem was it got that speed at the expense of a drop in image quality, and a somewhat substantial one at that. But if you launched the game and just played it you did get the exact same performance that the benchmarks were reporting. > I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates? Desktop GPU optimizations or mobile ones? Desktop GPU optimizations these days are not considered cheating because of 2 key points: 1) it improves the actual game experience (this is why reviews focus on game performance now) and 2) it doesn't impact image quality (periodically you'll see image quality comparisons to ensure both sides are still being honest) The mobile things that Huawei are doing though are cheating because it does not translate into improvements in actual user experience or gameplay. It's tuning the thermal & power consumption to hit the throttling point at the exact duration of the benchmark. This doesn't translate in any way to any real-world improvements. Thus, cheating.
- ekianjo 8y ago
  for benchmarks to work against such practices they need to be changed every single time, in a non predictable way.
- bri3d 8y ago
  Shader replacement was extremely popular for years - see AMD's infamous Doom3 shader replacement: https://www.anandtech.com/show/1488/6 https://www.anandtech.com/show/1488/6 . NVidia drivers did the same thing although they were (generally) a bit smarter about it, relying on shader fingerprinting rather than application fingerprinting to prevent embarrassing "rename the EXE" expose articles. They were notoriously caught using a benchmark's camera path as a fingerprint: https://www.extremetech.com/computing/54154-driver-irregularities-may-inflate-nvidia-benchmarks https://www.extremetech.com/computing/54154-driver-irregular... . GPU drivers are still loaded with such replacements, either to work around bugs or to optimize games where a developer partnership was impossible or otherwise rejected. I think that with the shift to lower-level GPU APIs this practice is beginning to fade, but it's basically been par for the course since programmable shaders became a thing.
  - hjfa0j 8y ago
    There's absolutely nothing wrong with shader replacement. Quite the opposite, as it shows the OEM cares about performance. They have two options: convince the game developer to use a better optimised shader or just replace it themselves. The latter is obviously better.
    - bri3d 8y ago
      The latter is obviously better? Shader replacement is fairly fragile and circumvents whatever QA process the game developers had in place - if the shader could be incorporated into the actual product I would think it would be better than shipping thousands of patches. However, I agree that shader replacement is great for the consumer especially in situations where developers have agreed to vendor lock-in/specialization agreements. It's only questionable to me in the sense of benchmarking applications.
  - lisk1 8y ago
    Doom 3 was selling title for Nvidia back in the days. Carmack and Co. didnt wanted to do any optimizations for ATI cards because it wont favor Nvidia cards and they had a contract with them back then.
Jerry2 8y ago
>we saw every single vendor do this Except Apple.
- vvanders 8y ago
  So PowerVR is the only GPU design I don't have direct experience with in the mobile space, never worked with an SoC that fielded it. However given that Apple isn't the only one who licenses their designs I'd put a large sum of money on them doing the same thing.
- StudentStuff 8y ago
  Apple plays this game, but in a different way. They know that apps like Geekbench test for only a short duration which won't necessarily hit their thermal throttling if the phone is cool enough. Detecting 100% load for a few dozen seconds and giving it full performance isn't hard, that is what Huawei and other vendors should have been doing. Issues like the Intel i9 performance going significantly downward with consistent usage are often missed by benchmarking apps: https://arstechnica.com/gadgets/2018/07/we-tested-throttling-on-the-macbook-pro-now-apple-says-it-has-a-firmware-fix/ https://arstechnica.com/gadgets/2018/07/we-tested-throttling...
  - jakevn 8y ago
    Are you implying that short burst performance is for gaming benchmarks?
    - StudentStuff 8y ago
      Nope, not implying that at all. For some games that may make sense, but I think that'd fall under the sustained heavy CPU/GPU usage category for most games. Same for compiling large apps, interactive HTML5 PWAs and such.
      - dmoy 8y ago
        I am now very confused about which usage of the term "gaming" y'all are referring to.

userbinator 8y ago

What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.

Shank 8y ago
No mobile device on the market runs at the same clock speed for any length of time. They’re always scaling down the clock speed for a variety of reasons (usually battery life and thermal management). If you ran at max speed all the time, you’d have trash battery life and probably cause overheating.
- vvanders 8y ago
  Each device has a pretty unique thermal profile(tablets dissipate heat better than small phones, more metal = better profile, etc) so these things tend to get tuned on a per-device basis. We had one device I was helping bring up where you could run the GPU full-tilt for ~60s. After that the device would reboot and fail to come up until it had cooled enough. Was a fun process characterizing that device and our workloads in 60s bursts. Couple weeks later they had it tuned in where you could run it 24/7 as the thermal limits kicked in pretty aggressively.
JoshTriplett 8y ago
> If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life Battery life is not the only concern; a device that sits in your hand or your pocket needs to carefully manage heat.
anyfoo 8y ago
If you enjoy a phone that runs hot, but only for a few minutes, depending on whether an empty battery or thermal protection kicks in first, sure. Power management and thermal budgets are a very real thing. The times where all device’s components would run at full speed are (thankfully) over for quite some time. 20 years ago PCs had multiple incredibly loud constantly running fans, that I think hardly anyone would put up with anymore (and would obviously not work for phones anyway). Today’s phones are extremely more powerful than those PCs in many ways, and yet consume a fraction of energy per time unit. Does not work without complex power management.

remlov 8y ago

Reminds me of the time way back where Futuremark confirmed Nvidia was cheating on synthetic benchmarks by doing something similar except in reverse where they would degrade texture quality and use other tricks to artificially inflate scores in 3dmark. I'm certain ATI was caught doing so as well at some point.

MBCook 8y ago
Yep. I believe ATI was the one where if you renamed quake.exe to quack.exe it suddenly performed significantly worse.
- bri3d 8y ago
  Shader replacement and fingerprinting is par for the course in the GPU industry. The ethics are arguable when it comes to applications that aren't benchmarks - is it really wrong for the vendor to apply post-facto optimization to give end-users a better experience? Certainly, though, in the case of benchmarks it's a slimy practice.
  - MBCook 8y ago
    It is now, and because it improves things for gamers it’s hard to argue with. Quack.exe was for Quake 3, Before shaders even existed. At that time drivers didn’t have special paths for certain games, which is what made it a scandal. If ATI had said they were doing it, perhaps it would have been fine. But they hid it and looked like they were doing something underhanded (true or not).

samsonradu 8y ago

Looks a lot like the Volkswagen’s emission software /s