5 ms·

Mobile Benchmark Cheating as a Service

51 points by uluyol 7y ago

pjc50 7y ago
There's a long history of this kind of nonsense: https://techreport.com/review/3089/how-atis-drivers-optimize-quake-iii/ https://techreport.com/review/3089/how-atis-drivers-optimize... (Drivers detect the name of the executable as QUAKE.EXE and behave differently!)
- jimbob45 7y ago
  That was a really interesting blast from the past.
justincredible 7y ago
This reminds me of Volkswagen cheating on their emissions tests. Goodhart's law and corporate nature combine to erode trust most effectively.
fxtentacle 7y ago
Isn't that kind of exactly what GPU companies have been doing all the time with game-optimized drivers? And since they cannot just skip work, but merely re-prioritize things, these cheats should be easily detectable by a reduced battery life.
- ReverseCold 7y ago
  > game-optimized drivers These generally actually make the games you're playing have somewhat better performance (via game-specific hacks). Applying tweaks to a mobile phone benchmark, but then not applying those same tweaks to other apps is just misrepresentation of actual performance.
drewg123 7y ago
Towards the end of the article, they talk about high performance modes added by manufacturers and say "The phone essentially goes into a high-power mode throwing away any attempt to be efficient; it’s a nonsensical mode that is unusable in every-day use-cases beyond getting high benchmark scores" What makes it "nonsensical" ? Are they exceeding the TDP of the phone? If so, then couldn't benchmarks just be made much longer, so that the phones would throttle or fail if they were cheating? Or is it just about battery life? If so, why is it unreasonable to have a phone plugged in to get great gaming performance?
- saagarjha 7y ago
  > why is it unreasonable to have a phone plugged in to get great gaming performance People don't always game near power outlets.
  - drewg123 7y ago
    There's a difference between "don't always" and "unreasonable." It seems like if you can get a phone that performs as well as something that costs 2x as much, and the compromise is gaming near an outlet, or carrying an external battery, that's a choice some people would like to have. And if it is the battery that's "unreasonable', then I wonder why the review sites don't just factor the cheating into their review by having a category like "battery life running PCMark" or something where they measure how long the phone will run the benchmark.
    - hobs 7y ago
      Customers are not informed, they just use the number - if you game the number you are unethical even if you pretend like its for consumer choice and enhanced information - no matter what that's not how consumers use it and you are juicing a number to get additional sales.
- codeflo 7y ago
  The problem is basically a variant of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." Publications want to use these benchmarks as a convenient proxy for real-world performance. Phones have "benchmark detection" and run their power management with different settings when they detect that a benchmark is running, defeating the purpose of running the benchmark. BTW, I don't think this is about gaming vs. non-gaming workloads (there are different benchmarks for those), or even running plugged-in vs. on battery. My understanding is that some of these "benchmark modes" are so aggressive that the phones would become too hot to hold in your hands. No sane phone manufacturer could use settings like these for real-world usage.
  - lonelappde 7y ago
DeathArrow 7y ago
Most popular benchmarks used on mobiles are pretty much flawed. I.e.Geekbench is highly optimized for Arm but it doesn't use AVX2 on PC, making ARM seem much better than it is. Geekbench runtime is too small. Good laptop reviewing websites like notebookcheck use many rounds of Cinebench R15 to test if scores go down due to thermal throttling and by how much and after what time. Something similar can be done on mobile phones instead of just running Geekbench once and writing down the number. Yes, some manufacturers are trying to cheat but benchmarks used are bad and many reviewers do a poor job.

51 points by uluyol 7y ago

There's a long history of this kind of nonsense: https://techreport.com/review/3089/how-atis-drivers-optimize-quake-iii/ https://techreport.com/review/3089/how-atis-drivers-optimize... (Drivers detect the name of the executable as QUAKE.EXE and behave differently!)

jimbob45 7y ago
That was a really interesting blast from the past.

justincredible 7y ago

This reminds me of Volkswagen cheating on their emissions tests. Goodhart's law and corporate nature combine to erode trust most effectively.

fxtentacle 7y ago

Isn't that kind of exactly what GPU companies have been doing all the time with game-optimized drivers? And since they cannot just skip work, but merely re-prioritize things, these cheats should be easily detectable by a reduced battery life.

ReverseCold 7y ago
> game-optimized drivers These generally actually make the games you're playing have somewhat better performance (via game-specific hacks). Applying tweaks to a mobile phone benchmark, but then not applying those same tweaks to other apps is just misrepresentation of actual performance.

drewg123 7y ago

Towards the end of the article, they talk about high performance modes added by manufacturers and say "The phone essentially goes into a high-power mode throwing away any attempt to be efficient; it’s a nonsensical mode that is unusable in every-day use-cases beyond getting high benchmark scores" What makes it "nonsensical" ? Are they exceeding the TDP of the phone? If so, then couldn't benchmarks just be made much longer, so that the phones would throttle or fail if they were cheating? Or is it just about battery life? If so, why is it unreasonable to have a phone plugged in to get great gaming performance?

saagarjha 7y ago
> why is it unreasonable to have a phone plugged in to get great gaming performance People don't always game near power outlets.
- drewg123 7y ago
  There's a difference between "don't always" and "unreasonable." It seems like if you can get a phone that performs as well as something that costs 2x as much, and the compromise is gaming near an outlet, or carrying an external battery, that's a choice some people would like to have. And if it is the battery that's "unreasonable', then I wonder why the review sites don't just factor the cheating into their review by having a category like "battery life running PCMark" or something where they measure how long the phone will run the benchmark.
  - hobs 7y ago
    Customers are not informed, they just use the number - if you game the number you are unethical even if you pretend like its for consumer choice and enhanced information - no matter what that's not how consumers use it and you are juicing a number to get additional sales.
codeflo 7y ago
The problem is basically a variant of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." Publications want to use these benchmarks as a convenient proxy for real-world performance. Phones have "benchmark detection" and run their power management with different settings when they detect that a benchmark is running, defeating the purpose of running the benchmark. BTW, I don't think this is about gaming vs. non-gaming workloads (there are different benchmarks for those), or even running plugged-in vs. on battery. My understanding is that some of these "benchmark modes" are so aggressive that the phones would become too hot to hold in your hands. No sane phone manufacturer could use settings like these for real-world usage.
- lonelappde 7y ago

DeathArrow 7y ago

Most popular benchmarks used on mobiles are pretty much flawed. I.e.Geekbench is highly optimized for Arm but it doesn't use AVX2 on PC, making ARM seem much better than it is. Geekbench runtime is too small. Good laptop reviewing websites like notebookcheck use many rounds of Cinebench R15 to test if scores go down due to thermal throttling and by how much and after what time. Something similar can be done on mobile phones instead of just running Geekbench once and writing down the number. Yes, some manufacturers are trying to cheat but benchmarks used are bad and many reviewers do a poor job.