8 ms·
Looking at CPU/GPU Benchmark Optimizations in Galaxy S 4
- rsynnott 13y agoSamsung is merely being retro: http://blogs.msdn.com/b/oldnewthing/archive/2012/04/30/10298919.aspx http://blogs.msdn.com/b/oldnewthing/archive/2012/04/30/10298...
- mokus 13y agoIf the higher clock speeds are truly never used for anything but benchmarks then that's downright dishonest. But is that established to be the case? Perhaps those speeds are available for extremely heavy loads under specific circumstances (wired power, etc) that are rare but can actually occur. If they are, then this does seem like a fairly reasonable thing to do. Still mildly shady, but no more so than the norm when it comes to benchmark optimization.
- alec 13y agoThe AnandTech link in the article says they found the string "BenchmarkBooster" in the apk alongside the class names of several well-known benchmarks. http://anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4 http://anandtech.com/show/7187/looking-at-cpugpu-benchmark-o...
- mokus 13y agoThe fact that they found a code path that ensures particular programs get the full hardware speed does not mean that no other code path enables that performance level. I'm asking whether it is established that there is no other way that performance level is activated, as it does not seem clear to me from the article that that is the case.
- 0x0 13y agoIt's pretty obvious that, at the very least, these specific benchmarking apps are given a hugely unfair advantage, and probably only based on internal app ID strings. Apparently the frequencies were boosted for one of the benchmarking apps even if it was just sitting idly showing a menu or something.
- mokus 13y agoYea, that much is clear. And the AnandTech article (which probably should be the one on the HN front page if it hasn't popped up there since I last looked) is clearer about some of the things they ran to establish that 480 MHz is the norm. I still wasn't sure from that article, though, whether the games and such were run with the device connected to power, whether there are other user-tunable power saving settings that could affect it, etc. My only point is that I personally feel that more investigation is needed in that direction before I'll feel very negatively about the story. If it's a rarely used but possible mode, it seems like it's on the same level as disabling power saving mode when running a recognized benchmark. I kinda expect benchmarks to represent the peak attainable performance, even if it's not the typical performance.
- 0x0 13y agoI find it extremely shady to read that "glbenchmark 2.7" is unable to exceed 480mhz while "glbenchmark 2.5.1" goes into 532mhz even when idle.
- MBCook 13y agoMy reading of the AnandTech article was that the CPU was boosted to max speed for the benchmarks, but the GPU was over-clocked past what normal apps had access to.
- deveac 13y ago>The fact that they found a code path that ensures particular programs get the full hardware speed does not mean that no other code path enables that performance level. When those particular programs are benchmarks and the path is coded specifically to boost performance on a unique set of said benchmarks in contrast to the most common use cases, that's an issue regardless of whether or not you can imagine an outlier real-world scenario that utilizes a code path that tweaks the performance. You have to bend over backwards to see that as anything other than dishonest.
- Schlaefer 13y ago> You have to bend over backwards to see that as anything other than dishonest. "Our new hot device has this awesome performance mode but those benchmarks aren't aware for it and it won't be utilized in the device reviews. So let's make sure it's used in these benchmark applications." If they just flip the switches available to every developer/application it's not necessarily dishonest just tech marketing.
- joesb 13y ago1. If this awesome performance mode is turned on automatically when the app needs it. Then it must have turn on automatically for the benchmark without actually checking the application name, right? 2. Otherwise, if it require the application to be aware to utilize this feature, then why hasn't Samsung announce this "mode" to developer to utilize it yet?
- deveac 13y agoA high-performant mode would be a feature that would be marketed and lauded by the OEM, not something hidden from view and hardcoded only to benchmark testers. Also, benchmarks are designed to compare apples to apples, so if, as you suggest, the high-performant mode wouldn't be triggered by benchmark use-cases normally, then hard coding the exception is dishonest, as the only explanation is turning your apple into an orange.
- drivebyacct2 13y agoHas anyone compared this to, for example, a CM build? I suppose I could run them on my phone, but my phone's not vanilla by a long shot.
- deleted 13y ago[deleted]
- corresation 13y agoThe processor/GPUs do absolutely scale by load, however in this case they're scaling for specific apps, presumably because scaling that high for any other use (even where the throughput would be beneficial) would yield too hot of a device/kill battery. It is incredibly dishonest, and there is simply no rationalizing it -- they are cheating, and everyone involved with this is kind of a scumbag.
- megablast 13y agoYou can certainly try to dig your head into the sand.
- 0x0 13y agoNote to self: Make sure to use "com.antutu.ABenchMark" as the internal app name for my next android game.
- JoshTriplett 13y agoYou probably wouldn't want to, even if you needed the extra performance. You'd get complaints from users about battery life, device heat, and similar things that people running benchmarks often don't check for.
- stephengillie 13y agoMaybe it could be an option in the settings? [checkbox] Enable "High performance option"? Caution: will drain battery life and may cause device to overheat!!
- ygra 13y agoYou can change tha app's name from within the app?
- cscheid 13y agoIt's Quack3 all over again! http://techreport.com/review/5226/further-nvidia-optimizations-for-3dmark03 http://techreport.com/review/5226/further-nvidia-optimizatio...
- bobbles 13y agoMan, can't believe this was 12 years ago.. History repeats itself
- MBCook 13y agoIt's so sad to see this kind of stuff. It's not like the Galaxy S4 is a slow device. It reminds me of Nokia used a fake video [1] to sell the Lumia 920, which was supposed to have a pretty good camera. It just ends up being an unfortunate distraction that causes people to mistrust you. Ars is right that this isn't new. I remember ATI being caught in the Quake/Quack thing [2] a dozen years ago. [1] - http://www.theverge.com/2012/9/5/3294545/nokias-pureview-ads-are-fraudulent http://www.theverge.com/2012/9/5/3294545/nokias-pureview-ads... [2] - http://www.hardocp.com/article/2001/10/23/optimizing_or_cheating_radeon_8500_drivers http://www.hardocp.com/article/2001/10/23/optimizing_or_chea...
- ngoel36 13y agoEven worse - when Microsoft ran Xbox One demos using high end NVIDIA PC's...
- spartango 13y agoAt least those weren't benchmarks that were part of reviews, where people are making decisions based on 1-1 comparisons...
- simonh 13y agoThat reminds me of when NT first came out. They had a demo where they'd show bar charts of CPU utilisation for multiple processors and show them ramping up and down exactly in sync as they ran more software on the box. Except the demo was a fake. It was actually showing CPU utilisation for one CPU on both bars. I actually saw someone from Microsoft show this at a trade show and I did a lot of work with Sun multicore boxes and it looked weird how the bars were exactly in step, like far too perfect to be a real world demo, but I figured they were just using a demo app that was rigged to use exactly the same cycles on both CPUs or something, not that the whole thing was a lie. Boy that dates me, apart from the fact this was NT maybe 3.5 or so, what happened to PC trade shows?
- Khaine 13y agoStay classy Samsung
- enraged_camel 13y agoThis is not the first time Samsung has been caught in a disgusting lie: http://www.techspot.com/news/52274-samsung-admits-to-posting-fake-user-reviews-on-the-web.html http://www.techspot.com/news/52274-samsung-admits-to-posting... I don't understand why people keep buying this company's god damn products. I mean, holy shit.
- czhiddy 13y agoWhy? Because they spend a fortune on advertising. Anyone who's used a S3/S4 knows most of the S-"features" are pure gimmicks that a fairly competent engineer could implement in less than a week. HTC phones are much more sturdy and better designed, IMO. I was vacationing in Hong Kong earlier this month, and the subway stations were just completely plastered with various 15-foot tall Galaxy posters. Some of the features they were promoting were just laughable - simultaneous photos with the front and back camera composed into a picture-in-picture result? Really?
- acchow 13y agoS-Voice. lol. It almost feels like they had a handwritten rules-based speech to text.
- mrich 13y agoWhile I agree that Samsung phones have quite a mishmash of features just for the heck of it, I think the picture in picture feature is quite nice and is something the competition (especially Apple) would heavily advertise had they come up with it themselves.
- nmridul 13y ago>> simultaneous photos with the front and back camera composed into a picture-in-picture result? I really wanted this feature for a long time and wondered why phones that had 2 cameras did not allow it. It was specially useful when taking photos of my 1 year old daughter. It helps capture our emotions at the same time we were capturing her. So now you can see photos where she is trying to learn walking and at the same time see my face that is at times smiling (when she walks couple of steps) and at times scared (when she is about to fall) in the photos.
- edderly 13y agoOn mobile devices it would seem sensible to also run your benchmark until the battery is "flat", it might be a crude heuristic but it probably the best you can do with product devices where you can't easily break out power rails and the like. You should treat benchmarks with a heavy pinch of salt, simply because it's caught up in the bullshit messaging world of marketing where the aim is just to produce the bigger or smaller number where appropriate. The necessity to satisfy marketing ignores a lot of things. Benchmarks often translate poorly to real use. Does GLbenchmark 2.7 provide any useful insight into the performance of the top fifty games in the Android mobile market? Probably not with a few 3d heavy exceptions. Some benchmarks will never die, you'll still see Dhrystone trotted out to this day. http://www.eejournal.com/archives/articles/20090602_coremark/ http://www.eejournal.com/archives/articles/20090602_coremark... And then you hit a grey area where software can behave badly on one piece of hardware versus the other because of implementation. So if the benchmark hits this area, is it a hardware problem or a benchmark software problem?
- fidz 13y agoReminds me about Nvidia with their fake benchmark several years ago
- sczkid 13y agoThis is a minor point, but I'm really glad the article title uses "allegedly". Even if it's quite probable Samsung is guilty here, I'm glad the article doesn't adjudicate the way so many news articles do.
- megablast 13y agoIt comes off to me like these sites are scared of Samsung, they are a big player now, with a lot of pull.
- tadfisher 13y agoAnandTech is not one of those sites: > Today, we are large enough to avoid these petty discussions of withholding review samples. Most manufacturers know that one way or another we'll get our hands on a product for review and don't try to play these sorts of games. Rarely we are faced with a manufacturer or advertiser who is looking to influence our content. We have a firm internal policy in place to deliver honest, balanced reviews to the best of our ability - regardless of external pressures. Fortunately, as I mentioned earlier, we have been around long enough and are large enough to avoid this being an issue in the vast majority of situations. http://www.anandtech.com/home/about http://www.anandtech.com/home/about
- induscreep 13y agoAnyone know if iPhone or any other phone does this kind of stuff? Downright disgusting.
- songco 13y agoHuawei's K3v2...
- workbench 13y agoSince when has Apple relied on spec checklists to sell a product
- corresation 13y agoEvery iPhone release is a sea of specs: Number of GPU cores, pixel density, thinnest, lightest, made of raw holistic unobtanium, number of apps in the App Store, display metrics, etc. Despite the truly ridiculous claims to the contrary, the iPhone has always been about specs, where in many cases it is the leader (such as having the perennial market leading GPUs)
- Dwolb 13y agoFrom an EE perspective, benchmark tests should include power consumption and thermal profile measured by third party, calibrated equipment. There are some errors induced through the act of measuring, but power consumption and thermal profile checks would cut down on the amount of CPU/GPU benchmark tampering.
- DominikR 13y agoSamsung has responded that the higher GPU speeds are also available certain other applications that run in full screen. (browser, camera app, video app - which are all apps that are used frequently) If that is the case, then I fail to see anything immoral. Bad engineering - yes - but there's nothing immoral in arbitrarily boosting the GPU to run the browser or the camera smoother. (and once you regularly do this for some standard apps, it would also be appropriate to run benchmarks with the fastest speed) Link: http://www.theverge.com/2013/7/31/4574210/samsung-benchmark-rigging-response http://www.theverge.com/2013/7/31/4574210/samsung-benchmark-... Edit: I also don't understand why anyone would investigate with a hex editor. (like Anandtech does) Samsung is quite open with its Android devices, and regularly releases the source code (Link: http://opensource.samsung.com/ http://opensource.samsung.com/) It also would be quite strange to cheat, and then release the source code so everyone can see.
- diroussel 13y ago> I also don't understand why anyone would investigate with a hex editor. (like Anandtech does) The binaries were on the device, but the source code was not, so the binaries are a natural place to begin the investigation. Is it ever clear which source from http://opensource.samsung.com/ http://opensource.samsung.com/ is on the device Brian was investigating?
- DominikR 13y agoIt is absolutely clear which source is on the device because you can look up Kernel version, Baseband version and the Build number in Settings->About Phone. Anyways, what I would have liked to see in this article is if the Intent they are referring to (the one that triggers the increase of the GPU clock) is only used for cheating on benchmarks, or actually a feature that is used on many apps - not just speculation/baseless accusation.
- diroussel 13y ago> not just speculation/baseless accusation. Having read the article, it is clear that it is not baseless speculation. Two benchmarks running the same code, but with a different name. One runs faster than the other. The one that runs faster has it's name hardcoded in the samsung binaries. You are right that the source code could show a fuller picture, but I'd say this is an example of good reporting and not speculation.
- simula67 13y agoI wonder the extend to which we can trust performance benchmark results about other closed source software like web browsers, operating systems, servers etc.