4 ms·

Google blocked Pixel 8 reviewers and owners from benchmarking the Tensor G3 chip

67 points by josephcsible 3y ago

jauntywundrkind 3y ago
I really detest that many Google products like some of their Chromebooks just list as having something vague like "Intel Core processor" or whatnot. No model, no idea what class chip (ultra low power? Low power? Mainline?). It feels incredibly consumer hostile, not telling people what they're getting.
- pjmlp 3y ago
  It sucks, on the other hand, it hardly matters for Web pages, which is the main application on an OS juggling Chrome instances.
metadat 3y ago
How very Oracle of them. https://stackoverflow.com/questions/12115397/is-it-against-license-to-publish-oracle-and-sql-server-performance-test https://stackoverflow.com/questions/12115397/is-it-against-l... https://www.brentozar.com/archive/2018/05/the-dewitt-clause-why-you-rarely-see-database-benchmarks/ https://www.brentozar.com/archive/2018/05/the-dewitt-clause-... https://news.ycombinator.com/item?id=14232976 https://news.ycombinator.com/item?id=14232976 (2017)
- o11c 3y ago
  Notably, it isn't against the law to call out "Oracle forbids us from benchmarking, so we are forced to rank it last".
deleted 3y ago
[deleted]
greatgib 3y ago
Can't such a block be considered as a proof against Google in an anti-trust trial? Google abusing of is position as the provider of the android app-store to protect his sales of phone as a manufacturer. That are different activities.
- mcphage 3y ago
  I’m not so sure. They don’t sell competing benchmark software, so it might not be relevant.
vore 3y ago
I would place my bet on this being some kind of app compatibility mishap rather than Google actively blocking Geekbench and 3DMark. Apparently Pixel 6a owners also had this problem, but with none of the Tensor G3 performance speculation in the article: https://www.reddit.com/r/GooglePixel/comments/wc4m3w/pixel_6a_not_compatible_with_benchmarking_apps/ https://www.reddit.com/r/GooglePixel/comments/wc4m3w/pixel_6...
- fh9302 3y ago
  There was no compatibility problem, Google intentionally blocked the installation to avoid reviewers testing the chip. The benchmarks worked without issues when sideloaded, but not every reviewer bothered to do that.
  - BenjiWiebe 3y ago
    Sounds like a compatibility problem. I don't understand how exactly the play store works, but if your device string is unexpected (like maybe a new device? or a lineageos phone) some apps can't be installed from the play store but work fine sideloaded.
    - nfriedly 3y ago
      As a side note to this, Aurora store is awesome. If you sign in with your Google account, then you can install paid apps that you've purchased, but the Play store thinks are incompatible with your device. In my experience, the majority of them work perfectly fine!
    - _imnothere 3y ago
      Definitely not one, Play Store is known to showing apps as not supported while it actually is since years.
  - vore 3y ago
    I think you are as blind to Google’s intentions as much as I am, but it sure seems silly for them to intentionally block it from the Play Store when side loading it is trivial, doesn’t it?
greesil 3y ago
FTA the performance difference isn't hugely worse from a comparable chip IMO. Maybe they did more damage by trying to suppress.
havnagiggle 3y ago
What exactly did they do here? X linked video is broken for me, and the article just says the performance is bad. My pixel 6 just got Android 14. Possibly just OS/new hardware related?
- josephcsible 3y ago
  When people using a Pixel 8 or Pixel 8 Pro tried to install benchmark apps from the Play Store, Google blocked the installation and falsely claimed "This app won't work for your device".
  - havnagiggle 3y ago
    IMO that's not evidence of anything other than a bug with package manifests and the current Playstore version. I have to side load apps all the time on my kids tablets because of API incompatibilities that don't ultimately matter, and it isn't Google out to get my kids. According to the 3DMark playstore page, the last update was Sept 21, 2023. Version 2.3.4869. Android 14 came out October 4th, making any incompatibility likely a temporary problem. Unless there's a quote from 3DMark explaining how everything is dandy in their end, I'm not buying what you are selling.
pseudosavant 3y ago
I find it interesting that the Geekbench scores for the latest iPhone isn't mentioned. About the same performance as an iPhone 12. Yet, the Pixel 8 and iPhone 15 lines are price comparably. Pixel 8 Pro: Single: 1760 Multi: 4442 iPhone 15 Pro: Single: 2894 Multi: 7192 iPhone 12: Single: 1995 Multi: 4401
- HumblyTossed 3y ago
  With Apple and Google doing custom chips, I don't think that matters as much as people think anymore. Surely, a lot of functionality that used to be done in software is now done on chip. A generic benchmark isn't going to capture that.
  - Arcanum-XIII 3y ago
    Like what? What's the ML performance on this chip? Can it help with game? Or apps I'll use everyday? Does it improve the web experience? The last one depends largely on those CPU benchmark... I want to be pragmatic here. Coprocessor were all the rage for the Amiga, then were out out of the loop, and were back for with the GPU. A bit like we're back to the mainframe experience with Kubernetes and all (don't throw tomatoes at me ;))
- Skunkleton 3y ago
  > Yet, the Pixel 8 and iPhone 15 lines are price comparably. But this isn't how pricing works. Even if it did the CPU isn't the most expensive component in the bom, and it isn't the most important component for user experience.
  - Arcanum-XIII 3y ago
    The problem is that you can't easily and objectively factor the user experience. CPU are easy to benchmark — and do honestly factor a lot in the pricing. Android (the OS) development is payed for by other means, and the hardware line is kind of a vanity project. People may, or may not prefer the Android experience. That's not relevant here.
oldpersonintx 3y ago
every CPU benchmark needs to have another line drawn waaaay under all of the lines drawn for various CPUs...we shall label this line "performance you actually need and can make use of" if all of the CPUs are above this line, then you should not use CPU performance as a determining factor typing this happily from a 2014 Mobile Celeron laptop that is still fast enough to do all the software dev I need
sxp 3y ago
Citation needed that Google actively blocked installation of the benchmarking app. None of the links support that claim. If Google want to block benchmarking, they would have had their lawyers add that clause to the reviewers' contracts/NDAs as a condition of getting the device early. Since this is a technical issue, I'm guessing the benchmarking software didn't properly configure their Play Store account and mark the P8 or the new Android OS as compatible with the software. That would prevent it from being installed via the Play Store but not via sideloading. Disclosure, I used to work at Google on Android so I have some familiarity with how Play Store rollouts work.

67 points by josephcsible 3y ago

I really detest that many Google products like some of their Chromebooks just list as having something vague like "Intel Core processor" or whatnot. No model, no idea what class chip (ultra low power? Low power? Mainline?). It feels incredibly consumer hostile, not telling people what they're getting.

pjmlp 3y ago
It sucks, on the other hand, it hardly matters for Web pages, which is the main application on an OS juggling Chrome instances.

metadat 3y ago

How very Oracle of them. https://stackoverflow.com/questions/12115397/is-it-against-license-to-publish-oracle-and-sql-server-performance-test https://stackoverflow.com/questions/12115397/is-it-against-l... https://www.brentozar.com/archive/2018/05/the-dewitt-clause-why-you-rarely-see-database-benchmarks/ https://www.brentozar.com/archive/2018/05/the-dewitt-clause-... https://news.ycombinator.com/item?id=14232976 https://news.ycombinator.com/item?id=14232976 (2017)

o11c 3y ago
Notably, it isn't against the law to call out "Oracle forbids us from benchmarking, so we are forced to rank it last".

deleted 3y ago

[deleted]

greatgib 3y ago

Can't such a block be considered as a proof against Google in an anti-trust trial? Google abusing of is position as the provider of the android app-store to protect his sales of phone as a manufacturer. That are different activities.

mcphage 3y ago
I’m not so sure. They don’t sell competing benchmark software, so it might not be relevant.

vore 3y ago

I would place my bet on this being some kind of app compatibility mishap rather than Google actively blocking Geekbench and 3DMark. Apparently Pixel 6a owners also had this problem, but with none of the Tensor G3 performance speculation in the article: https://www.reddit.com/r/GooglePixel/comments/wc4m3w/pixel_6a_not_compatible_with_benchmarking_apps/ https://www.reddit.com/r/GooglePixel/comments/wc4m3w/pixel_6...

fh9302 3y ago
There was no compatibility problem, Google intentionally blocked the installation to avoid reviewers testing the chip. The benchmarks worked without issues when sideloaded, but not every reviewer bothered to do that.
- BenjiWiebe 3y ago
  Sounds like a compatibility problem. I don't understand how exactly the play store works, but if your device string is unexpected (like maybe a new device? or a lineageos phone) some apps can't be installed from the play store but work fine sideloaded.
  - nfriedly 3y ago
    As a side note to this, Aurora store is awesome. If you sign in with your Google account, then you can install paid apps that you've purchased, but the Play store thinks are incompatible with your device. In my experience, the majority of them work perfectly fine!
  - _imnothere 3y ago
    Definitely not one, Play Store is known to showing apps as not supported while it actually is since years.
- vore 3y ago
  I think you are as blind to Google’s intentions as much as I am, but it sure seems silly for them to intentionally block it from the Play Store when side loading it is trivial, doesn’t it?

greesil 3y ago

FTA the performance difference isn't hugely worse from a comparable chip IMO. Maybe they did more damage by trying to suppress.

havnagiggle 3y ago

What exactly did they do here? X linked video is broken for me, and the article just says the performance is bad. My pixel 6 just got Android 14. Possibly just OS/new hardware related?

josephcsible 3y ago
When people using a Pixel 8 or Pixel 8 Pro tried to install benchmark apps from the Play Store, Google blocked the installation and falsely claimed "This app won't work for your device".
- havnagiggle 3y ago
  IMO that's not evidence of anything other than a bug with package manifests and the current Playstore version. I have to side load apps all the time on my kids tablets because of API incompatibilities that don't ultimately matter, and it isn't Google out to get my kids. According to the 3DMark playstore page, the last update was Sept 21, 2023. Version 2.3.4869. Android 14 came out October 4th, making any incompatibility likely a temporary problem. Unless there's a quote from 3DMark explaining how everything is dandy in their end, I'm not buying what you are selling.

pseudosavant 3y ago

I find it interesting that the Geekbench scores for the latest iPhone isn't mentioned. About the same performance as an iPhone 12. Yet, the Pixel 8 and iPhone 15 lines are price comparably. Pixel 8 Pro: Single: 1760 Multi: 4442 iPhone 15 Pro: Single: 2894 Multi: 7192 iPhone 12: Single: 1995 Multi: 4401

HumblyTossed 3y ago
With Apple and Google doing custom chips, I don't think that matters as much as people think anymore. Surely, a lot of functionality that used to be done in software is now done on chip. A generic benchmark isn't going to capture that.
- Arcanum-XIII 3y ago
  Like what? What's the ML performance on this chip? Can it help with game? Or apps I'll use everyday? Does it improve the web experience? The last one depends largely on those CPU benchmark... I want to be pragmatic here. Coprocessor were all the rage for the Amiga, then were out out of the loop, and were back for with the GPU. A bit like we're back to the mainframe experience with Kubernetes and all (don't throw tomatoes at me ;))
Skunkleton 3y ago
> Yet, the Pixel 8 and iPhone 15 lines are price comparably. But this isn't how pricing works. Even if it did the CPU isn't the most expensive component in the bom, and it isn't the most important component for user experience.
- Arcanum-XIII 3y ago
  The problem is that you can't easily and objectively factor the user experience. CPU are easy to benchmark — and do honestly factor a lot in the pricing. Android (the OS) development is payed for by other means, and the hardware line is kind of a vanity project. People may, or may not prefer the Android experience. That's not relevant here.

oldpersonintx 3y ago

every CPU benchmark needs to have another line drawn waaaay under all of the lines drawn for various CPUs...we shall label this line "performance you actually need and can make use of" if all of the CPUs are above this line, then you should not use CPU performance as a determining factor typing this happily from a 2014 Mobile Celeron laptop that is still fast enough to do all the software dev I need

sxp 3y ago

Citation needed that Google actively blocked installation of the benchmarking app. None of the links support that claim. If Google want to block benchmarking, they would have had their lawyers add that clause to the reviewers' contracts/NDAs as a condition of getting the device early. Since this is a technical issue, I'm guessing the benchmarking software didn't properly configure their Play Store account and mark the P8 or the new Android OS as compatible with the software. That would prevent it from being installed via the Play Store but not via sideloading. Disclosure, I used to work at Google on Android so I have some familiarity with how Play Store rollouts work.