26 ms·
The Pixel 8 Pro's Tensor G3 off-loads all generative AI tasks to the cloud
- jan_Sate 3y agoIf they're offloading to the cloud, what's the AI processor in Tensor G3 even used for?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- coder543 3y agoIt is used for a few non-generative tasks like speech recognition, but other chips are also capable of these tasks. The only thing the Tensor G3 seems to be better at is avoiding Qualcomm's habit of EoLing consumer smartphone chips.
- deleted 3y ago[deleted]
- fh9302 3y agoIf that's the case, why do phones with the Tensor G1 and G2 only receive 3 major updates? That even applies to the Pixel Fold, a $1800 phone released a couple months ago. There's no Qualcomm to blame here, it appears to be purely a business decision by Google.
- coder543 3y agoWhere did I mention the Tensor G1 and G2? I didn't. I only said "Tensor G3". Yes, feel free to blame Google for not supporting the G1 and G2 better. None of that changes that Qualcomm does not support consumer smartphone chips beyond a couple of years.
- fh9302 3y agoCan you show me evidence that Qualcomm refuses to support chipsets even in the case where they would get paid to extend support?
- coder543 3y agoFairphone made some claims about this when they launched the Fairphone 5. That's the sole reason it's not using a normal consumer-facing Snapdragon chip, and instead using some kind of industrial Qualcomm chip. Fairphone pointing this out wasn't exactly news to anyone who has been paying attention to this stuff. I also wouldn't be too surprised if Qualcomm starts to change their tune under pressure from other manufacturers who want to stay competitive with Google.
- fh9302 3y agoMishaal Rahman claims it's purely a business decision and not an artifical limitation set by Qualcomm: https://twitter.com/MishaalRahman/status/1710442249778086260 https://twitter.com/MishaalRahman/status/1710442249778086260
- coder543 3y ago"Business decision" and "artificial limitation" are frequently synonymous. I think @MishaalRahman was correct to say "If OEMs keep buying a particular chipset [...] then it'll continue being supported", especially with Qualcomm, if the sales volume remains high enough. I also agree it is generally correct to say you can pay an SoC vendor to continue supporting a chipset, but that doesn't seem to have been correct for Qualcomm. If a particular mid-range chip was still selling well years after release, then Qualcomm may have seen fit to continue supporting it so that those new phones could continue selling well. Qualcomm primarily cares about selling chips. Supporting old chips that have already sold (and aren't continuing to sell well) does not help Qualcomm sell more chips, so they haven't wanted to do it. I think Qualcomm may come around to supporting chips better (maybe even at their next announcement in a few days), but their historical behavior in this regard has been less than ideal. In fact, Qualcomm cares so much about selling chips that they've allegedly nearly sunk the Oryon SoC in the process.[0] They apparently saw an opportunity to force-bundle more chips, in this case the PMICs, which were so unsuitable that the manufacturers wanted to buy them and literally throw them in the garbage just to keep Qualcomm happy, while using alternative PMICs instead. But, Qualcomm supposedly baked the decision into Oryon so deeply that only Qualcomm's PMICs are compatible. Apparently, this is causing manufacturers to consider abandoning Oryon entirely. Most people would logically think that Qualcomm cares primarily about making money, but instead their first priority actually seems to be selling chips, even if it means leaving money on the table, regardless of whether that is logical or not. Why did Fairphone choose to use an industrial SoC if they could have just paid Qualcomm a little more money to extend the life of the Snapdragon 8 Gen 2? [0]: https://semiaccurate.com/2023/09/26/whats-going-on-with-qualcomms-oryon-soc/ https://semiaccurate.com/2023/09/26/whats-going-on-with-qual...
- fidotron 3y agoI think the truth is the Tensor is a barely modified Samsung Exynos.
- coder543 3y agoThe Tensor G1 was probably pretty close to that, but I think subsequent generations have been diverging more. They are still made at Samsung's fabs, and I wouldn't be surprised if Samsung is still helping to design the chips. Either way, I haven't seen anything to be impressed by when it comes to the Tensor SoCs themselves. The performance is lackluster, and in spite of Google's marketing, the AI acceleration isn't clearly better than what Qualcomm or Apple are offering.
- kcb 3y agoAll the tensor chips so far have had different CPU core configurations than available Exynos chips. So it isn't just a rebadge.
- imjonse 3y agoMaybe it has hw acceleration support for HTTP operations :)
- baz00 3y agoMarketing street cred.
- jauntywundrkind 3y ago> For all of this generative AI stuff, for anything that actually has to use AI to create things like the AI Wallpaper making, the Magic Editor needs a permanent internet connection... It feels so sluggish, that you are constantly reminded that it's not running on-device... This is fine. Generative AI has gotten good by being beastly huge, consuming gobs of resources. It's not particularly fast even on >$1000 consumer GPUs. It feels like it would be an enormous waste of time and effort to try to scale these generative tasks down to small on-device footprints, where they can fit and run in reasonable time, knowing how much worse the results would be. I don't see what the alternative is here. But it certainly does leave the question open of what exactly are the great tensor tasks for the edge. With chipmaker like AMD and Qualcomm and ARM all putting on sizable neural/tensor cores too, the question is a somewhat concerning one.
- Moldoteck 3y agoIt's fine, but then this means you have a poor processor that is not even capable of doing what it was advertised for. G3 is worse than snap gen2 in terms of performance, maybe close to gen 1(and I'm not sure battery consumption is comparable) and snap is about to release gen3
- jessfyi 3y agoIt's not fine when this "magic" is being advertised as on-device. After reading (and attempting to quickly implement the models ensembles within) both the RealFill[0] and Break-A-Scene[1] papers published from Google researchers just prior to the Pixel 8 launch I was expecting either a leap in their G3 tensor core akin to 2013 Moto X NLP+contextual awareness cores[2] (which provided better implementations of Active Display, gesture recognition, and voice recognition in loud environs than 95% of current mobile devices) or the Coral[3], the edge TPU they developed that got shockingly amazing inference performance from (though HW production handed off to ASUS in 2022--thanks to the chip shortage, the general arbitrary nature of the company, and their wholesale divestment from IoT) I expected more. All that to say this: your assumptions of inference performance on >$1000 hardware are fundamentally flawed (the fact that you reach for the buzzy "generative" prefix suggests they're erroneously informed by twitter influencers and attempting to deploy current LLMs.) Custom hardware can and has been developed in the past (on mobile devices) that could've been tailored to the task at hand. If they failed to meet performance, power draw, or processing time requirements, they should've reframed their pitch instead of exposing themselves to what is likely going to be yet another class action suit focusing on their hardware. [0] https://realfill.github.io/ https://realfill.github.io/ [1] https://omriavrahami.com/break-a-scene/static/paper/Break-A-Scene.pdf https://omriavrahami.com/break-a-scene/static/paper/Break-A-... [2] https://en.wikipedia.org/wiki/Moto_X_(1st_generation)#Hardware https://en.wikipedia.org/wiki/Moto_X_(1st_generation)#Hardwa... [3] https://coral.ai/ https://coral.ai/
- user_7832 3y agoQuote from Google: > Our work with Tensor has never been about speeds and feeds, or traditional performance metrics. It’s about pushing the mobile computing experience forward. And in our new Tensor G3 chip, every major subsystem has been upgraded, paving the way for on-device generative AI. It includes the latest generation of Arm CPUs, an upgraded GPU, new ISP and Imaging DSP and our next-gen TPU, which was custom-designed to run Google’s AI models. Then why the internet requirement? Were they trying something that failed last minute forcing them to ship it the way it is? Or was this always just advertisement material?
- ForkMeOnTinder 3y ago> Were they trying something that failed last minute Seems likely. Your chip underperforms? Just block people from using benchmarking apps, and also tell the press you never cared about speed. Problem solved.
- josephcsible 3y agoSee https://www.notebookcheck.net/Google-blocked-Pixel-8-Pixel-8-Pro-reviewers-from-using-popular-benchmarks-to-test-the-Tensor-G3-chip-new-owners-too.759260.0.html https://www.notebookcheck.net/Google-blocked-Pixel-8-Pixel-8... for details about "block people from using benchmarking apps".
- AlexandrB 3y agoWtf? How is this not an abuse of market power?
- canucker2016 3y agothe notebookcheck.net article states: "In a highly unusual move, it has been revealed that Google has blocked reviewers of the Pixel 8 and Pixel 8 Pro from installing popular benchmarks including Geekbench and 3D Mark." there's an embedded URL linking to a YouTube video, which shows that the Google Play store won't allow the user to install Geekbench or 3dMark on the pre-released Pixel 8 review devices. see https://www.youtube.com/watch?v=bk4ZUmKqRm0&t=330 https://www.youtube.com/watch?v=bk4ZUmKqRm0&t=330
- retskrad 3y agoGoogle is not a hardware company. Their hardware can never be trusted.
- xgudwilx 3y agoGoogle's Ai says: The Google Pixel 8 Pro can run generative AI models locally on the device, but not all of them. Google announced at the Made by Google event in October 2023 that the Pixel 8 Pro's custom-built Tensor G3 chip can run "distilled" versions of Google's text- and image-generating models. These models can power a range of applications on the phone, such as image editing and smart replies in Gboard. However, some generative AI tasks, such as running large language models like Bard, still require too much computing power to run locally on a smartphone. These tasks are offloaded to the cloud, where Google has access to more powerful servers. Here are some examples of generative AI models that can run locally on the Google Pixel 8 Pro: Magic Eraser Zoom Enhance Best Take Audio Magic Eraser Gboard Smart Replies AI summaries in Google Recorder Google is still working on developing new ways to run generative AI models locally on devices. As the Tensor chip continues to improve, we can expect to see more and more generative AI features running on-device in future Pixel phones. ʘ ‿ ʘ
- avipars 3y agoWhat is the point of hyping up the G3 then?
- clnq 3y agoWeasel words marketing is still marketing.
- dicriseg 3y agoThe headline immediately reminded me of the Juicero. How can they advertise on-device if they’re immediately sending your data elsewhere?
- nmstoker 3y agoIt's strange as the Magic Editing feature, whilst showing promise turned out to be massively underwhelming... It's kind of slow, so you can't really experiment freely and the results usually aren't that great (maybe with practice it would be easier to get good results but that's not happening due to the clunky speed)
- quitit 3y agoGoogle's smartphone/AI announcements are used to portray them as being massively ahead of their smartphone competitors. The actual product on the other hand is not what they advertise. Take Duplex for example. It was meant to be this flexible and natural AI that even the store wouldn't notice. In 2023 it's a clunky chatbot experience, it has a patchwork of features depending on the user's location and sounds so much like a robocall that there is a problem with how many stores hang up on it.
- annaaa 3y agoIf I remember correctly, Duplex was intentionally modified this way after controversy from the initial demo, when, during a call, it didn't obviously identify itself as an automated system.
- quitit 3y agoHaving to identify as a chatbot and provide a call back number was a requirement under some states' laws - while this seems like something they should have looked into before advertising it, my comment was with a wider scope to how it performs: it's not flexible and it doesn't sound natural.
- causality0 3y agoMan I'd kill for a Pixel 8 Pro with a snapdragon.
- sergiotapia 3y agoThe days of pining for the highest geekbench score are over. It's all about utility and how much I can do with my phone. I don't care about how many cpu megarams my phone has. Can I fix a picture I took of my wife and kids as they passed by on the Hulk rollercoaster?
- causality0 3y agoFor me it's not about performance, it's about the Mali GPU cores. They seriously gimp game console emulation.
- sva_ 3y agoI think the Snapdragon 8 Gen 2 does just fine with doing some matmuls, just as the Pixel tensor chip does. Moreover, the Snapdragon uses less battery overall and stays cooler. But the differences seem marginal.
- wkat4242 3y agoThere's a huge difference in battery life between Snapdragon and Exynos devices though. It's so clear because Samsung makes the same device in both configurations. For tensors we don't have such an exact comparison but it does seem that Samsung's fab plays a role in this. And battery life is still very much a hot topic.
- fotta 3y agoHas anyone looked at network traffic while using the Magic Editor to see if it’s actually offloading the processing? Or is this just an inferred conclusion based on needing an internet connection? Is it possible that it requires an internet connection to fetch new models or something but the processing is still on device?
- partiallypro 3y agoThis seems like the most likely answer to me, but getting little fanfare here.
- privacyking 3y agoThe magic editor (rainbow button on Google photos) only works if you enable cloud backup, so yes.
- hedora 3y agoAll sorts of stuff in photos breaks in degoogled phones. It wouldn’t surprise me at all if they decided to disable on device features unless you sent them your data so they could monetize it, share with oppressive regimes, etc.
- izacus 3y ago"Tensor" isn't offloading anything, it's the Android apps that are doing that to Google cloud. Not sure what the SoC has to do with that?
- wmf 3y agoBut why don't those apps use the local AI engine?
- abeyer 3y agoBecause ads aren't local
- kibwen 3y agoThe mothership needs to inform the phone whether it should inject a subliminal Pepsi logo into the generated image, or a subliminal Nike logo.
- izacus 3y agoEr, have you seen the hardware requirements for latest high quality AI models? They won't even run on your MacBook, much less on a mobile SoC. This really shoulnd't be a hard thing to check on a HACKER news forum which constantly talks about modern AI.
- jrflowers 3y agoWho wrote the Android apps you’re talking about?
- baby_souffle 3y agoThis makes some amount of sense: generative AI is _very_ expensive. But generative is only one class of AI workload; predictive/inference is probably what the tensor is used for most. I would not be shocked to find out that the "find object in this photo" part of magic eraser worked on device but the "figure out what to put in picture after object removed" bit is done server side.
- lopkeny12ko 3y agoEverything old is new again. After delivering technology over the last 20 years to seamlessly offload arbitrary computation to a distributed cluster of dedicated hardware, managed entirely transparently by people you've never met, yes let's just go back to slow, local compute because that's "cooler" now or something.
- randyrand 3y agocooler, fully reliable, and privacy preserving (ideally).
- liuliu 3y agoIt is a lie that we tell people "your compute is offloaded to the cloud" until now. User-centric heavy compute in the past 20 years simply don't happen in "the cloud". Monetization and engagement tricks yes (ads bidding, behavior analysis, friend / content recommendations, content recompression), but not user-centric ones.
- whartung 3y agoIt's going to be an interesting conundrum for Apple, with their drive to keep everything on phone. Both the computing requirements and the data requirement on modern AI make it a challenging fit. So it will be a curiosity about how Apple tries to balance a feature set that they can power on the phone, net take advantage of the modern AI capabilities. Or, just bail, and funnel it to the net.
- wallaBBB 3y agoMy initial reaction was completely opposite to yours: - Oh this is going to burn google's money, especially if the phone becomes popular. Unless you want to charge a subscription for a feature it really isn't smart for Apple to try something like this.
- foota 3y agoAren't the M1 and friends relatively capable for LLM tasks?
- brucethemoose2 3y agoSort of. Apple is super stingy about RAM capacity on their devices, which is a problem for GenAI Also, the M1 NPU (the equivalent to Google's offloading) is not very fast in Apple's own Stable Diffusion port, and I'm not aware of any community LLM frameworks that even use the NPU. They all run Metal implementations on the M1's GPU.
- ActorNightly 3y agoApple doesn't care about on or off phone, its only goal is to make trendy devices that sell because of the badge.
- astrange 3y agoApple spends over $20 billion a year on R&D.
- 3y ago
- kelnos 3y agoGoogle said: > And in our new Tensor G3 chip, every major subsystem has been upgraded, paving the way for on-device generative AI. This is definitely weasel-worded, but note that this does not actually have to mean that all the generative AI stuff can be done on Tensor G3. They could (credibly, but, again, weasel-y) claim that the stuff they've done on G3 is prep work that is "paving the way" for a future chip to be able to do it all. But either way, I'm not terribly surprised? Generative AI on a mobile SoC (even with special-purpose hardware) still sounds like a bit of a stretch at this point, no? At least, doing it on-device with acceptable performance and power consumption seems a little unlikely?
- dagmx 3y agoThere are apps on iOS that do run stable diffusion like models locally on device. https://apps.apple.com/ca/app/draw-things-ai-generation/id6444050820 https://apps.apple.com/ca/app/draw-things-ai-generation/id64... I believe the Tensor chip is powerful enough to do the same but I suspect the efficiency is low enough that they don’t want to run it locally.
- hotstickyballs 3y agoThat’s the difference between apple approach and google approach. Apple does a lot of computation on your own devices (in the name of privacy and what not) and ends up saving a lot of server compute. Googles strategy so far has been just burning cloud compute costs.
- valianteffort 3y agoFrom what I understand google's hardware doesn't measure up to apple even when just comparing ML cores.
- fooblaster 3y agoGoogle has an enormous advantage over apple in data center compute. For one thing, they don't have to pay the Nvidia monopoly tax to do inference. It's silly to color this as a disadvantage.
- 5cott0 3y agoOn-device is already possible on mobile? https://apps.apple.com/us/app/tarot-raven/id1539747669 https://apps.apple.com/us/app/tarot-raven/id1539747669
- edandersen 3y agoNow we know where the “7 years of support” came from - that’s when they will/can turn the servers off, remotely removing advertised features.
- astrange 3y agoIt's because that's the storage writes and repair parts availability lifespan.
- kristjansson 3y agoDuh? High quality generative features on-device would be a shocking leap.
- tortoise_in 3y agoBut If they won't say it then no one will buy it. But last it's still cleanest Android experience and you can flash custom roms anytime.