11 ms·
Three chips in and Google Tensor is on life support
- userbinator 3y agoJust three Tensor chips in, and many of Google’s new AI features need to be off-loaded to the cloud for processing. They won't ever give up their ability to have control over our data.
- dylan604 3y agoI like to dish on Goog's constant slurping up of data as well, but I could be willing to give some benefit of the doubt that they just can't make it work locally. Oh, nevermind, I can't even type that with a straight face.
- jjeaff 3y agoI see no reason that Google, seeing as they control the whole phone OS, couldn't provide the benefits of local processing and still send all the data back to themselves, perhaps when on a charger and with WiFi.
- dylan604 3y agoSo you can only use the feature when plugged in? That's a limitation for a mobile device that pretty much renders it totally useless. Hey Google! Please do this thing for me. I'm sorry Dave, but I can't do that. You must be plugged in and connected to WiFi. Hey Google! I'm at my kids sportsball event, and that's not really conducive. I'm sorry Dave, I didn't get that. Besides, we're talking about 3 chip designs and still not there for on device processing at the same levels as competitor(s). So just punt and send it to the cloud.
- jjeaff 3y agoNo, it runs the process anytime and saves the data to be sent when on WiFi later.
- elromulous 3y agoIn general I agree that the tensor hardware has been mostly a disappointment. I mean this more in general performance than in ML. In ML I think the story is a little different. Hardware design/production pipelines are measured in years. The LLM "revolution" is relatively recent. For the Pixel8, I imagine some folks at Google had to make the really tough decision of whether to run weaker, smaller models on device, or run the giant LLM models in the cloud, giving much better results, albeit slower (and with ofc the requirement for an Internet connection).
- crest 3y agoI’m sure Google had to think long and hard if they wanted all those queries forwarded to their data centers to analyse (enrich their user profiles).
- wmf 3y agoThey could process stuff locally and also send data to the cloud.
- matrix_overload 3y agoThere's a big difference. If you process stuff locally, uploading the entire unprocessed data to the cloud would spark a lot of criticism. So, you are stuck running analysis locally and uploading things like relevant ad topics. If you run your processing through the cloud, there is no optics or bandwidth cost of keeping the entire raw data as long as you want, let humans review it, and figure out which metrics to extract from it at a later point. I would say, the latter raises more privacy concerns than the former.
- vineyardmike 3y agoYea but then they have to process everything. Sending just the relevant information for ads means they save on cost of running compute. Considering Azure has run out of GPUs and is renting from Oracle, it’s not crazy to imagine that they could run out of TPUs and the price of compute out weighs the slightly (?) worse ad targeting.
- SketchySeaBeast 3y agoMy problem with the pixel, and the only reason I returned it, is the modem. The chip was fine, I don't care that it's "underpowered", but when reception got spotty it chugged power. Google made an excellent SMARTphone, but they seem to have forgotten it's also supposed to be a smartPHONE and it can't always be on wifi where it will stay cool and efficient.
- AlotOfReading 3y agoI would bet quite a bit of money they didn't forget, based on the people I've known on the team. That's just Samsung's modem not being as good as Qualcomm. Choosing Qualcomm would have led to a very different phone where the feature differentiation they were trying to deliver with Tensor simply wouldn't have been possible. There are serious consequences to the middle ground solutions as well.
- runako 3y ago> Choosing Qualcomm would have led to a very different phone where the feature differentiation they were trying to deliver with Tensor simply wouldn't have been possible. Would you mind elaborating on this point?
- crest 3y agoIIRC Qualcomm puts their own Hexagon DSP on the SoC which is where they placed their own low precision matrix multiplication accelerator.
- maven29 3y agoHas Qualcomm ever engaged in significant semi-custom work? (no, adjusting dvfs for a special "for galaxy" SKU or adding pluton to 8cX gen 3 doesn't count). Samsung was a semi-custom partner for Apple in the early days of the iPhone (to the extent that the M1 still has some Samsung IP in it[0]). I heavily doubt that Qualcomm is even capable of working in such a role. In fact, the C-suite probably believes that their core competency is lawsuits and not modems, anyway. [0] https://twitter.com/calebccff/status/1472517091970494465 https://twitter.com/calebccff/status/1472517091970494465 (Edit: I meant semi-custom SoC design/architecture not cores, like https://www.anandtech.com/show/7281/understanding-amd-semi-custom-strategy https://www.anandtech.com/show/7281/understanding-amd-semi-c... )
- mostthingsweb 3y agoI just want decent on-device voice recognition for things like asking the Assistant to send text messages. So tired of having to repeat myself 5 times.
- callalex 3y agoThe latest Apple Watch models do this and it’s so much nicer.
- nmfisher 3y agoEven with the recent improvements to iOS speech recognition, I've felt that Android recognition has always been better. Not the case?
- smt88 3y agoI've extensively used both iPhones and Pixel phones for years. Siri is astonishingly, infuriatingly bad. It's hard to imagine anyone on the Siri team uses it daily like I do. It works well ~30% of the time and is bafflingly non-deterministic. Google is much, much better, but it's still not objectively great. It's closer to 80% of responses being reasonable, which is still unacceptably low.
- Angostura 3y agoSiri and speech recognition are somewhat different. iOS seems to transcribe my spoken words reall very well most of the time.
- mostthingsweb 3y agoThe actual recognition is quite good. It's just the handling of poor/intermittent network issues that is abysmal.
- cglong 3y agoI've had this going back to the Pixel 4. Do non-Google phones still not have this?
- jeffbee 3y agoThe actual experience of using Pixel ML features is extremely good, so it feels like the implementation details are inside baseball. Just like the last line of the article says: they are class-leading features. Speech-to-text and the reverse are the most noticable improvements over iOS.
- fooblaster 3y agoThe "on life support" title has absolutely no justification and is just a sad editorialization with no evidence from insiders at google. They could have just as easily said "tensor chip isn't so good with tensors" and gone with a more factual headline.
- dangus 3y ago“Tensor chip isn’t so good with tensors” “My car isn’t so good at driving.” Kind of sounds like life support.
- crest 3y agoAs if that had ever stopped one of the big established car companies from building more bad cars.
- rvnx 3y ago"Sorry I am just an AI chip, I can't help you running LLMs"
- pclmulqdq 3y agoYou laugh, but the requirements are actually very different between computer vision and other pre-2023 "AI" tasks and LLMs. LLMs are hopelessly memory bandwidth bound. Most other models are compute bound.
- wongarsu 3y agoAI goes through cycles where whichever field is currently making the most advances gets all the hype. But just because LLMs are currently flashy doesn't mean that object detection, image segmentation, facial recognition, image generation, text-to-speech, speech-to-text or anomaly detection are suddenly less useful.
- dangus 3y agoWell, bad cars don’t offload the driving to some kind of metaphorical cloud service. So I guess there’s some limitations to my analogy.
- dangus 3y agoThe reason Apple was right about on-device processing isn’t about the privacy marketing speak or anything like that, it’s because so many of cloud services are run at a loss and have little hope of ever becoming profitable businesses. Every time you use Google AI that’s offloaded to their cloud you’re basically getting free “money” from Google, and all of these functions are places where advertising can’t be inserted in a way that makes any sense. The most profitable cloud services are things like “sell storage that sits unused” or “sell a subscription to fitness classes.” If you want to run something complex and resource intensive you better hope you have some enterprise customers lined up.
- heyoni 3y agoIn another decade your photos will be enhanced but a Carl’s Junior will be added to the background.
- immibis 3y agoSelling egress bandwidth also seems to be very lucrative, at 50-100x cost price.
- deleted 3y ago[deleted]
- mst 3y agoAlso means that once you've got enough data accumulated inside a cloud service the situation becomes rather Hotel California.
- shoker 3y agoGenerative text AI (Large language models) require a ton of RAM for inference which just isn't a thing on mobile devices yet. It makes more sense to keep these features in the cloud vs making phones with unrealistic specs.
- borissk 3y agoMy experience using Pixel 7 is it has the smoothest UI of any Android phone I've tried - including flagships like S23 and Xiaomi 13T. Also the author assumes Google wants to do on-device AI processing, but given Google's desire for user data they may be uploading pictures and videos in the cloud just for the sake of having the data (and maybe use it to train say self-driving AI models ).
- Al-Khwarizmi 3y agoMy experience with my Pixel 6 Pro is also amazingly snappy UI and great software, but absolutely crap battery, the worst of any smartphone I've ever had (and I suppose that's the chip's fault, because the battery itself has a normal amount of mAh). The model is also quite unreliable, to the point that I sometimes disable 5G and 4G because they do more harm than good. I could live with the modem issue (3.5G is OK for 99% of my smartphone usage anyway), but having to care so much about battery running off is a quite important drawback for daily life. The fact that they seem to be keeping the same path with the chip makes me think that I'll have to look for another brand for my next phone.
- KJBweb 3y agoGot the 6 Pro too, me battery life isn't great either, but I WFH so I just mitigate with a power bank or spot charge for the times when I am on the go all day. Never had issues with reliability though, I've been really impressed with the hardware aside from the battery.
- Al-Khwarizmi 3y agoSorry, potentially very confusing typo. "The model is also quite unreliable" should read "The modem is also quite unreliable". I only have reliability problems with the data connection (specifically 4G and 5G), in other aspects the phone works fine.
- dd_xplore 3y agoI haven't used 3G/3.5G in over a decade now(almost). Even 90% of our carriers have disabled 3G services , some are 4G/5G only and others support 2G/4G/5G
- singularity2001 3y agowhy did they never sell their tensor chips as stand-alone components in competition to NVIDIA?
- Zetobal 3y agonobody would buy them without google substituting it. the chips are just not competetive.
- singularity2001 3y agomaybe the economics of scale would have kicked in after some substitution?
- lvh 3y agoThey sold (and continue to sell) TPUs under the Coral brand, and they're pretty popular for edge inference (e.g. Frigate).
- singularity2001 3y agohttps://www.amazon.com/s?k=Coral+tpu https://www.amazon.com/s?k=Coral+tpu interesting, these are raspberry toy processors, maybe they should have released 'real' ones.
- lvh 3y agoThey may often be used with small embedded computers, but 1 Coral TPU handily beats even a top-end CPU for the stuff Frigate needs it to do -- and it does it at minuscule power draw. Sure, it's inference-focused, and that means it has limitations: it sucks for e.g. speech models, too. But it's pretty great at what it does and gets used in real applications too. My guess is you saw the USB-C model, but they do M2, big cards with multiple M2 slots, and miniPCIe (popular in industrial applications) too.
- mikecoles 3y ago'Toys' that work for the intended function makes them tools. Why blow money on hardware and power when a Coral will do the job well?
- blackoil 3y agoDoes the article has anything to say apart from clickbait? From article, Google is working on a 3nm chip with TSMC, so it is surely not abandoned. Current chip sometimes run hot, that's bad but not an existential crisis. Not all AI things can be on-device, was that ever a promise? Is that even feasible? Is any competitor doing better? LLMs with 100s of billions of params can't be run on a beefy desktop, so why is it a surprise or concern that bard can't run on Pixel.
- Roark66 3y agoI came here to say exactly the same. Where is any substance in this article? What are the capabilities of the chips? How many Tflops can they do? I'm not buying it. I have a couple of Google's (mini) Edge TPUs doing object recognition in my Cctv server. They have been working 24/ 4 for about 2 years and are consistently delivering on the 4W/4Tflops promise(although for some reason they stopped advertising this number focusing on benchmarks instead). I for one wish Google would have us their big (or even medium) sized TPUs to play with on an add on card rather than in their phone.
- brucethemoose2 3y ago> I for one wish Google would have us their big (or even medium) sized TPUs to play with on an add on card rather than in their phone. Yeah, no kidding. I know they want to push GCS, but what a boon that would be for TPU support in ML projects.
- deleted 3y ago[deleted]
- dacryn 3y agoheadlines of google products being end of life and perception of being killed off generates a lot of clicks, and it is so tiring
- gorbypark 3y agoWhile intentionally a bit click-baitey, I think the author is saying that features _using_ the ML tensor cores are on life support. They announced/hinted at that a bunch of features would use on-device ML and in the end they are just using the cloud like they always have.
- atorodius 3y agoChips take a lot of time from design to roll out. Giant LLMs being something you want to serve is a very recent development and it will take time for the deployed hardware to catch up, no surprises here.
- elAhmo 3y agoOther than bringing up Google Assistant with Bard, the article doesn't mention giant LLMs at all. It lists features such as Gboard proof-reading that are not done on device, as well as the chips heating up when doing some of the general tasks such as downloading, so from that regard, it seems that most AI tasks are not done on the device. Which was the promise sold with these chips.
- moffkalast 3y agoThe NCS2 looks really pathetic now with its 512 megs of RAM. Nobody seemed to expect what the demands will be just a few years ago.
- Havoc 3y agoThe bigger issue imo is Google backed the wrong horse. PyTorch vs tensorflow I gather PyTorch on Google hardware now works but it’s kinda stolen their thunder
- peterhull90 3y agoCan anyone just clarify for me what these AI / Machine learning chips are? As far as I can tell they are general purpose microprocessors with some added instructions which accelerate operations that are commonly used (matrix multiplication possibly?) but there's a lot I don't quite understand because some of the info is marketing and some of it it heavy technical stuff and my knowledge falls in between!
- electricships 3y agohere is my good deed for the day: modern AI is just vector multipication. any AI chip is just 10,000s of very simple cores which can do vector float operations and little else. this also entails clever trade offs of shared cache and internal bandwidth. (as a thought experiment, consider a naive million by million matrix multipication. this will take a single cpu about 1 year! how do we reduce this to 1s ?) the end
- automatic6131 3y ago>vector float operations and little else I thought they were generally int8 or int16 vector multiply adds and occasionally float16 added in.
- deleted 3y ago[deleted]
- exikyut 3y agoAs someone with a lot of interest in but no fluency with chip design, or the dividing and conquering of math within silicon, for that matter, how would you multiply a 1m² matrix?
- financltravsty 3y agoParallelization. Each "unit of work" in matrix multiplication is not dependent on any other unit of work. Stuff as many cores as you can into a chip, and then simply feed in all your vectors at the same time. I.e. basically a beefed up GPU or an "AI" chip.
- phkahler 3y agoMaybe if I had better understanding of the Pixel capabilities I may have bought one. Maybe Google can't market anything because they're so entrenched in the least effective advertising methods out there? Where are the catchy short commercials on YouTube compelling me to this great tech? Just because everyone knows Google doesn't mean they know what Google has to offer.
- jmcgough 3y agoGoogle is absolutely awful at marketing. Strada was really impressive tech that let me play AAA games almost seamlessly on my old Thinkpad, and none of my friends knew what Strada was because of how it was marketed.
- justinsaccount 3y agoThey might have heard of Stadia.
- ratg13 3y agoI’ve heard the name, but honestly could not tell you what it is.
- deleted 3y ago[deleted]
- theshackleford 3y agoEveryone In my circle knew what stadia was, and knowing it was google was enough for them to avoid it. A choice that was ultimately the correct one.
- JOnAgain 3y agoDo you mean Stadia? Is it ironic you got the name wrong. It wasn’t good enough. It purported to be great, but it had random connection issues. Given that destiny, a 1st person shooter, was their flagship game these flaws really jumped out. You also really needed good internet. Like really good. Mine performed well sometimes when it was hardwired, but lagged regularly over WiFi. I think these things were fixable, but that thing was dead within a month of launch. Google didn’t get fully behind it, so all the people that launched it got their promos, saw the writing on the wall, and transferred out.
- fredgrott 3y agoWhat does this about specialized AI vs AGI? It would seem that an investor bet like with bitcoin would be on the specialized SoC not the tech above the chip itself.
- KBme 3y agoAll I see is a lot of cope and bickering about small things. Fact of the matter is that google doesn't make good interesting and innovative products because it has started drinking it's own cool aid. About at the same time as when they started becoming evil, getting political and influencing elections everywhere, they also stopped even trying to make innovative products and instead went full communist. Everyone knows the effect that has on innovation but somehow SF techies and american wokies are unable to comprehend it, and vehemently deny it.
- refulgentis 3y agoDude what? Lmao. I'd relax on my politics and ragebait consumption for a year or two if I were you, this is unintelligible
- SomeHacker44 3y agoAs a user of all three chips vis Pixels 6, 7 and 8 Pro as well as this current Pixel Tablet, I do not get it. These devices work fine for me. I have also had Samsung and OnePlus devices. I have never had performance problems. I switched from Samsung to Pixel due to the messy software situation- there were two or more apps for every purpose and which to use was incomprehensible to me, and I am not too dumb. Granted I am not a heavy user of things that might need extreme capabilities. I do not play complex games or do media creation.
- ranting-moth 3y agoA friend of mine says that all Google products are on life support. It's just how long they get to stay there before the cable is pulled.
- luuurker 3y agoI'd like Google to go back to Snapdragon, but this is just an opinion piece. It doesn't back up the claim that Tensors are "on life support". Tensors are not as fast or as efficient as the alternatives (the reason why I didn't pick the 8 Pro recently and went with a Galaxy S23 Ultra), but it's fast enough for most users and the efficiency could probably be fixed by using TSMC instead of Samsung, just see the difference between Qualcomm's 8Gen1 vs 8+Gen1.