2 ms·
> For all of this generative AI stuff, for anything that actually has to use AI to create things like the AI Wallpaper making, the Magic Editor needs a permanen
by jauntywundrkind 3y ago
> For all of this generative AI stuff, for anything that actually has to use AI to create things like the AI Wallpaper making, the Magic Editor needs a permanent internet connection... It feels so sluggish, that you are constantly reminded that it's not running on-device...
This is fine.
Generative AI has gotten good by being beastly huge, consuming gobs of resources. It's not particularly fast even on >$1000 consumer GPUs.
It feels like it would be an enormous waste of time and effort to try to scale these generative tasks down to small on-device footprints, where they can fit and run in reasonable time, knowing how much worse the results would be.
I don't see what the alternative is here. But it certainly does leave the question open of what exactly are the great tensor tasks for the edge. With chipmaker like AMD and Qualcomm and ARM all putting on sizable neural/tensor cores too, the question is a somewhat concerning one.
- Moldoteck 3y agoIt's fine, but then this means you have a poor processor that is not even capable of doing what it was advertised for. G3 is worse than snap gen2 in terms of performance, maybe close to gen 1(and I'm not sure battery consumption is comparable) and snap is about to release gen3
- jessfyi 3y agoIt's not fine when this "magic" is being advertised as on-device. After reading (and attempting to quickly implement the models ensembles within) both the RealFill[0] and Break-A-Scene[1] papers published from Google researchers just prior to the Pixel 8 launch I was expecting either a leap in their G3 tensor core akin to 2013 Moto X NLP+contextual awareness cores[2] (which provided better implementations of Active Display, gesture recognition, and voice recognition in loud environs than 95% of current mobile devices) or the Coral[3], the edge TPU they developed that got shockingly amazing inference performance from (though HW production handed off to ASUS in 2022--thanks to the chip shortage, the general arbitrary nature of the company, and their wholesale divestment from IoT) I expected more. All that to say this: your assumptions of inference performance on >$1000 hardware are fundamentally flawed (the fact that you reach for the buzzy "generative" prefix suggests they're erroneously informed by twitter influencers and attempting to deploy current LLMs.) Custom hardware can and has been developed in the past (on mobile devices) that could've been tailored to the task at hand. If they failed to meet performance, power draw, or processing time requirements, they should've reframed their pitch instead of exposing themselves to what is likely going to be yet another class action suit focusing on their hardware. [0] https://realfill.github.io/ https://realfill.github.io/ [1] https://omriavrahami.com/break-a-scene/static/paper/Break-A-Scene.pdf https://omriavrahami.com/break-a-scene/static/paper/Break-A-... [2] https://en.wikipedia.org/wiki/Moto_X_(1st_generation)#Hardware https://en.wikipedia.org/wiki/Moto_X_(1st_generation)#Hardwa... [3] https://coral.ai/ https://coral.ai/
- jpalawaga 3y ago> Custom hardware can and has been developed in the past (on mobile devices) that could've been tailored to the task at hand. You think google doesn't know that? are you aware of what's inside of Google's phones? I'm not sure what performance benefit you expect out of custom hardware. How many orders of magnitude? You're going to probably need at least a few, probably more, to make generative AI work well in the palm of your hand. Oh, and if you've figured that out, Apple, Google, OpenAI, and other AI companies would like a word.
- Topfi 3y ago> But of course, on-device generative AI is really complex: 150 times more complex than the most complex model on Pixel 7 just a year ago. Tensor G3 is up for the task, with its efficient architecture co-designed with Google Research. This is a direct quote from an official press release [0]. They claimed Tensor G3 is "up for the task" that is "on-device generative AI". I'd say "if you can't do it, simply don't promise it", but the fact is, this is the third time Tensor has been outright incapable of what was promised. People pointing that out are more than justified. Prior to the launch of the Pixel 6, with their first generation of Tensor SOC, they made big promises concerning HDR video performance [1], implying heavily or outright stating (depending on whom generous you want to be) that they'd finally manage to be on par with Apple. They weren't, by a lot. Pixel 6 video performance was neither on par with Apple nor did it exceed the Pixel 5 on an upper-mid SD765G. Still, first-gen and a bit of overhyping happen to the best of us. During the Pixel 7 launch [2], they claimed Tensor G2 enabled users to finally get computational photography for high-quality videos. Spoiler alert: It didn't. Fool me once... Now, on the Pixel 8 with their third generation of Tensor, they finally have a solution that gets their nighttime video processing results competitive with the current iPhone in the form of Video Boost. Instead of doing that processing on their amazing Tensor SOC though, they offload that to the cloud [3]. At least they didn't promise on-device processing improvements to video with the G3, only a tone of GenAI capabilities... I have followed Tensor extensively, and I am happy to see that they are at least utilizing their control over the silicon to provide a longer update cycle. But few of their local processing promises have held water, and even fewer appear to be impossible on contemporary SOCs from competitors such as Qualcomm (who are by no means angles and need all the competition the market can provide). If the Pixel team were more honest about their SOCs capabilities and proactively transparent on what they run locally vs off-load to datacenters, that'd be appreciated. With Video Boost they did just that, though I fear that was mainly because of the upload times... [0] https://blog.google/products/pixel/google-tensor-g3-pixel-8/ https://blog.google/products/pixel/google-tensor-g3-pixel-8/ [1] https://9to5google.com/2021/08/02/google-pixel-6-video-hdr-teaser/ https://9to5google.com/2021/08/02/google-pixel-6-video-hdr-t... [2] https://www.youtube.com/live/2NGjNQVbydc?si=2Gg1mPrdOkmu1L44&t=1701 https://www.youtube.com/live/2NGjNQVbydc?si=2Gg1mPrdOkmu1L44... [3] https://blog.google/products/pixel/google-pixel-8-pro/ https://blog.google/products/pixel/google-pixel-8-pro/
- hiddencost 3y agoSpeech recognition, wakeword detection, object recognition, auto complete, ... Wakeword especially is a continuously running process that needs a relatively small foot print CNN over a fixed window, where power is extremely important.
- ip26 3y agoLiterally anything latency sensitive is appropriate for edge. Off-device voice recognition is a miserable experience anytime you have less than a perfect connection. Imagine clicking an object in photoshop, using AI to determine the edges… and every time you click, photoshop bundles the image, uploads it to the cloud, and waits five seconds for every select clicking to be scheduled, dispatched, and returned. Imagine running the camera app with ML object recognition focusing, except the cloud is focusing your camera and tracking objects in the viewfinder, with a three second lag.