5 ms·

Gemini Nano sounds like the most exciting part IMO. IIRC Several people in the recent Pixel 8 thread were saying that offloading to web APIs for functions like

Gemini Nano sounds like the most exciting part IMO. IIRC Several people in the recent Pixel 8 thread were saying that offloading to web APIs for functions like Magic Eraser was only temporary and could be replaced by on-device models at some point. Looks like this is the beginning of that.

xnx 3y ago
I think a lot of the motivation for running it in the cloud is so they can have a single point of control for enforcing editing policies (e.g. swapping faces).
- bastawhiz 3y ago
  Do you have evidence of that? Photoshop has blocked you from editing pictures of money for ages and that wasn't in the cloud. Moreover, how does a Google data center know whether you're allowed to swap a particular face versus your device? It's quite a reach to assume Google would go out of their way to prevent you from doing things on your device in their app when other AI-powered apps on your device already exist and don't have such policy restrictions.
  - xnx 3y ago
    When I try to remove the head of a person using Magic Editor, I get the message "Magic Editor can't complete this edit. Try a different edit." Also documented here: https://www.androidauthority.com/google-photos-magic-editor-prohibited-edits-3383291/ https://www.androidauthority.com/google-photos-magic-editor-... I have no doubt Google could (and might) enforce a lot of these rules on the device, but they likely route it through the cloud if there's a new "exploit" that they want to block ASAP instead of waiting for the app to update. This is an example of the reputational risk Google has to deal with that small startups don't. If some minor app lets you forge photos, it's not a headline. If an official Google app on billions of devices lets you do it, it's a hot topic.
    - bastawhiz 3y ago
      It could simply also be that their inpainting model is quite bad at certain things, and replacing a person's head produces consistently bad results. Hiding the problem could simply be easier than fixing it.
sva_ 3y ago
Other phones already do this on-device
angm128 3y ago
Yeah, even more exciting than the small PaLM2 from Google IO
TheCaptain4815 3y ago
Unless they make it open source, what benefits would it have compared to the tiny OpenAi models?
- WatchDog 3y ago
  The fact that it's multimodal is very interesting, they might not make it open source, but if they intend run it on people's devices, even if they intend to implement DRM, someone will figure out how to extract the weights and get it running outside.
pablobaz 3y ago
> "Using the power of Google Tensor G3, Video Boost on Pixel 8 Pro uploads your videos to the cloud where our computational photography models adjust color, lighting, stabilization and graininess."* I wonder why the power of Tensor G3 is needed to upload your video to the cloud... *https://blog.google/products/pixel/pixel-feature-drop-december-2023/ https://blog.google/products/pixel/pixel-feature-drop-decemb...
- solardev 3y ago
  It runs an on-device LLM to generate a HTTP POST every time. It took four interns half a week to reduce the hallucinations, but a PM got a promotion after that.
- gundmc 3y ago
  Using the power of a Bugatti Veyron, opens and closes the garage door.

xnx 3y ago

I think a lot of the motivation for running it in the cloud is so they can have a single point of control for enforcing editing policies (e.g. swapping faces).

bastawhiz 3y ago
Do you have evidence of that? Photoshop has blocked you from editing pictures of money for ages and that wasn't in the cloud. Moreover, how does a Google data center know whether you're allowed to swap a particular face versus your device? It's quite a reach to assume Google would go out of their way to prevent you from doing things on your device in their app when other AI-powered apps on your device already exist and don't have such policy restrictions.
- xnx 3y ago
  When I try to remove the head of a person using Magic Editor, I get the message "Magic Editor can't complete this edit. Try a different edit." Also documented here: https://www.androidauthority.com/google-photos-magic-editor-prohibited-edits-3383291/ https://www.androidauthority.com/google-photos-magic-editor-... I have no doubt Google could (and might) enforce a lot of these rules on the device, but they likely route it through the cloud if there's a new "exploit" that they want to block ASAP instead of waiting for the app to update. This is an example of the reputational risk Google has to deal with that small startups don't. If some minor app lets you forge photos, it's not a headline. If an official Google app on billions of devices lets you do it, it's a hot topic.
  - bastawhiz 3y ago
    It could simply also be that their inpainting model is quite bad at certain things, and replacing a person's head produces consistently bad results. Hiding the problem could simply be easier than fixing it.

sva_ 3y ago

Other phones already do this on-device

angm128 3y ago

Yeah, even more exciting than the small PaLM2 from Google IO

TheCaptain4815 3y ago

Unless they make it open source, what benefits would it have compared to the tiny OpenAi models?

WatchDog 3y ago
The fact that it's multimodal is very interesting, they might not make it open source, but if they intend run it on people's devices, even if they intend to implement DRM, someone will figure out how to extract the weights and get it running outside.

pablobaz 3y ago

> "Using the power of Google Tensor G3, Video Boost on Pixel 8 Pro uploads your videos to the cloud where our computational photography models adjust color, lighting, stabilization and graininess."* I wonder why the power of Tensor G3 is needed to upload your video to the cloud... *https://blog.google/products/pixel/pixel-feature-drop-december-2023/ https://blog.google/products/pixel/pixel-feature-drop-decemb...

solardev 3y ago
It runs an on-device LLM to generate a HTTP POST every time. It took four interns half a week to reduce the hallucinations, but a PM got a promotion after that.
gundmc 3y ago
Using the power of a Bugatti Veyron, opens and closes the garage door.