4 ms·

One hidden note from Gemini 2.5 Flash when diving deep into the documentation: for image inputs, not only can the model be instructed to generated 2D bounding b

One hidden note from Gemini 2.5 Flash when diving deep into the documentation: for image inputs, not only can the model be instructed to generated 2D bounding boxes of relevant subjects, but it can also create segmentation masks! https://ai.google.dev/gemini-api/docs/image-understanding#segmentation https://ai.google.dev/gemini-api/docs/image-understanding#se... At this price point with the Flash model, creating segmentation masks is pretty nifty. The segmentation masks are a bit of a galaxy brain implementation by generating a b64 string representing the mask: https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Spatial_understanding.ipynb#scrollTo=WQJTJ8wdGOKx https://colab.research.google.com/github/google-gemini/cookb... I am trying to test it in AI Studio but it sometimes errors out, likely because it tries to decode the b64 lol.

behnamoh 2y ago
Wait, did they just kill YOLO, at least for time-insensitive tasks?
- minimaxir 2y ago
  YOLO is probably still cheaper if bounding boxes are your main goal. Good segmentation models that work for arbitrary labels, however, are much more expensive to set up and run, so this type of approach could be an interesting alternative depending on performance.
- daemonologist 2y ago
  No, the speed of YOLO/DETR inference makes it cheap as well - probably at least five or six orders of magnitude cheaper. Edit: After some experimentation, Gemini also seems to not perform nearly as well as a purpose-tuned detection model. It'll be interesting to test this capability and see how it evolves though. At some point you might be able use it as a "teacher" to generate training data for new tasks.
- vunderba 2y ago
  Well no. You can run/host YOLO which means not having to submit potentially sensitive information to a company that generates a large amount of revenue from targeted advertising.
daemonologist 2y ago
Interestingly if you run this in Gemini (instead of AI Studio) you get: I am sorry, but I was unable to generate the segmentation masks for _ in the image due to an internal error with the tool required for this task. (Not sure if that's a real or hallucinated error.)
ipsum2 2y ago
The performance is basically so bad it's unusable though, segmentation models and object detection models are still the best, for now.
msp26 2y ago
I've had mixed results with the bounding boxes even on 2.5 pro. On complex images where a lot of boxes need to be drawn they're in the general region but miss the exact location of objects.
simonw 2y ago
This is SO cool. I built an interactive tool for trying this out (bring your own Gemini API key) here: https://tools.simonwillison.net/gemini-mask https://tools.simonwillison.net/gemini-mask More details plus a screenshot of the tool working here: https://simonwillison.net/2025/Apr/18/gemini-image-segmentation/ https://simonwillison.net/2025/Apr/18/gemini-image-segmentat... I vibe coded it using Claude and O3.
xnx 2y ago
There is a starter app in AI Studio that demos this: https://aistudio.google.com/apps/bundled/spatial-understanding https://aistudio.google.com/apps/bundled/spatial-understandi...

behnamoh 2y ago

Wait, did they just kill YOLO, at least for time-insensitive tasks?

minimaxir 2y ago
YOLO is probably still cheaper if bounding boxes are your main goal. Good segmentation models that work for arbitrary labels, however, are much more expensive to set up and run, so this type of approach could be an interesting alternative depending on performance.
daemonologist 2y ago
No, the speed of YOLO/DETR inference makes it cheap as well - probably at least five or six orders of magnitude cheaper. Edit: After some experimentation, Gemini also seems to not perform nearly as well as a purpose-tuned detection model. It'll be interesting to test this capability and see how it evolves though. At some point you might be able use it as a "teacher" to generate training data for new tasks.
vunderba 2y ago
Well no. You can run/host YOLO which means not having to submit potentially sensitive information to a company that generates a large amount of revenue from targeted advertising.

daemonologist 2y ago

Interestingly if you run this in Gemini (instead of AI Studio) you get: I am sorry, but I was unable to generate the segmentation masks for _ in the image due to an internal error with the tool required for this task. (Not sure if that's a real or hallucinated error.)

ipsum2 2y ago

The performance is basically so bad it's unusable though, segmentation models and object detection models are still the best, for now.

msp26 2y ago

I've had mixed results with the bounding boxes even on 2.5 pro. On complex images where a lot of boxes need to be drawn they're in the general region but miss the exact location of objects.

simonw 2y ago

This is SO cool. I built an interactive tool for trying this out (bring your own Gemini API key) here: https://tools.simonwillison.net/gemini-mask https://tools.simonwillison.net/gemini-mask More details plus a screenshot of the tool working here: https://simonwillison.net/2025/Apr/18/gemini-image-segmentation/ https://simonwillison.net/2025/Apr/18/gemini-image-segmentat... I vibe coded it using Claude and O3.

xnx 2y ago

There is a starter app in AI Studio that demos this: https://aistudio.google.com/apps/bundled/spatial-understanding https://aistudio.google.com/apps/bundled/spatial-understandi...