5 ms·
does anyone know if they can output bonding box coordinates? like "where is the ball" -> [50, 75, 150, 175]. so far cogvlm is the only one I've seen that works
by exe34 2y ago
does anyone know if they can output bonding box coordinates? like "where is the ball" -> [50, 75, 150, 175].
so far cogvlm is the only one I've seen that works but it's a bit of a pain to run.
- _ea1k 2y agoYes, just use "detect ball" as the prompt. It will give you y1,x1,y2,x2 coordinates on a scale of 0-1024. It is really good at this. Unfortunately, without fine tunes you can't have it just detect everything and return all detected objects, afaict.
- exe34 2y agoThe demo is impressive! https://huggingface.co/spaces/big-vision/paligemma https://huggingface.co/spaces/big-vision/paligemma
- xnx 2y agoYes: "The initial four location tokens represent the coordinate of the bounding box, ranging from 0 to 1023. These coordinates are independent of the aspect ratio, as the image is assumed to be resized to 1024 x 1024." https://developers.googleblog.com/en/gemma-explained-paligemma-architecture/ https://developers.googleblog.com/en/gemma-explained-paligem...
- exe34 2y agoThank you! Will have a play with that.
- __jl__ 2y agoGemini is surprisingly good at this. Look at example 5 here: https://developers.googleblog.com/en/7-examples-of-geminis-multimodal-capabilities-in-action/ https://developers.googleblog.com/en/7-examples-of-geminis-m... They also have a colab notebook with more examples linked in the article.
- exe34 2y agoI meant weights-available ones, but thank you!