14 ms·
I don't understand the dog segmentation example, where the prompt was "segment dog". The article shows a screenshot with a red overlay on the dog - how was tha
by simonw 3y ago
I don't understand the dog segmentation example, where the prompt was "segment dog".
The article shows a screenshot with a red overlay on the dog - how was that data returned by the model, did it return a co-ordinate polygon of some sort?
UPDATE: Figured it out using this tool, the mask looks like this: https://huggingface.co/spaces/google/paligemma https://huggingface.co/spaces/google/paligemma
<loc0099><loc0092><loc0874><loc0926><seg014><seg009><seg123><seg126><seg004><seg074><seg092><seg112><seg000><seg021><seg099><seg015><seg096><seg043><seg012><seg019>
- sigmoid10 3y agoYou could have also just looked at the documentation linked in the original post...
- simonw 3y agoHave you seen any documents that help explain how to interpret those tokens?
- zerojames 3y ago(I work at Roboflow) We're actively working on this! Our ML team has noted that the segmentation masks in particular from PaLiGemma are a bit tedious and unintuitive to decode. We should be pushing out (more!) open source software that uses this model in the coming days. Look forward to an easy way to fine-tune PaLiGemma and broader support for its task types in `inference`, the package used in the blog post.
- llama_person 3y agohttps://huggingface.co/spaces/google/paligemma/blob/main/paligemma_parse.py#L132 https://huggingface.co/spaces/google/paligemma/blob/main/pal... the blog post details it but essentially to convert from PaliGemma tokens to bbox: y0 / 1024 * h x0 / 1024 * w y1 / 1024 * h x1 / 1024 * w have not played with segmentation yet.
- sigmoid10 3y agoSure. Click on the github link in the post and look at the readme. It's all there. There is even a linked paper.