5 ms·

PaliGemma 2: Powerful Vision-Language Models, Simple Fine-Tuning

218 points by meetpateltech 2y ago

minimaxir 2y ago
Hugging Face's blog post on the release is more technical: https://huggingface.co/blog/paligemma2 https://huggingface.co/blog/paligemma2
- xnx 2y ago
  Even more technical detail here: https://arxiv.org/html/2412.03555v1 https://arxiv.org/html/2412.03555v1
dmvdoug 2y ago
Saw name, was expecting something to do with wreaking AI upon the Pali Canon (https://en.m.wikipedia.org/wiki/Pali_Canon https://en.m.wikipedia.org/wiki/Pali_Canon).
turnsout 2y ago
Does anyone know how this stacks up against other multimodal vision models?
- lofaszvanitt 2y ago
  Why does that matter? Everyone has their own uses for a vlm. Compare them for the given task at hand.
- mountainriver 2y ago
  They do an exceptionally poor job at evaluating it against competitors.
  - swyx 2y ago
    how about leaderboards that can pop them in?
exe34 2y ago
does anyone know if they can output bonding box coordinates? like "where is the ball" -> [50, 75, 150, 175]. so far cogvlm is the only one I've seen that works but it's a bit of a pain to run.
- _ea1k 2y ago
  Yes, just use "detect ball" as the prompt. It will give you y1,x1,y2,x2 coordinates on a scale of 0-1024. It is really good at this. Unfortunately, without fine tunes you can't have it just detect everything and return all detected objects, afaict.
  - exe34 2y ago
    The demo is impressive! https://huggingface.co/spaces/big-vision/paligemma https://huggingface.co/spaces/big-vision/paligemma
- xnx 2y ago
  Yes: "The initial four location tokens represent the coordinate of the bounding box, ranging from 0 to 1023. These coordinates are independent of the aspect ratio, as the image is assumed to be resized to 1024 x 1024." https://developers.googleblog.com/en/gemma-explained-paligemma-architecture/ https://developers.googleblog.com/en/gemma-explained-paligem...
  - exe34 2y ago
    Thank you! Will have a play with that.
- __jl__ 2y ago
  Gemini is surprisingly good at this. Look at example 5 here: https://developers.googleblog.com/en/7-examples-of-geminis-multimodal-capabilities-in-action/ https://developers.googleblog.com/en/7-examples-of-geminis-m... They also have a colab notebook with more examples linked in the article.
  - 2y ago
sigmar 2y ago
It is probably hard to come up with good benchmarks for VLMs like this, but I feel like the "Non entailment sentences" benchmark seems ill-suited. The examples for sentences that were non entailment included[1]: "There is a pile of horse manure in front of the horse." Which is true if you mean "in front of the [photosubject] from the perspective of the camera," but I think they marked it as non entailment because the pile is not in front of the horse's face(?) [1] page 20 https://arxiv.org/pdf/2412.03555 https://arxiv.org/pdf/2412.03555
timmg 2y ago
I recently wanted to try to get an LLM to help me organize my photos. (I'm someone who takes a lot of photos when I travel and then back them up to a hard drive -- assuming someday I'll organize them :) I created a prompt to try to get an LLM to do high-level organization: > Help categorize this photo for organization. Please output in JSON. > First, add a field called "type" that is one of: Wildlife, Architecture, Landscape, People or other. Pick the one that most appropriately reflects the subject of the photo. Use Other if you don't feel confident in your answer. > Next, if it is Wildlife, please add another field called "animal" that gives a general type of animal that is the focus of the photo. Use large, common types like Elephant, Bird, Lion, Fish, Antelope, etc. Do not add this field if your confidence is low. > If the type of animal is Bird, add a field called "bird" that gives the common type of bird, if you can clearly determine it. For example: Eagle, Hummingbird, Vulture, Crow, etc. > If it is an Architecture photo, and you can determine with good confidence what specific building (or city) it is a photo of, please add a field called "place" with that name. (Name only, please -- no description). I've tried with llama-vision using Ollama and it worked reasonably well for the top-level categories. A little less-well for identifying specific birds or places. And it didn't always generate proper JSON (and sometimes added new fields to JSON.) I also tried with Claude's API -- and it seemed to work perfectly (for a small sample size). It will be interesting to try with PaliGemma and see what I get. I have like 50k photos, so I don't want to pay $$$ for the Claude API to categorize them all. It will be cool someday (soon?) for an open-source DAM to have something like one of these models available to call locally.
- nulld3v 2y ago
  Are you aware of Immich? I believe it does mostly everything you were trying to do: https://immich.app/docs/features/smart-search https://immich.app/docs/features/smart-search It's open source, and is fairly polished too. I think the main part that's missing is how the classifications are not exposed to you in the UI. But I haven't used this feature much and my instance isn't running right now (SSD failed, still trying to get it replaced) so not able to check.
- visarga 2y ago
  use the json mode in ollama
pilooch 2y ago
Paligemma proves easy to train and useful in fine-tuning. It's main drawback was not being able to handle multiple images without being partly retrained. This new version dies not seem to support multiple images as input at once. Qwen2vl does. This is useful for vision rag typically.

218 points by meetpateltech 2y ago

Hugging Face's blog post on the release is more technical: https://huggingface.co/blog/paligemma2 https://huggingface.co/blog/paligemma2

xnx 2y ago
Even more technical detail here: https://arxiv.org/html/2412.03555v1 https://arxiv.org/html/2412.03555v1

dmvdoug 2y ago

Saw name, was expecting something to do with wreaking AI upon the Pali Canon (https://en.m.wikipedia.org/wiki/Pali_Canon https://en.m.wikipedia.org/wiki/Pali_Canon).

turnsout 2y ago

Does anyone know how this stacks up against other multimodal vision models?

lofaszvanitt 2y ago
Why does that matter? Everyone has their own uses for a vlm. Compare them for the given task at hand.
mountainriver 2y ago
They do an exceptionally poor job at evaluating it against competitors.
- swyx 2y ago
  how about leaderboards that can pop them in?

exe34 2y ago

does anyone know if they can output bonding box coordinates? like "where is the ball" -> [50, 75, 150, 175]. so far cogvlm is the only one I've seen that works but it's a bit of a pain to run.

_ea1k 2y ago
Yes, just use "detect ball" as the prompt. It will give you y1,x1,y2,x2 coordinates on a scale of 0-1024. It is really good at this. Unfortunately, without fine tunes you can't have it just detect everything and return all detected objects, afaict.
- exe34 2y ago
  The demo is impressive! https://huggingface.co/spaces/big-vision/paligemma https://huggingface.co/spaces/big-vision/paligemma
xnx 2y ago
Yes: "The initial four location tokens represent the coordinate of the bounding box, ranging from 0 to 1023. These coordinates are independent of the aspect ratio, as the image is assumed to be resized to 1024 x 1024." https://developers.googleblog.com/en/gemma-explained-paligemma-architecture/ https://developers.googleblog.com/en/gemma-explained-paligem...
- exe34 2y ago
  Thank you! Will have a play with that.
__jl__ 2y ago
Gemini is surprisingly good at this. Look at example 5 here: https://developers.googleblog.com/en/7-examples-of-geminis-multimodal-capabilities-in-action/ https://developers.googleblog.com/en/7-examples-of-geminis-m... They also have a colab notebook with more examples linked in the article.
- 2y ago

sigmar 2y ago

It is probably hard to come up with good benchmarks for VLMs like this, but I feel like the "Non entailment sentences" benchmark seems ill-suited. The examples for sentences that were non entailment included[1]: "There is a pile of horse manure in front of the horse." Which is true if you mean "in front of the [photosubject] from the perspective of the camera," but I think they marked it as non entailment because the pile is not in front of the horse's face(?) [1] page 20 https://arxiv.org/pdf/2412.03555 https://arxiv.org/pdf/2412.03555

timmg 2y ago

I recently wanted to try to get an LLM to help me organize my photos. (I'm someone who takes a lot of photos when I travel and then back them up to a hard drive -- assuming someday I'll organize them :) I created a prompt to try to get an LLM to do high-level organization: > Help categorize this photo for organization. Please output in JSON. > First, add a field called "type" that is one of: Wildlife, Architecture, Landscape, People or other. Pick the one that most appropriately reflects the subject of the photo. Use Other if you don't feel confident in your answer. > Next, if it is Wildlife, please add another field called "animal" that gives a general type of animal that is the focus of the photo. Use large, common types like Elephant, Bird, Lion, Fish, Antelope, etc. Do not add this field if your confidence is low. > If the type of animal is Bird, add a field called "bird" that gives the common type of bird, if you can clearly determine it. For example: Eagle, Hummingbird, Vulture, Crow, etc. > If it is an Architecture photo, and you can determine with good confidence what specific building (or city) it is a photo of, please add a field called "place" with that name. (Name only, please -- no description). I've tried with llama-vision using Ollama and it worked reasonably well for the top-level categories. A little less-well for identifying specific birds or places. And it didn't always generate proper JSON (and sometimes added new fields to JSON.) I also tried with Claude's API -- and it seemed to work perfectly (for a small sample size). It will be interesting to try with PaliGemma and see what I get. I have like 50k photos, so I don't want to pay $$$ for the Claude API to categorize them all. It will be cool someday (soon?) for an open-source DAM to have something like one of these models available to call locally.

nulld3v 2y ago
Are you aware of Immich? I believe it does mostly everything you were trying to do: https://immich.app/docs/features/smart-search https://immich.app/docs/features/smart-search It's open source, and is fairly polished too. I think the main part that's missing is how the classifications are not exposed to you in the UI. But I haven't used this feature much and my instance isn't running right now (SSD failed, still trying to get it replaced) so not able to check.
visarga 2y ago
use the json mode in ollama

pilooch 2y ago

Paligemma proves easy to train and useful in fine-tuning. It's main drawback was not being able to handle multiple images without being partly retrained. This new version dies not seem to support multiple images as input at once. Qwen2vl does. This is useful for vision rag typically.