4 ms·

I just ran my image recognition benchmark on it ("is this XXX public landmark"?) and it misses a lot that bytedance seed 2.1 turbo gets right; for example: Aske

I just ran my image recognition benchmark on it ("is this XXX public landmark"?) and it misses a lot that bytedance seed 2.1 turbo gets right; for example: Asked "Is this Salisbury Cathedral" and supplied a picture of Wells Cathedral, it answers "Yes, the west facade of Salisbury Cathedral". Bytedance seed 2.1 turbo correctly says no. Similar results for a picture of Manhattan Bridge sent as Brooklyn Bridge, Chartres Cathedral sent as Notre Dame, etc. I have a benchmark of 12 such images and seed gets 11/12 and deepseek only gets 6/12.

throwa356262 1mo ago
This is a fairly small model for coding and agentic work. Training it on images like yours would just make it worse in other areas.
- jerkstate 1mo ago
  > The deepseek-v4-flash-vision-exp model accepts images alongside text, so you can ask the model to describe pictures it doesn't specify what type of images it can and can't describe, I'm pointing out what type it isn't good at compared to other models.
wolfgangK 1mo ago
I have zero interest in world knowledge for my LLMs but this got me wondering : are there RAGs for that kind of data ? How could a LLM like DeepSeek-v4-flash-vision-exp accurately answer you question with an indexed database of labeled landmark pictures (or even 3D models ?).
addandsubtract 1mo ago
Do you publish your benchmark somewhere? What's the best vision (captioning) AI you've come across?
- jerkstate 1mo ago
  My benchmark is tiny compared to WorldVQA or FG-BMK, which are available, so I'd point you in that direction if you're interested in a VLM benchmark. My use-case isn't exactly captioning as in "what is in this image?" -> caption, I am using the VLM to validate captions, as in "is this an image of [supposed subject]?" - my ranking of models I've benchmarked is gemini-3.7-flash > seed-2.1-turbo > gpt-5.6-luna > qwen-3.7-plus > qwen-3.7-flash. Gemini is almost perfect on my test dataset, only failing on some esoteric pop-culture minor celebrities and being over-specific in some cases (i.e. Q: is this [common name of fruit]? A: that's a [latin species name of fruit], not a [common name of fruit]; false). However, gemini-3.7-flash is only in my test list because openrouter has it on 75% introductory discount; otherwise it would be about 4x more expensive than seed.

throwa356262 1mo ago

This is a fairly small model for coding and agentic work. Training it on images like yours would just make it worse in other areas.

jerkstate 1mo ago
> The deepseek-v4-flash-vision-exp model accepts images alongside text, so you can ask the model to describe pictures it doesn't specify what type of images it can and can't describe, I'm pointing out what type it isn't good at compared to other models.

wolfgangK 1mo ago

I have zero interest in world knowledge for my LLMs but this got me wondering : are there RAGs for that kind of data ? How could a LLM like DeepSeek-v4-flash-vision-exp accurately answer you question with an indexed database of labeled landmark pictures (or even 3D models ?).

addandsubtract 1mo ago

Do you publish your benchmark somewhere? What's the best vision (captioning) AI you've come across?

jerkstate 1mo ago
My benchmark is tiny compared to WorldVQA or FG-BMK, which are available, so I'd point you in that direction if you're interested in a VLM benchmark. My use-case isn't exactly captioning as in "what is in this image?" -> caption, I am using the VLM to validate captions, as in "is this an image of [supposed subject]?" - my ranking of models I've benchmarked is gemini-3.7-flash > seed-2.1-turbo > gpt-5.6-luna > qwen-3.7-plus > qwen-3.7-flash. Gemini is almost perfect on my test dataset, only failing on some esoteric pop-culture minor celebrities and being over-specific in some cases (i.e. Q: is this [common name of fruit]? A: that's a [latin species name of fruit], not a [common name of fruit]; false). However, gemini-3.7-flash is only in my test list because openrouter has it on 75% introductory discount; otherwise it would be about 4x more expensive than seed.