5 ms·

The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our b

The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated ! GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner. Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.

MrBuddyCasino 2mo ago
Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
- bicx 2mo ago
  Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap. Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
  - msp26 2mo ago
    [dead]
  - MrBuddyCasino 2mo ago
    Yeah Gemini 3.5 Flash Lite is really good. Which Chinese models can you recommend?
    - b345 2mo ago
      I've been using Qwen3.5-9B, hosted locally for PDF data extraction and it performs pretty well when extracting data from tables and infographics
    - SkalskiP 2mo ago
      Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now. Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/ https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/ https://blog.roboflow.com/qwen3-8-max/) If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20 https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20 https://x.com/skalskip92/status/2088411215441621469?s=20)
      - kanemcgrath 2mo ago
        Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
        SkalskiP 2mo ago
        Really? Gemma4-31B should be better than Qwen3.8-27B? I'm happy to test that.
  - ComputerGuru 2mo ago
    Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
    - MrBuddyCasino 2mo ago
      Yes you have to continuously tune the prompts ever so subtly. 3.5 is a lot better than than 3.1 tho. Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
      - ComputerGuru 2mo ago
        This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate. 3.0 flash (not lite) handled it like a champ though, fwiw.
- dannyw 2mo ago
  Gemini is honestly an excellent LLM with many capability strengths. For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable. Agentic coding is only one dimension.
  - fau 2mo ago
    Anecdotally, Gemini Flash is the leader for a particular use case of mine and has been since at least version 2.5. But now there's also Luna as the first real competitor thanks to the price cut. My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.
deleted 2mo ago
[deleted]
SkalskiP 2mo ago
Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price. Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20 https://x.com/skalskip92/status/2088032652301304121?s=20
- MostlyStable 2mo ago
  Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
  - bastawhiz 2mo ago
    3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
    - MostlyStable 2mo ago
      That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.
    - tziki 2mo ago
      I think it definitely makes sense since it's still the best Google has to offer in the "pro" tier.
      - bastawhiz 2mo ago
        3 and 3.1 Pro are both marked as deprecated by Google. Even if they're the best Google offers, it would be foolish to choose a model that's explicitly deprecated. It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.
Damjanski 2mo ago
thats so helpful - tysm
ImageXav 2mo ago
Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1]. [0] https://playground.roboflow.com/evals https://playground.roboflow.com/evals [1] https://playground.roboflow.com/evals/object-detection https://playground.roboflow.com/evals/object-detection

MrBuddyCasino 2mo ago

Yeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.

bicx 2mo ago
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap. Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
- msp26 2mo ago
  [dead]
- MrBuddyCasino 2mo ago
  Yeah Gemini 3.5 Flash Lite is really good. Which Chinese models can you recommend?
  - b345 2mo ago
    I've been using Qwen3.5-9B, hosted locally for PDF data extraction and it performs pretty well when extracting data from tables and infographics
  - SkalskiP 2mo ago
    Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now. Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/ https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/ https://blog.roboflow.com/qwen3-8-max/) If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20 https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20 https://x.com/skalskip92/status/2088411215441621469?s=20)
    - kanemcgrath 2mo ago
      Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
      - SkalskiP 2mo ago
        Really? Gemma4-31B should be better than Qwen3.8-27B? I'm happy to test that.
- ComputerGuru 2mo ago
  Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
  - MrBuddyCasino 2mo ago
    Yes you have to continuously tune the prompts ever so subtly. 3.5 is a lot better than than 3.1 tho. Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
    - ComputerGuru 2mo ago
      This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate. 3.0 flash (not lite) handled it like a champ though, fwiw.
dannyw 2mo ago
Gemini is honestly an excellent LLM with many capability strengths. For example, 3.7 Flash is #1 on MMLU Pro and AA’s agentic spreadsheets/docs benchmark, etc. Yes, beating Fable. Agentic coding is only one dimension.
- fau 2mo ago
  Anecdotally, Gemini Flash is the leader for a particular use case of mine and has been since at least version 2.5. But now there's also Luna as the first real competitor thanks to the price cut. My worry is that this is a zero-sum game and when Gemini catches up on coding, it'll regress to the mean in other areas.

deleted 2mo ago

[deleted]

SkalskiP 2mo ago

Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price. Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20 https://x.com/skalskip92/status/2088032652301304121?s=20

MostlyStable 2mo ago
Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
- bastawhiz 2mo ago
  3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
  - MostlyStable 2mo ago
    That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.
  - tziki 2mo ago
    I think it definitely makes sense since it's still the best Google has to offer in the "pro" tier.
    - bastawhiz 2mo ago
      3 and 3.1 Pro are both marked as deprecated by Google. Even if they're the best Google offers, it would be foolish to choose a model that's explicitly deprecated. It's not a technical problem, it's a commercial one. If Google can't ship a model to replace the one they deprecated, that tells you everything you need to know about choosing a Gemini model for whatever you're trying to do.

Damjanski 2mo ago

thats so helpful - tysm

ImageXav 2mo ago

Gemini tops their vision evals [0] by a mile, with 4/5 top spots going to variants of it. Qwen is the only other contender, likely due to how good it is for object detection, where it crushes the competition [1]. [0] https://playground.roboflow.com/evals https://playground.roboflow.com/evals [1] https://playground.roboflow.com/evals/object-detection https://playground.roboflow.com/evals/object-detection