7 ms·
GPT 5.6 Sol is the best "vision" model OpenAI ever released
- hn7jmxa7oc 2mo ago[dead]
- weli 2mo agoAnecdotal, opinion: Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.
- velcrovan 2mo agoAssessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.
- rib3ye 2mo agoanthropic frontend-design skill does a great job with it.
- rafram 2mo agoHave you actually read the frontend design skill? It’s placebo at best. Very short and barely focused on design: https://github.com/anthropics/skills/blob/main/skills/frontend-design/SKILL.md https://github.com/anthropics/skills/blob/main/skills/fronte...
- MallocVoidstar 2mo agoWhat an annoying time for GitHub to go down.
- KeplerBoy 2mo agoLike every time
- rib3ye 2mo agoHave you actually tried using it?
- rafram 2mo agoOf course. It’s OK, but it tends to generate very cliched “AI” UIs with little originality. Despite the skill spending a lot of time coaching the model into avoiding that!
- akoboldfrying 2mo ago> UIs with little originality Sounds like the kind of UI I like. (Take me back to Windows XP...)
- rafram 2mo agoI mean they all look like generic, annoying SaaS landing pages/overwrought dashboards, not that they’re simple and functional.
- DaiPlusPlus 2mo agoMy exposure to Claude-produced UIs is limited, but I have started to notice certain design trends they tend to have in-common, which might be becoming hallmarks of AI-produced UIs - the same way we've started noticing the clichés of low-effort LLM-generated text. FWIW, the summary-description[1] of "frontend-design"[2] gives me a few things to pick at: > create polished code Methinks only if you're using it with a very popular framework like React. What happens if you ask Claude to make the UI in WinForms or MFC? > high-impact animations That's bad UX 101 right there: animations in a UI exist as an affordance to the user, and never for its own sake (e.g. macOS's "genie" animation when you minimize a window to the dock exists so the user knows where they can restore the window from). The only people who actually want "high impact animations" in software are salespeople who want something for demo purposes. > generic system fonts, predictable purple gradients, and cookie-cutter components. This screams wanting to be different for the sake of standing-out, not because it results in a better software product; users benefit when their software fits-in with platform conventions: if you refuse to use a stock checkbox <input> or <select> drop-down and instead use your own entirely custom component solely for aesthetic reasons then you are producing worse software. There's nothing wrong with system-fonts, but your site will look ugly after your third-party font-host CDN shuts-down and turns into a walking CSRF factory. > thoughtful typography with unexpected font pairings The above fragment set my alarm-bells off. Yikes. > scroll-triggered interactions Not every web-page should be an Apple.com product brochure page. This is also a fantastic way to make your webpage horribly inaccessible. ------ The SKILL.md itself[3] grinds my gears too: > Approach this as the design lead at a small studio known for giving every client a visual identity that could not be mistaken for anyone else's. Claude has no way of knowing what designs are actually unique or not... > For web designs, the hero is a thesis. Open with the most characteristic thing in the subject's world, in whatever form makes sense for it: a headline, an image, an animation, a live demo, an interactive moment ...this is exactly what everyone else's web-pages look like! > For calibration: AI-generated design right now clusters around three looks: (1) a warm cream background (near #F4F1EA) with a high-contrast serif display and a terracotta accent; (2) a near-black background with a single bright acid-green or vermilion accent; (3) a broadsheet-style layout with hairline rules, zero border-radius, and dense newspaper-like columns ...I called this out weeks ago[4], lol. and I could go on. This is all quite painful to read. ------ [1] https://claude.com/plugins/frontend-design https://claude.com/plugins/frontend-design [2] https://github.com/anthropics/claude-plugins-official/tree/main/plugins/frontend-design https://github.com/anthropics/claude-plugins-official/tree/m... [3] https://github.com/anthropics/claude-plugins-official/blob/2a5cd1f39f0d8e5fbb68d77a13884f69c4b0c516/plugins/frontend-design/skills/frontend-design/SKILL.md https://github.com/anthropics/claude-plugins-official/blob/2... [4] https://news.ycombinator.com/item?id=49187385 https://news.ycombinator.com/item?id=49187385
- TeMPOraL 2mo agoThere's a lot of objective principles and decisions that go into subjective quality; if you don't know the field well, asking LLM for assessment is a good way to discover all that.
- keeganpoppen 2mo agoi'd say this is something that has gotten orders of magnitude better with recent releases than it used to be, fwiw
- velcrovan 2mo agoWhen asked to produce a thing, the output has a much better baseline of quality. But when you ask it to evaluate the quality of a thing, how do you evaluate the quality of its response, which is necessarily subjective and not quantifiable? How can a thing which has no experience of friction be said to subjectively evaluate quality? Either you are yourself already a better judge of the thing’s quality, in which case the response can be of no use to you, or you are a poor judge of the thing’s quality, in which case you will be blind to the flaws in the synthesized opinion handed back to you.
- DaiPlusPlus 2mo agoWhat is a "non-normative UI block"?
- SkalskiP 2mo agoHi! I’m the author of this blog. GPT-5.6 is much better at vision than previous GPT versions, but it’s still much weaker than Gemini 3.5 Flash or Gemini 3.7 Flash, which was released last week. One interesting approach is to use Gemini through a tool call.
- Tactical45 2mo agoThis response is not relevant to the this comment
- sscaryterry 2mo agoMy anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.
- howdareme 2mo agoHow can a vision model have taste?
- sscaryterry 2mo agoReplace taste with consistent if that helps you. Can it follow a design system...
- velcrovan 2mo agoSo, formulaic output…the opposite of taste
- sscaryterry 2mo agoNot really. Compliance with the letter of the law doesn't mean the intent is complied with.
- yreg 2mo agoAs a design system engineer I usually have to fight against the taste of the designers. (And I consider it natural.) But, if you have a proper well documented design system and you tell the LLM to use the DS and to avoid styling hacks they can generally do it. Even the dumber ones than Sol 5.6. Of course only if the design is achievable in the design system.
- sscaryterry 2mo agoThis is not my experience at all.
- sarreph 2mo ago
- Razengan 2mo agoFor the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background. After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription. An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.
- thatcat 2mo agodid you try segmenting it first?
- Razengan 2mo agoAt first I intended to create a tileset and asked it for several variations of what a hypothetical tilemap created from the planned tileset would look like. The previews it generated were amazing but wouldn't really be possible as a grid-based tilemap, with lots of clusters and overlaps of elements of varying sizes. So I just decided to use the preview as a static scrolling background, but it's been a pain to get it to add more content around the edges that still tiles with the existing image at the same scale.
- sscaryterry 2mo ago> it constantly keeps getting something wrong no matter what I tell it This 100%
- dev_hugepages 2mo agoI'm unsure why you're using an LLM to generate images. Don't we already have models (some made by the same company) that do this?
- kzrdude 2mo agoIn the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight. In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.
- defrim 2mo agoSeems to be due to the detection area being not fully accurate. Green vs red shows the difference between actual and detected
- kzrdude 2mo agoThere is an extra green square where no egg is present, so it's a false positive in the expected.
- SkalskiP 2mo agoHi! I’m the author of this blog and benchmark. You’re right. I’ll fix it in the ground-truth dataset. Thanks for pointing it out.
- kzrdude 2mo agoGreat, happy that it was helpful
- evrimoztamur 2mo agoPenny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.
- SkalskiP 2mo agoHi! I’m the author of this blog. I had the same intuition, but together with the OpenAI team we figured out that the issue was image resolution. GPT-5.6 doesn’t handle large images well.
- evrimoztamur 2mo agoOpenAI team sounds like they've misidentified the root cause for this particular case then.
- DustinBrett 2mo agoHaha, ya at least to some degree, those boxes are in the right position, but rotated.
- DustinBrett 2mo agoGood call out, I noticed the same rotation issue but pointing at EXIF data sounds about right.
- bob1029 2mo agoI've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).
- hathym 2mo ago[dead]
- logicallee 2mo agoI agree. It did very well on an extremely challenging task. I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass. In addition, the poster itself also happened to contain similar clothing. You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-recognition-and-generation-5b0d0329a46f https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco... While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.
- zyvop1 2mo ago[flagged]
- iamniels 2mo agoI understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right? In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.
- throwup238 2mo agoGenerating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).
- rhplus 2mo agoEspecially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).
- jacquesm 2mo agoI think that is because people perceive OpenCV as 'hard to use' and LLMs as easy to use.
- TeMPOraL 2mo agoOpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count. To use an LLM, you just prompt it with an image + text saying "count the pills in this image". To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it". (I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.) Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.
- 5555watch 2mo agoAll of your use cases are very advanced. I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful. I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well. I will add that all frontier models were fine with such tasks from the early 2024's.
- adroitboss 2mo agoI didn't expect Gemini 3.5 Flash to top basically every metric in this article.
- LollipopYakuza 2mo agoSame. I scrolled back up to see if I read the title correctly. It's important to note that it is the best... OpenAI released. Not the best overall.
- SweetSoftPillow 2mo agoIn my practice Gemini models are far better than anything on the market in terms of vision, also it's worth to mention that current Gemini flash is 3.7, so it got 2 updates since 3.5 which beat GPT-5.6 Sol in this comparison.
- WarmWash 2mo agoGemini has long been the vision champion, but there aren't many benchmarks and coding is where all the hype is. Demis had a pretty big interest in vision, more so than text, so I hope they don't lose that with all the recent shuffling.
- SkalskiP 2mo agoHi! I’m the author of this blog. I wrote it 4 weeks ago, and it’s already a bit outdated. Gemini 3.7 Flash came out last week, and considering the price, it’s easily the best vision model right now: https://x.com/skalskip92/status/2088032652301304121?s=20 https://x.com/skalskip92/status/2088032652301304121?s=20
- iamleppert 2mo agoWhere are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.
- ImageXav 2mo agoMe too. This is an interesting comparison but in my experience Qwen and Gemini have typically been the top contenders for image related tasks. For that reason it would be great to have the comparison here, as I'm not surprised by Gemini's dominance over the other models.
- SkalskiP 2mo agoHi! I’m the author of this blog. I regularly benchmark new VLM releases. You can check the results for Qwen3.8-Max and Qwen3.8-27B here: https://playground.roboflow.com/evals https://playground.roboflow.com/evals
- catigula 2mo agoStill not quite as good as gemini.
- trumbitta2 2mo ago"Best iPhone ever" vibes.
- comboy 2mo agoDoes any popular NVR make a good use of LLMs (especially local models) getting decent at vision?
- eks391 2mo agoI've been using Reolink for years and been very satisfied with it. The only quip is the default UI isn't very good. When changing that reaches the top of my priority list, I'll switch it since they don't force you into a walled garden. Plan is to run it through frigate into HomeAssistant and use a UI from them. I've never used frigate before though so it'll be a learning process if plug and play solutions aren't already available
- deleted 2mo ago[deleted]
- chasd00 2mo agoOne of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.
- mv4 2mo agoIronically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
- maxime_cb 2mo agoI'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :) https://ieeexplore.ieee.org/document/1674847 https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847
- mv4 2mo agoExactly my point. Template rotation is a trivial operation as well.
- lebek 2mo agoThe point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.
- glaslong 2mo agoSuppose now we need to test Sol vs OpenCV vs Sol implementing OpenCV
- geysersam 2mo agoI'm sure a typical frontier model would also be happy to write that opencv script for you, and it would do it well. That is certainly pretty far from what was possible 25 years ago.
- mv4 2mo ago
- prathje 2mo agoI would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from... I usually go to https://arena.ai/leaderboard/vision/pareto https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.
- HarHarVeryFunny 2mo agoThe summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated ! GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner. Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.
- MrBuddyCasino 2mo agoYeah I was thinking about giving Luna a go with my PDF data extraction, but I think I‘ll stay on Gemini. It does a very good job.
- bicx 2mo agoGemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap. Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
- msp26 2mo ago[dead]
- MrBuddyCasino 2mo agoYeah Gemini 3.5 Flash Lite is really good. Which Chinese models can you recommend?
- b345 2mo agoI've been using Qwen3.5-9B, hosted locally for PDF data extraction and it performs pretty well when extracting data from tables and infographics
- SkalskiP 2mo agoHi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now. Over the last two weeks, Qwen released two new models. Qwen3.8-Max is totally insane, but it’s only available through the Alibaba Cloud API. I wrote a similar blog covering Qwen3.8-Max: [https://blog.roboflow.com/qwen3-8-max/ https://blog.roboflow.com/qwen3-8-max/](https://blog.roboflow.com/qwen3-8-max/ https://blog.roboflow.com/qwen3-8-max/) If you’re looking for something you can run locally, Qwen3.8-27B might be a great option. On Friday, I did a quick comparison between Qwen3.8-Max and Qwen3.8-27B: [https://x.com/skalskip92/status/2088411215441621469?s=20 https://x.com/skalskip92/status/2088411215441621469?s=20](https://x.com/skalskip92/status/2088411215441621469?s=20 https://x.com/skalskip92/status/2088411215441621469?s=20)
- WarmWash 2mo agoIt's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself. I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.
- kherud 2mo agoSo far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.
- adrianh 2mo agoFor a bespoke model that transcribes sheet music images well, check out our system at Soundslice: https://www.soundslice.com/sheet-music-scanner/ https://www.soundslice.com/sheet-music-scanner/ It's not an LLM, it's a custom thing we built. Here's a comprehensive list of support for various notation glyphs: https://www.soundslice.com/help/en/creating/pdf-import/294/supported-notations/ https://www.soundslice.com/help/en/creating/pdf-import/294/s...
- bearjaws 2mo agoIt is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills). We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.
- repeekad 2mo agoHow are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks
- SkalskiP 2mo agoHi! I’m the author of this blog. I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector. I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20 https://x.com/skalskip92/status/2080334344061694429?s=20
- mhaberl 2mo agoDid you evaluate any that could be self-hosted (or at least ow models), if so which one is the best you seen?
- kooi 2mo agoAgreed, this like asking a chainsaw to carve a wooden spoon. Impressive it can, but definitely not the right tech to scale. LLM needs to setup an image classifier to use as a tool call.
- RugnirViking 2mo agoIt's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure
- fpgaminer 2mo agoGemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
- bastawhiz 2mo ago3 Flash never left "preview" status and is listed as deprecated. https://ai.google.dev/gemini-api/docs/deprecations https://ai.google.dev/gemini-api/docs/deprecations
- zuzululu 2mo agobut 3.7 flash is expensive for img inputs no ?
- fpgaminer 2mo agoAs usual for something so simple, Google's docs seem unclear: https://ai.google.dev/gemini-api/docs/pricing https://ai.google.dev/gemini-api/docs/pricing For 3, pricing for image tokens was the same as text tokens. Since they don't indicate a difference on 3.7, I would assume the same holds. And as far as I know the number of image tokens is the same for both (depending on the detail level you pick, but it's generally around 1k per image). So they're about the same, 3.7 is slightly more expensive. At least until the end of the year (when they raise 3.7's pricing). Anyway, my point was that 3.5 tended to have worse performance and significantly higher costs. 3 and 3.7 are both better and cheaper than 3.5.
- zuzululu 2mo agomystery to me is how the image tokens are calculated? 1MB is 1000 tokens ?
- criddell 2mo agoAre any of these vision benchmarks binocular in order to introduce depth perception? I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.
- deleted 2mo ago[deleted]
- CurbStomper 2mo ago[dead]
- ParanoidShroom 2mo agoI run the free service https://countrx.app/ https://countrx.app/ so i have some idea what goes into counting. The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models. Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set. Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations. But for being able to one shot with this accuracy... IMPRESSIVE
- IncreasePosts 2mo agoHow are you running it for free? Are you self funding or do you have sponsors?
- ParanoidShroom 2mo agoSelf funded. It's a custom trained efficient model on CPU so it's borderline free
- cdolan 2mo agoLuna is pretty strong as well. been using it for projects the last two weeks and its strong
- fintuner 2mo ago[flagged]
- jug 2mo agoI really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.
- deleted 2mo ago[deleted]
- schopra909 2mo agoFrom our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago. When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate. The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate). Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.
- Vacyyyy 2mo agoHave you checked versus more recent Gemini models like 3.5 or perhaps 3.7?
- schopra909 1mo agoYep checked 3.7
- faxmeyourcode 2mo agoIt's not clear to me from the article, are they asking sol to output bounding box coordinates with some kind of structured outputs? Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation. I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.
- ALLTaken 2mo agoI actually favor Qwen3.8 and run it locally + use the Token-Plan on AlibabaCloud, when I need faster results. Kind of favor it over GPT5.6 Sol. Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights. Here's some benchmarks: https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max https://qwen.ai/blog?id=qwen3.8#full-benchmark-table https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos) https://venturebeat.com/technology/qwen3-8-max-arrives-with-a-bold-claim-it-outperforms-gpt-5-6-sol-max-and-fable-5-on-agentic-computer-use https://venturebeat.com/technology/qwen3-8-max-arrives-with-... EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?
- ALLTaken 2mo agohuh, why am I being shadow banned? Does YC have similar problems like those at wikipedia/reddit? (wikipedia-editor-wars, or reddit-mod-wars)
- HDBaseT 2mo agoYou aren't being "shadow banned". I concur with your conclusion, Qwen 3.8 has exceptional Vision Capabilities. The other commenter mentioned "I thought Qwen 3.8 didn't have Vision", it does, just not on the open weights version, only via official API.
- barrenko 2mo agoI thought Qwen 3.8 max doesn't have vision?
- ALLTaken 2mo agoIt actually has vision + tool_use even the 27B param model. The community tries to produce even a MoE version of Qwen3.8 now, because that'd allow to run the full model with some experts being pruned like with Ornith 1.5 35B A3B. You can get this to run on 24GB Ram: https://huggingface.co/baa-ai/Qwen3.8-27B-RAM-24GB-MLX https://huggingface.co/baa-ai/Qwen3.8-27B-RAM-24GB-MLX I found the FULL Qwen3.8-2.4T-A95B-MLX-reap50-3bit, but it's 540GB. So, I can't run it on my tiny laptop, but hope that the community finds ways to bring the size down and memory requirements too =)))
- alessandrobinda 2mo ago[dead]
- fooker 2mo agoI'm a little bit disappointed that vision seems to fall before language at scale. It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.
- TZubiri 2mo agoWhich is to say, still not ready for any production workloads yet. As in, it cannot reliably count the amount of objects in an image. Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning
- chistev 2mo agohttps://news.ycombinator.com/item?id=46444508 https://news.ycombinator.com/item?id=46444508
- terhechte 2mo agoFuck ack. I'm working on a new benchmark that combines strong visual requirements with tool and coding requirements. I haven't even tested Sol yet, but between Sonnet, Terra & Luna I already see much better results from OpenAI's models. I'm not releasing anything yet as I still have issues in my harness that need to be fixed.
- ZeroDayDreamer 2mo ago[dead]
- lwarfield 2mo agoI currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated." Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.
- 1saadcodes 2mo ago5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too
- ComputerGuru 2mo agoWe have 3.7 Flash now, actually, and it costs just a hair over the old 3 Flash Preview while being better!
- wahid_seddiqi 2mo agoDo you think we’re getting closer to models that actually understand what they’re seeing, or are they just getting really good at recognizing patterns?
- Culonavirus 2mo agoAll I'm fine with for now is that I can almost exclusively communicate with Sol through collages and my scribblings (all kinds of web page / block screens with all kinds of arrows and text all over the place) This was not practically ppossible in 5.5 and a tragedy in 5.4. Not sure how much weight is codex uploading in higher res carrying here but it's great to work with.
- edude03 2mo ago"yes" but that's a philosophical question. I think they're getting better at "early fusion" IE, training the model that "apple" and these visual tokens are the the same concept, but LLMs are fundamentally a pattern matching machine so even with perfect fusion I personally wouldn't call it understanding.
- johnxianren 2mo ago[dead]
- slibhb 2mo agoOne of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.
- dllu 2mo agoVision is still embarrassingly bad. ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b67 https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b... The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3fs-public/images/christmas_present_match_up_460.jpg https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...
- TeMPOraL 2mo agoThe second answer is far more revealing than the first: OP: > do you think you did a good job there ChatGPT: > I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction. My observations: 1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat. 2) The part about "no Python or tools" is setting the model up for failure. I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like: 1. Identify start and end points 2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached. 3. Report answer It's literally what every human facing this task does. EDIT: My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes. https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d49 https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d... INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths). Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.
- deleted 2mo ago[deleted]
- apinstein 2mo agoIt’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.
- deleted 2mo ago[deleted]
- slybot 2mo agoAm I the only one who cannot read the date on the blister pack even fully zoom in my phone? If that is the full quality image given to the model, I think it's not surprising that the model confused with 03/2022.
- drak0n1c 2mo agoSeed Turbo 2.1 is incredibly detailed in describing every physical feature. I use that one for vision tool calls through Venice API.
- sam0x17 2mo ago> GPT 5.6 Sol is the best "vision" model OpenAI ever released I mean I should hope so, as it is also the latest one
- dangoodmanUT 2mo agoI hate these "The best X thing Y has ever released". Unlike when Apple says "it's the best iphone we've ever made", LLMs are more or less interchangeable. So "OpenAI's best model" means nothing if "Anthropic wipes the floor with them" or "[open weights model] is 10x cheaper for 1% less quality". As a reader, it feels like these titles are click bait.
- dzonga 2mo agothe last image - it's barely visible to human eyes
- deleted 2mo ago[deleted]
- theteapot 2mo agoDumb question: When your testing "ChatGPT 5.6 Sol" are you testing an actual LLM or some visual pre-processor stack that sits in front of it (along with a maybe a bunch of other such pre-processors) that is bundled into what's call "ChatGPT 5.6 Sol"? I.e. last I checked LLMs had a something like a 30-100K token alphabet to work with and it's hard to imagine how throwing pixels arrays at one directly would work.
- MoltenMan 2mo agoI'm ~95% certain that images are tokenized, just like regular text, and fed directly in; that's the 'multimodal' part of these models. Now how this tokenizing works I don't know, and there might be some level of preprocessing, but it's certainly not converting the image into text and feeding it in to a regular LLM.
- mherrmann 2mo agoA better headline would be "Gemini 3.5 Flash is the best vision model". It tops almost every single benchmark shown in the article.
- mavensum 2mo ago[flagged]
- shipsitself 2mo ago[flagged]
- dostick 2mo agoI wonder what about UI review, which model is the best?
- trevortaylorai 2mo agoThe prompt sensitivity is fascinating. A simple coordinate-format change shifting detection performance this much shows how much interface design still matters.
- locitra 2mo ago[flagged]
- cpnwaugha 2mo agoYeah, GPT 5.6 Sol is very good. Generally, Gemini models remain SOTA for VLM tasks with 3.7-flash at the top. --- That said, considering variables like cost (say, over 100k PDF pages) and accuracy requirements (e.g., construction documents with dense images & tables), Gemini and other SOTA VLMs are expensive and inaccurate, and therefore unsuitable. This is where niche, open-weight, and task-specific OCR/VL models come in. With a one-line change, you can switch between DeepSeek OCR 2, GLM-OCR, dots.mocr, Paddle OCR VL, PP-OCRv6, etc., and process 100K+ pages for under $60 on VLM Run Gateway This is why we built VLM Run Gateway, one OpenAI-compatible endpoint for open-weight OCR and VLM models. Try it out quickly via OpenAI SDK: ``` client = OpenAI( base_url="https://gateway.vlm.run/v1/openai https://gateway.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>", ) response = client.chat.completions.create( model="rednote-hilab/dots.mocr", messages=[{ "role": "user", "content": [{ "type": "document_url", "document_url": {"url": "https://.../invoice.pdf"}, }], }], extra_body={"document_dpi": 72}, ) ``` or via our CLI: ``` pip install vlmrun vlmrun gw models vlmrun config set --api-key 'vlmrun' # anon-user, rate-limited vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr --json-mode vlmrun gw chat <doc>.pdf -m deepseek-ai/deepseek-ocr-2 vlmrun gw chat <doc>.pdf -m rednote-hilab/dots.mocr vlmrun gw chat <doc>.pdf -m paddleocr/pp-ocrv6 ``` Docs: https://docs.vlm.run/gateway https://docs.vlm.run/gateway Catalog: https://docs.vlm.run/gateway/models https://docs.vlm.run/gateway/models MCP: https://docs.vlm.run/gateway/mcp-server https://docs.vlm.run/gateway/mcp-server Colab Quickstart: https://colab.research.google.com/drive/1RkuVIyuc5Po-UlcSlFyJCam5tjCm9IHM?usp=sharing https://colab.research.google.com/drive/1RkuVIyuc5Po-UlcSlFy... Read the full post here: https://huggingface.co/blog/vlm-run/intro-to-vlmrun-gateway https://huggingface.co/blog/vlm-run/intro-to-vlmrun-gateway