8 ms·
Gemini 3.0 Pro – early tests
- maerch 1y agoI still have a bad taste in my mouth after all those GPT-5 hype articles that claimed the model was just one step away from AGI.
- gardnr 1y agoTBF, they all believed that scaling reinforcement learning would achieve the next level. They had planned to "war-dial" reasoning "solutions" to generate synthetic datasets which achieved "success" on complex reasoning tasks. This only really produced incremental improvements at the cost of test-time compute. Now Grok is publicly boasting PhD level reasoning while Surge AI and Scale AI are focusing on high quality datasets curated by actual PhD humans. Surge AI is boasting $1B in revenue, and I am wondering how much of that was paid in X.ai stock: https://podcasts.apple.com/us/podcast/the-startup-powering-the-data-behind-agi/id1504567418 https://podcasts.apple.com/us/podcast/the-startup-powering-t... In my opinion the major advancements of 2025 have been more efficient models. They have made smaller models much, much better (including MoE models) but have failed to meaningfully push the SoTA on huge models; at least when looking at the USA companies.
- svachalek 1y agoSame, qwen3 omni blows my mind for what a 30b-A3b model can do. I had a video chat with it and it correctly identified plant species I showed it.
- ACCount37 1y agoRaw model size is still pegged by the hardware. You can try to build a monster the size of GPT-4.5, but even if you could actually make the training stable and efficient at this scale, you still would suffer trying to serve it to the users. Next generation of AI hardware should put them in reach, and I expect that model scale would grow in lockstep with new hardware becoming available.
- adastra22 1y agoWithout defining “AGI” that’s always true, and trivially so.
- simonw 1y agoI've seen a bunch of tweets like this recently, as far as I can tell they're all from people using https://aistudio.google.com/ https://aistudio.google.com/ who got served an A/B test. A few more in this genre: https://x.com/cannn064/status/1973818263168852146 https://x.com/cannn064/status/1973818263168852146 - "Make a SVG of a PlayStation 4 controller" https://x.com/cannn064/status/1973415142302830878 https://x.com/cannn064/status/1973415142302830878 "Create a single, self-contained HTML5 file that mimics a macOS Sonoma-style desktop: translucent menu bar with live clock, magnifying dock, draggable/resizable windows, and a dynamic wallpaper. No external assets; use inline SVG for icons." https://x.com/synthwavedd/status/1973405539708056022 https://x.com/synthwavedd/status/1973405539708056022 "Write full HTML, CSS and Javascript for a very realistic page on Apple's website for the new iPhone 18" I've not seen it myself so I'm not sure how confident they are that it's Gemini 3.0.
- ceejayoz 1y ago> a very realistic page on Apple's website… Is this supposed to be a good example? It looks like something I'd put together, and you don't want me doing design work.
- ajcp 1y agoAt this point until I see one run through the Pelican Benchmark I can't really take a new model seriously.
- diggan 1y agoUnfortunately, as every public benchmark, once it ends up in the training sets and/or the developers aware of it, it stops being effective, and I think we've started to reach that point. The only thing I've found to give me some sort of quantitative idea of how good a new model is, is my own private benchmarks. It doesn't cover everything I want to use LLMs for, and only has 20-30 tests per "category", but at least I'm 99% sure it isn't in the training datasets.
- simonw 1y agoI have a few "SVG of an X riding a Y" tests that I don't publish online which I run occasionally to see if a model is suspiciously better at drawing a pelican riding a bicycle than some other creature on some other form of transport. I would be so entertained if I found out an AI lab had wasted their time cheating on my dumb benchmark!
- esafak 1y agoWe can't see the code and the challenge is pedestrian. Nothing to see here.
- Oras 1y agoThese tests mean nothing; I yet to see a model that is better than Sonnet 4 for coding. I tried many, all of them are sub-par, even with a small code base.
- nnevatie 1y agoWell, Codex with GPT5 High wins Claude Sonnet 4.5 - this is anecdotal, but I've used both extensively.
- solarkraft 1y agoAt what speed? At some point you’ll have to compare to Opus.
- adastra22 1y agoAnd Sonnet 4.5 is better than Opus.
- Bolwin 1y agoWell yeah no surprise. You should try glm 4.6
- Oras 1y agoI tried it, and it was shockingly bad compared to their benchmarks and to Claude Sonnet 4. I tried it with Claude Code CLI, it didn't follow instructions correctly (I had a Claude.md file with clear instructions), stopped after a few implementations (less than 3 minutes), and produced code that does not work. For the benefit of the doubt, I changed instructions to be NextJS platform as I thought it's a known framework and it might do better, but still, same quality issues.
- adastra22 1y agoWell, Sonnet 4.5 is better.
- strongpigeon 1y agoGoogle's biggest problem in my opinion (and I'm saying that as an ex-googler) is that Google doesn't have a product culture. Google had the tech for something like ChatGPT for a long time, but couldn't come up with that product. Instead it had to rely on another company showing it the way and then copy them and try to out-engineer them... I still think ultimately (and somewhat sadly) Google will win the AI race due to its engineering talent and the sheer amount of data it has (and Android integration potential).
- sho_hn 1y agoTo be fair, according to OpenAI they started ChatGPT as a demo/experiment and were taken by surprise when it went viral. It may well be that they also didn't have a product culture as an organization, but were willing to experiment or let small teams do so. It's still a lesson, but maybe a different one. With organizational scale it becomes harder and harder to launch experiments under the brand. Red tape increases, outside scrutiny increases. Retaining the ability to do that is difficult. Google does experiment a fair bit (including in AI, e.g. NotebookLLM and its podcast feature are I think a standout example of trying to see what sticks) but they also tend to try to hide their experiments in developer portals nowadays, which makes it difficult to get a signal from a general consumer audience.
- strongpigeon 1y agoGoogle is definitely good at experimenting (and yeah NotebookLLM is really cool), which is a product of the bottom-up culture. The lack of a consistent story with regard to AI products however is a testament to the lack of product vision from the top.
- ajcp 1y agoNotebookLM came out of Google Labs though, and in collaboration with outside stakeholders. I'm not sure I would call it a success of "bottom-up" culture, but a well realized idea from a dedicated incubator. That doesn't necessarily mean the rest of the company is so empowered or product oriented.
- vunderba 1y agoOutside of the aesthetic, the very first example on that twitter post is "balls bouncing around a constrained rotating rigid physics environment" which has been trivially one-shottable since Claude Code was first announced. It was one of the first things I tried when Claude Code went GA: https://gondolaprime.pw/hex-balls https://gondolaprime.pw/hex-balls
- Synaesthesia 1y agoThey have differing degrees of fidelity to the simulation, this one looks pretty good and it's got parameters, but yes the LLM's are really advanced now in what they can do. I was actually blown away during the Gemini 2.5 announcement with some of the demos people came up with.
- ACCount37 1y agoI hope this is the one that unfucks the multi-turn instruction following. One of the biggest issues holding Gemini back, IMO, compared to the competitors. Many LLMs are still plagued by "it's easier to reset the conversation than to unfuck the conversation", but Gemini 2.5 is among the worst.
- solarkraft 1y agoGemini‘s loops are a real problem. Within a few minutes of using it in the CLI it happened to me me („I can verify that I fulfilled the user’s request, I can verify that I fulfilled the user’s request …“). It’s telling that the CLI has a detection for this. The other day I asked 2.5 Pro for suggestions. It would provide one, which I rejected with some reasoning. It would provide another, which I also rejected. Asked for more it would then loop between the two, repeating the previous suggestions verbatim. It went on for 3-4 times, even after being told to reflect on it and it being able to recite the rejection reasons.
- deleted 1y ago[deleted]
- renewiltord 1y agoEvery three months there's some mind blowing hype around a Google product, lots of people talk about it, and then when I use it it's not nearly as good.
- robots0only 1y agoIn all of these posts there is someone claiming Claude is the best, then somebody else claiming they have tried a bunch of times and for them Gemini is the best while others find GPT-5 is supreme. Obviously, all of these are subjective narrow experiences. My conclusion is that all frontier models are both good and bad with no clear winner and making good evals is really hard.
- binary132 1y agoThe fact that there is so much astroturf out there also makes it difficult to evaluate these claims
- Robdel12 1y agoYeah, my take is it’s sort of up to the person using the LLM and maybe how they match to that LLM. That’s my hunch as to why we hear wildly different takes on these LLMs working for people. Gemini can be the most productive model for some while others find it entirely unworkable.
- jiggawatts 1y agoNot just personalities and preferences, but the purpose for which the AI is being used also affects the results. I primarily use AIs for complex troubleshooting along the lines of: "Here's a megabyte of logs, an IaC template, and a gibberish error code. What's the reason?" Right now, only Gemini Pro 2.5 has any chance of providing a useful output given those inputs, because its long-context attention is better than any other model's.
- SkyPuncher 1y agoI'll be that person: * Gemini has the highest ceiling out of all of the models, but has consistently struggled with token-level accuracy. In other words, it's conceptual thinking it well beyond other models, but it sometimes makes stupid errors when talking. This makes it hard to reliably use for tool calling or structured output. Gemini is also very hard to steer, so when it's wrong, it's really hard to correct. * Claude is extremely consistent and reliable. It's very, very good at the details - but will start to forget things if things get too complex. The good news is Claude is very steerable and will remember those details if you remind it. * GPT-5 seems to be completely random for me. It's so inconsistent that it's extremely hard to use. I tend to use Claude because I'm the most familiar with it and I'm confident that I can get good results out of it.
- nharada 1y agoGemini has always been the leader in multimodal work like images and video, I expect this won't be any different but am interested to see how it is
- whywhywhywhy 1y agoThese influencer tests are so pointless and don't represent the reality of model use at all when things are constantly being downgraded when people actually use the thing. Not to mention every team will have the bouncing balls in the polygon in their dataset now.
- geraldalewis 1y agoThis seems like a parody, but I think it's not.
- baxuz 1y agoIs there a source that isn't Twitter?
- alberth 1y agoAm I the only one who struggles to even find where to use Googles AI offerings. It took me way too long to figure out how to even access & use Veo 3. It’s like Google doesn’t know how to package a product.
- seandoe 1y agogemini.google.com
- Incipient 1y agoAll these AI reviews seem to be following the axiom(?) "proof of the pudding is in the eating" but frankly I don't think that applies to code. I can't get even gpt5 to create a new feature without generating completely awful code - making up facts where it can't find how it fits into the rest of the code - and functionality spawning error ridden unmaintainable mess. I've spent this whole week debugging AI trash. And it's not fun.
- daft_pink 1y agoDid they fix the fact that they train on your data on personal plans that you pay for unless you disable chat history? They are literally the worst major provider in terms of privacy for consumer paid service.
- theknarf 1y agoThe current problem with Gemeni 2.5 Pro is not that its not intelligent or can't oneshot problem, the problem is that its _terrible_ at tool calling and waste most of its context on trying to correct itself from mistaken tool calling. If they can solve that with 3.0 then they may have a useful model for agentic coding, if not its not keeping up with Anthropic and OpenAI.
- XCSme 1y agoI wanted to use Gemini 2.5 Pro, but couldn't, because their structured_data response is broken (doesn't support all JSON properties, or often simply returns garbage).