16 ms·
Gemini 2.5
- noisy_boy 2y agoAre Gemini and Bard same? I asked it a question and it said "... areas where I, as Bard, have...."
- guyzero 2y agoNormal Google rollout process: Bard is deprecated, Gemini is not ready yet.
- fourseventy 2y agoDoes it think the founding fathers were a diverse group of mixed races and genders like the last model did?
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jasonpeacock 2y agoIsn't every new AI model the "most <adjective>"? Nobody is going to say "Announcing Foobar 7.1 - not our best!"
- wigster 2y agogobble 2.0 - a bit of a turkey
- minimaxir 2y agoGPT-4.5's announcement was the equivalent of that. "It beats all the benchmarks...but you really really don't want to use it."
- forbiddenvoid 2y agoThey even priced it so people would avoid using it. GPT-4.5's entire function was to be the anchor of keeping OpenAI in the news, to keep up the perception of releasing quickly.
- glenstein 2y agoMy assumption was that the pricing was because it really was that expensive for whatever reason. I'm keeping fingers crossed that they're going to do some kind of 4.5 mini at some point that will be more affordable.
- thethimble 2y agoExcept for GPT 4.5 and Claude 3.7 :/
- smilekzs 2y agoYou're not wrong, but that just means the <adjective> is where the bulk of information resides. The trade-off matters. Maybe it's a model with good enough quality but really cheap to serve. Maybe it's a model that only plays poker really well but sucks at everything else because it bluffs too much. Etc. etc.
- Maxatar 2y agoSure but that adjective matters. Could be cheapest, "intelligent", fastest, etc... it's rarely all three of them.
- zamadatix 2y agoStable Diffusion definitely had a few of those.
- SirMaster 2y agoSame with new phones. The new phone is always the fastest cpu, gpu, and best camera ever!
- jharohit 2y agowhy not enable Canvas for this model on Gemini.google.com? Arguably the weakest link of Canvas is the terrible code that Gemini 2.0 Flash writes for Canvas to run..
- aoeusnth1 2y agoI'm guessing it should be enabled eventually. @logankilpatrick thoughts?
- cj 2y agoSlight tangent: Interesting that they use o3-mini as the comparison rather than o1. I've been using o1 almost exclusively for the past couple months and have been impressed to the point where I don't feel the need to "upgrade" for a better model. Are there benchmarks showing o3-mini performing better than o1?
- boldlybold 2y agoI find o3 at least faster to get to the response I care about, anecdotally.
- FloorEgg 2y agoI noticed this too, I have used both o1 and o3 mini extensively, and I have ran many tests on my own problems and o1 solves one of my hardest prompts quite reliably but o3 is very inconsistent. So from my anecdotal experience o1 is a superior model in terms of capability. The fact they would exclude it from their benchmarks seems biased/desperate and makes me trust them less. They probably thought it was clever to leave o1 out, something like "o3 is the newest model lets just compare against that", but I think for anyone paying attention that decision will backfire.
- jnd0 2y agoProbably because It is more similar to o3 in terms of size/parameters as well as price (although I would expect this to be at least half price)
- logicchains 2y agoIt's a reasonable comparison given it'll likely be priced similarly to o3-mini. I find o1 to be strictly better than o3-mini, but still use o3-mini for the majority of my agentic workflow because o1 is so much more expensive.
- PunchTornado 2y agoWhy would you compare against all the models from a competitor. You take their latest one that you can test. Openai or anthropoc don’t compare against the whole gemini family.
- 2y ago
- vineyardmike 2y agoI wonder what about this one gets the +0.5 to the name. IIRC the 2.0 model isn’t particularly old yet. Is it purely marketing, does it represent new model structure, iteratively more training data over the base 2.0, new serving infrastructure, etc? I’ve always found the use of the *.5 naming kinda silly when it became a thing. When OpenAI released 3.5, they said they already had 4 underway at the time, they were just tweaking 3 be better for ChatGPT. It felt like a scrappy startup name, and now it’s spread across the industry. Anthropic naming their models Sonnet 3, 3.5, 3.5 (new), 3.7 felt like the worst offender of this naming scheme. I’m a much bigger fan of semver (not skipping to .5 though), date based (“Gemini Pro 2025”), or number + meaningful letter (eg 4o - “Omni”) for model names.
- forbiddenvoid 2y agoI would consider this a case of "expectation management"-based versioning. This is a release designed to keep Gemini in the news cycle, but it isn't a significant enough improvement to justify calling it Gemini 3.0.
- jstummbillig 2y agoI think it's reasonable. The development process is just not really comparable to other software engineering: It's fairly clear that currently nobody really has a good grasp on what a model will be while they are being trained. But they do have expectations. So you do the training, and then you assign the increment to align the two.
- 8n4vidtmkvmk 2y agoI figured you don't update the major unless you significantly change the... algorithm, for lack of a better word. At least I assume something major changed between how they trained ChatGPT 3 vs GPT 4, other than amount of data. But maybe I'm wrong.
- eru 2y agoThe number is purely for marketing. If you could get much better performance without changing the algorithm (eg just by scaling), you'd still bump the number.
- ekojs 2y ago> This will mark the first experimental model with higher rate limits + billing. Excited for this to land and for folks to really put the model through the paces! From https://x.com/OfficialLoganK/status/1904583353954882046 https://x.com/OfficialLoganK/status/1904583353954882046 The low rate-limit really hampered my usage of 2.0 Pro and the like. Interesting to see how this plays out.
- chipgap98 2y agoAny word on what that pricing is? I can't seem to find it
- rvnx 2y agoTraditionally at Google experimental models are 100% free to use on https://aistudio.google.com https://aistudio.google.com (this is also where you can see the pricing) with a quite generous rate limit. This time, the Googler says: “good news! you will be charged for experimental models, though for now it’s still free”
- chipgap98 2y agoRight but the tweet I was responding to says: "This will mark the first experimental model with higher rate limits + billing. Excited for this to land and for folks to really put the model through the paces!" I assumed that meant there was a paid version with a higher rate limit coming out today
- altbdoor 2y agoThe parent Twitter post mentions: Available as experimental and for free right now in Google AI Studio + API, with pricing coming very soon! And the pricing page [1] still does not show 2.5 yet. [1]: https://ai.google.dev/gemini-api/docs/pricing https://ai.google.dev/gemini-api/docs/pricing
- KoolKat23 2y agoI expect this might be pricier. Hoping not unusable level expensive.
- M4v3R 2y agoThe Long Context benchmark numbers seem super impressive. 91% vs 49% for GPT 4.5 at 128k context length.
- Workaccount2 2y agoGoogle has the upperhand here because they are not dependent on nvidia for hardware. They make and uses their own AI accelerators.
- alexdzm 2y agoKeen to hear more about this benchmark. Is it representative of chat-to-document style usecases with big docs?
- sebzim4500 2y agoLooks like it's this benchmark [1]. It's certainly less artificial than most long context benchmarks (that are basically just a big lookup table) but probably not as representative as Fiction.LiveBench [2], which asks specific questions about works of fanfiction (which are typically excluded from training sets because they are basically porn). [1] https://arxiv.org/pdf/2409.12640 https://arxiv.org/pdf/2409.12640 [2] https://fiction.live/stories/Fiction-liveBench-Feb-20-2025/oQdzQvKHw8JyXbN87 https://fiction.live/stories/Fiction-liveBench-Feb-20-2025/o...
- swyx 2y ago"MRCR (multi-round coreference resolution)" for those looking for the link to Michaelangelo
- sebzim4500 2y agoUpdate: Gemini 2.5 also crushes fiction.livebench
- falcor84 2y agoI'm most impressed by the improvement on Aider Polyglot; I wasn't expecting it to get saturated so quickly. I'll be looking to see whether Google would be able to use this model (or an adapted version) to tackle ARC-AGI 2.
- serjester 2y agoI wish they’d mention pricing - it’s hard to seriously benchmark models when you have no idea what putting it in production would actually cost.
- impulser_ 2y agoIt's experimental. You shouldn't be using it in production.
- chipgap98 2y ago> This will mark the first experimental model with higher rate limits + billing. Excited for this to land and for folks to really put the model through the paces! From https://x.com/OfficialLoganK/status/1904583353954882046 https://x.com/OfficialLoganK/status/1904583353954882046
- Oras 2y agoThese announcements have started to look like a template. - Our state-of-the-art model. - Benchmarks comparing to X,Y,Z. - "Better" reasoning. It might be an excellent model, but reading the exact text repeatedly is taking the excitement away.
- Mistletoe 2y agoI’m sure the AI helps write the announcements.
- ototot 2y agoMaybe they just asked Gemini 2.5 to write the announcement.
- cpeterso 2y agoAnd it was trained on the previous announcements.
- xlbuttplug2 2y ago... which were also written by earlier Gemini versions.
- belter 2y agoLLMs all the way down
- cratermoon 2y agoNot all the way. At the bottom are a bunch of unpaid writers and artists and a horde of low-paid mturk workers in Nigeria.
- finebalance 2y agoThat's ok. AI will kill those off soon enough, and like all winners, rewrite history enough so that that inconvenient theft never happened anyway. It's manifest destiny, or something.
- jnd0 2y ago> with Gemini 2.5, we've achieved a new level of performance by combining a significantly enhanced base model with improved post-training. Going forward, we’re building these thinking capabilities directly into all of our models, so they can handle more complex problems and support even more capable, context-aware agents. Been playing around with it and it feels intelligent and up to date. Plus is connected to the internet. A reasoning model by default when it needs to. I hope they enable support for the recently released canvas mode for this model soon it will be a good match.
- Workaccount2 2y agoIt is almost certainly the "nebula" model on LLMarena that has been generating buzz for the last few days. I didn't test coding but it's reasoning is very strong.
- joelthelion 2y agoIs this model going to be restricted to paying users?
- hackermeows 2y ago[flagged]
- vivzkestrel 2y agohi, here is our new AI model, it performs task A x% better than our competitor 1, task B y% better than our competitor 2 seems to be the new hot AI template in town
- andai 2y agoCan anyone share what they're doing with reasoning models? They seem to only make a difference with novel programming problems, like Advent of Code. So this model will help solve slightly harder advent of codes. By extension it should also be slightly more helpful for research, R&D?
- bravura 2y agoWe're using it to RCA infrastructure incidents.
- fn-mote 2y agoSeriously? That doesn't require a human?! Are we talking about some kind of "generic" incident? (Type 3: forgot to manually update the xxxx file.) Or what's going on?
- deleted 2y ago[deleted]
- sandos 2y agoSounds unbelievable to me, but hey... :) If theyre that easy, why not fix the casues for the needs for RCA? Our RCAs will not be solved by AI for decades, let me tell you that.
- liuliu 2y agoI found reasoning models are much more faithful at text related tasks too (i.e. 1. translating long key-value pairs (i.e. Localizable.strings), 2. long transcript fixing and verification; 3. look at csv / tabular data and fix) probably due to the reflection mechanism built into these reasoning models. Using prompts such as "check your output to make sure it covers everything in the input" letting the model to double-check its work, avoiding more manual checks on my end.
- mgens 2y agoHave been using them for non-interactive coding where latency is not an issue. Specifically, turning a set of many free-text requirements into SQL statements, so that later when an item's data is entered into the system, we can efficiently find which requirements it meets. The reasoning models' output quality is much better than the non-reasoning models like 3.5 Sonnet, it's not a subtle difference.
- throwaway13337 2y agoGoogle has this habit of 'releasing' without releasing AI models. This looks to be the same? I don't see it on the API price list: https://ai.google.dev/gemini-api/docs/pricing https://ai.google.dev/gemini-api/docs/pricing I can imagine that it's not so interesting to most of us until we can try it with cursor. I look forward to doing so when it's out. That Aider bench mixed with the speed and a long context window that their other models are known for could be a great mix. But we'll have to wait and see. More generally, it woud be nice for these kinds of releases to also add speed and context window as a separate benchmark. Or somehow include it in the score. A model that is 90% as good as the best but 10x faster is quite a bit more useful. These might be hard to mix to an overall score but they're critical for understanding usefulness.
- minimaxir 2y agoIt's available now as an option in Google AI Studio and Google Gemini.
- ototot 2y agoand the price is 0.0 usd, lol
- wenyong3124 2y ago[dead]
- throwaway13337 2y agoThanks. I think my post lacked clarity of what I was talking about. I meant that most people care about API access to use with their favorite editor. It's a big limiter with grok, for example. But I did mingle that with my knowledge of google's history of releasing without releasing these models which, as you point out, isn't true with this release.
- kmod 2y agoIt's "experimental", which means that it is not fully released. In particular, the "experimental" tag means that it is subject to a different privacy policy and that they reserve the right to train on your prompts. 2.0 Pro is also still "experimental" so I agree with GP that it's pretty odd that they are "releasing" the next version despite never having gotten to fully releasing the previous version.
- deleted 2y ago[deleted]
- barrenko 2y agoThe incumbent has awoken.
- andai 2y agoHow does Gemini have such a big context window? I thought memory requirement grows exponentially with context size?
- mmoskal 2y agoMemory grows linearly, compute grows quadratically (but with small constant - until ~100k the inference will be still dominated by non-quadratic factors).
- xiphias2 2y agoAlso reusing key/values for different queries can compress the KV cache, it can be an 1000x or 10000x improvement in bandwidth if the model is trained for it.
- mmoskal 2y agoJust to clarify: simple prefix KV cache doesn't require any special model training. It does require the inference framework to support it, but most do by now. You can see dramatic improvements in latency and throughput if there is a large shared prefix of the queries.
- ototot 2y agoThe Gemini 1.5 tech report do reference some papers about supporting large context window.
- throitallaway 2y agoFunnyish story: the other night I asked my Pixel 9 to generate an image via Gemini, then I asked it to make a change. It didn't consider the previous context, so I asked it "Are you capable of keeping context?" No matter how clearly I enunciated "context", it always interpreted what I was saying as "contacts." After the 4th try, I said "context, spelled "c-o-n-t-e-x-t" and it replied with "Ah, you meant context! Yes..." This stuff has a long way to go.
- 2y ago
- famouswaffles 2y agoFrom the 2.0 line, the Gemini models have been far better at Engineering type questions (fluids etc) than GPT, Claude especially with questions that have Images that require more than just grabbing text. This is even better.
- andrewinardeer 2y agoGoogle is overly cautious with their guardrails. Granted, Gemini answers it now, however, this one left me shaking my head. https://cdn.horizon.pics/PzkqfxGLqU.jpg https://cdn.horizon.pics/PzkqfxGLqU.jpg
- slongfield 2y agoFor better or worse, Google gets more bad press when their models get things wrong compared to smaller AI labs.
- rfoo 2y agoHa, I still remember that super hilarious "You are under 18, so you should not write C++, as it is unsafe..." log from ... a year ago?
- summerlight 2y agoLooks like they're gradually removing guardrails, it returns Nixon for me.
- daquisu 2y agoWeird, they released Gemini 2.5 but I still can't use 2.0 pro with a reasonable rate limit (5 RPM currently).
- noisy_boy 2y agoIs Gemini and Bard same? I asked it a question and it said "... areas where I, as Bard, have..."
- zxilly 2y agoThere is no point in asking such questions, the model doesn't know what it is on its own, and you could get many different answers if you repeat it a few more times.
- comex 2y agoI was recently trying to replicate ClaudePlaysPokemon (which uses Claude 3.7) using Gemini 2.0 Flash Thinking, but it was seemingly getting confused and hallucinating significantly more than Claude, making it unviable (although some of that might be caused by my different setup). I wonder if this new model will do better. But I can't easily test it: for now, even paid users are apparently limited to 50 requests per day [1], which is not really enough when every step in the game is a request. Maybe I'll try it anyway, but really I need to wait for them to "introduce pricing in the coming weeks". Edit: I did try it anyway and so far the new model is having similar hallucinations. I really need to test my code with Claude 3.7 as a control, to see if it approach the real ClaudePlaysPokemon's semi-competence. Edit 2: Here's the log if anyone is curious. For some reason it's letting me make more requests than the stated rate limit. Note how at 11:27:11 it hallucinates on-screen text, and earlier it thinks some random offscreen tile is the stairs. Yes, I'm sure this is the right model: gemini-2.5-pro-exp-03-25. https://a.qoid.us/20250325/ https://a.qoid.us/20250325/ [1] https://ai.google.dev/gemini-api/docs/rate-limits#tier-1 https://ai.google.dev/gemini-api/docs/rate-limits#tier-1
- comex 2y agoUpdate: I tried a different version of the prompt and it's doing really well! Well, so far it's gotten out of its house and into Professor Oak's lab, which is not so impressive compared to ClaudePlaysPokemon, but it's a lot more than Gemini 2.0 was able to do with the same prompt.
- ototot 2y agoAnd OpenAI is announcing their ImageGen in 4o https://news.ycombinator.com/item?id=43474112 https://news.ycombinator.com/item?id=43474112
- malisper 2y agoI've been using a math puzzle as a way to benchmark the different models. The math puzzle took me ~3 days to solve with a computer. A math major I know took about a day to solve it by hand. Gemini 2.5 is the first model I tested that was able to solve it and it one-shotted it. I think it's not an exaggeration to say LLMs are now better than 95+% of the population at mathematical reasoning. For those curious the riddle is: There's three people in a circle. Each person has a positive integer floating above their heads, such that each person can see the other two numbers but not his own. The sum of two of the numbers is equal to the third. The first person is asked for his number, and he says that he doesn't know. The second person is asked for his number, and he says that he doesn't know. The third person is asked for his number, and he says that he doesn't know. Then, the first person is asked for his number again, and he says: 65. What is the product of the three numbers?
- bbstats 2y agoCould you explain "The sum of two of the numbers is equal to the third"??
- deleted 2y ago[deleted]
- malisper 2y agoif the three numbers are a, b, and c, then either a+b=c, a+c=b, or b+c=a
- bena 2y agoAnd they must all be positive integers. So A + B = C and A + C = B. But we know that A + B = C, so we can replace C with (A + B). So we know that A + A + B = B. So 2A + B = B. Or 2A = 0. And this holds any way you slice it. Even if you were to try and brute force it. A = 1 B = 2 Then C = 3. But A + C has to equal B. That's 1 + 3 = 2? That's not true. I don't see a case where you can add to the sum of two numbers one of the numbers and get the other number. I'm guessing that's a misreading of the problem. Because it looks like the third number is the sum of the first two.
- honeybadger1 2y agoClaude is still the king right now for me. Grok is 2nd in line, but sometimes it's better.
- nickandbro 2y agoWow, was able to nail the pelican riding on a bicycle test: https://www.svgviewer.dev/s/FImn7kAo https://www.svgviewer.dev/s/FImn7kAo
- anon373839 2y agoThat's actually too good to believe. I have a feeling simonw's favorite test has been special-cased...
- Workaccount2 2y agoIt seems pretty good at it. The hair on the boy is messed up, but still decent. "A boy eating a sandwhich" https://www.svgviewer.dev/s/VhcGxnIR https://www.svgviewer.dev/s/VhcGxnIR "A multimeter" https://www.svgviewer.dev/s/N5Dzrmyt https://www.svgviewer.dev/s/N5Dzrmyt
- sebzim4500 2y agoI doubt it is explicitly special cased, but now that it's all over twitter etc. it will have ended up many times in the training data.
- KTibow 2y agoThey could've RLed on SVGs - wouldn't be hard to render them, test adherence through Gemini or CLIP, and reward fittingly
- locallost 2y agoWhat does nail mean? That's not a bicycle.
- TonyTrapp 2y agoTo be honest, it's in good company with real humans there: https://www.behance.net/gallery/35437979/Velocipedia https://www.behance.net/gallery/35437979/Velocipedia Maybe it learned from Gianluca's gallery!
- xnx 2y agoIt will be huge achievement if models can get to the point where so much selection effort isn't required: gemini.google.com currently lists 2.0 Flash, 2.0 Flash Thinking (experimental), Deep Research, Personalization (experimental), and 2.5 Pro (experimental) for me.
- lxgr 2y agoThere's probably a sweet spot here. On the flip side, ChatGPT currently doesn't indicate whether a given image generation request was serviced by multimodal GPT-4o [1] or Dall-E. Personally, I do like the "use web search" and "extended thinking" buttons, but ultimately, the models should probably be able to figure out whether doing so would be useful themselves too. [1] https://news.ycombinator.com/item?id=43474112 https://news.ycombinator.com/item?id=43474112
- resource_waste 2y agoI'll try it tonight, but I'm not excited, its just work. ChatGPT4.5, I was excited. Deepseek, I was excited. (then later disappointed) I know Gemini probably wont answer any medical question, even if you are a doctor. ChatGPT will. I know I've been disappointed at the quality of Google's AI products. They are backup at best.
- FergusArgyll 2y agoIt interpreted blood work for me (Everything's ok, I'm just testing it ;)
- eenchev 2y ago"My info, the stuff I was trained on, cuts off around early 2023." - Gemini 2.5 to me. Appears that they did a not-so-recent knowledge cutoff in order to use the best possible base model.
- staticman2 2y agoIt's unlikely the model knows its actual cutoff date. Try asking 2024 news- for example in my test it knows the January 2024 Oscar nominees. On AI studio the model told me today is June 13 2024.
- afro88 2y agoIs this the first model announcement where they show Aider's Polyglot benchmark in the performance comparison table? That's huge for Aider and anotherpaulg!
- Dowwie 2y agoThis model is a fucking beast. I am so excited about the opportunities this presents.
- arjun_krishna1 2y agoI've been using Gemini Pro for my University of Waterloo capstone engineering project. Really good understanding of PDF documents and good reasoning as well as structured output Recommend trying it out at aistudio dot google dot com
- summerlight 2y agoThis looks like the first model where Google seriously comes back into the frontier competition? 2.0 flash was nice for the price but it's more focused on efficiency, not the performance.
- dcchambers 2y ago> Developers and enterprises can start experimenting with Gemini 2.5 Pro in Google AI Studio now, and Gemini Advanced users can select it in the model dropdown on desktop and mobile. It will be available on Vertex AI in the coming weeks. I'm a Gemini Advanced subscriber, still don't have this in the drop-down model selection in the phone app, though I do see it on the desktop webapp.
- ehsankia 2y agoI see it in both, probably just some gradual rollout delays.
- cp9 2y agodoes it still suggest glue on pizza
- lvl155 2y agoWith recent pace of model updates, I wonder which factor is more important: hardware assets, software/talent, or data access. Google clearly is in the lead in terms of data access in my view. If I am a top talent in AI, I’d go where I can work with the best data no?
- CuriouslyC 2y agoThe problem Goog has is its insane bureaucracy and lack of vision from Sundar, which isn't very attractive from an employee position. If you're working close to Demis I imagine the situation is better though.
- rfoo 2y agoNow that Noam is back I'm a little bit more optimistic.
- danpalmer 2y agoI think an argument could be made for hardware too. Perhaps in absolute terms Nvidia is ahead, but in terms of knowing how to get the most out of the hardware, Google making its own chips, building on their networking, etc, is a pretty big advantage. (Disclaimer, Googler, but I don’t work on any of this, I only have an external layperson’s understanding of it)
- jillesvangurp 2y agoUX is actually increasingly the bottleneck. Most of the top models are very good if you micromanage their context and prompts. But people aren't very good at that stuff. Some of the desktop chat clients are turning into great productivity tools. I tried the Claude one last week and quickly went back to Chat GPT. Claude might be a better model for coding. But it's less effort to make Chat GPT do what I want at this point and it's kind of good enough for a lot of stuff. Every release it's getting better. It connects to my IDE automatically, it can look at the files I have open. It can patch those files (I actually disabled that because it's too slow for my taste), etc. But most importantly, I can trigger all that with option+shift+1. I do this gazillions times per day. Mostly simple stuff with really short prompts, "check this" (file, selection, current line, etc.), fix that, what do you think about x, "address the FIXMEs/TODOs", "document this", etc. I can ask other models the same questions and they'd get the job done. But then I have to do more work to give them the same context. Claude has a Github connect option, which is great. But unfortunately it's just a glorified file picker, which really sucks. I have files open in my editor, just look at those. I don't want to have to manually open files do that for me or specify what files to look at every time I go near the tool. Chat GPT actually asked me yesterday whether it could add a different file than the one it was looking at. I said "yes" and it did. That's a great UX. Don't make me do work. That's a good UX. I use Gemini mainly because it's integrated into google's tools. So it's kind of there. And chat GPT for whatever reason does can not look at the browser window. But from a UX point of view, that kind of deep integration is what you want. You have this implicit shared context which is the thing you are looking at that you don't have to spell out anymore. The UX of populating the context is the deciding factor in how useful models are at this point, not how well it solves pet benchmark questions or renders pelicans on bicycles. I have good hopes for agentic coding tools progressing rapidly this year. The ones I've tried recently need a lot of work though. I keep going back to Chat GPT because it's just the quickest & easiest to use at this point.
- d3nj4l 2y agoA model that is better on Aider than Sonnet 3.7? For free, right now? I think I'll give it a spin this weekend on a couple of projects, seems too good to be true.
- istjohn 2y agoWith a rate limit of 50 requests per day
- sagarpatil 2y agoCould use multiple Google accounts to increase the rate limit.
- Alifatisk 2y agoThis is why we can't have nice things
- deleted 2y ago[deleted]
- pachico 2y agoIt really surprises me that Google and Amazon, considering their infrastructure and the urge to excel at this, aren't leading the industry.
- WasimBhai 2y agoI do not intend to take anything away from the technical achievement of the team. However, as Satya opined some weeks back, these benchmarks do not mean a lot if we do not see a comparable increase in productivity. But then there are two questions. First, are the white collar workers specifically consultants, engineers responsible for increase in productivity? Or is the white collar workers at the very right tail e.g., scientists? I think consultants and engineers are using these technologies a lot. I think biologists at least are using these models a lot. But then where is the productivity increases?
- ZeroTalent 2y agoAlso, why not compare to GPT-o3 in the benchmarks?
- KoolKat23 2y agoThe models not really available.
- ZeroTalent 2y agothey have access to o3, I do. Thousands of people do(tens of thousands at this point?). Come on. Compare to SOTA, when you're saying it's the best AI you have.
- KoolKat23 2y agoI honestly wasn't aware it's available to the few outside of pro.
- KoolKat23 2y agoAs it stands currently I'd say this is difficult to measure. They're not baked into workflows where the measurable output is attributed easily to the model use. Productivity in its current form is transformative in the sense that the use case and gain differs for the individual (who even provide different prompts). So some are keeping the gains for themselves, others are using it to improve quality rather than quantity. It'll come in time, it's important to remember gpt 4 was released 2 years ago this month. The newer models are more reliable and could probably be introduced into workflows more frequently. Today I spoke to a company who are looking to use it to reduce cost in the next year.
- jorl17 2y agoJust a couple of days ago I wrote on reddit about how long context models are mostly useless to me, because they start making too many mistakes very fast. They are vaguely helpful for "needle in a haystack" problems, not much more. I have a "test" which consists in sending it a collection of almost 1000 poems, which currently sit at around ~230k tokens, and then asking a bunch of stuff which requires reasoning over them. Sometimes, it's something as simple as "identify key writing periods and their differences" (the poems are ordered chronologically). Previous models don't usually "see" the final poems — they get lost, hallucinate and are pretty much worthless. I have tried several workaround techniques with varying degrees of success (e.g. randomizing the poems). Having just tried this model (I have spent the last 3 hours probing it), I can say that, to me, this is a breakthrough moment. Truly a leap. This is the first model that can consistently comb through these poems (200k+ tokens) and analyse them as a whole, without significant issues or problems. I have no idea how they did it, but they did it. The analysis of this poetic corpus has few mistakes and is very, very, very good. Certainly very good in terms of how quickly it produces an answer — it would take someone days or weeks of thorough analysis. Of course, this isn't about poetry — it's about passing in huge amounts of information, without RAG, and having a high degree of confidence in whatever reasoning tasks this model performs. It is the first time that I feel confident that I could offload the task of "reasoning" over large corpus of data to an LLM. The mistakes it makes are minute, it hasn't hallucinated, and the analysis is, frankly, better than what I would expect of most people. Breakthrough moment.
- Alifatisk 2y agoTwo years ago, Claude was known for having the largest context window and being able to remember tokens throughout the whole conversation. Today, it seems like Google has beat them and supports way larger context window and is way better at keeping track of what has being said and memorize older tokens.
- greatgib 2y agoIf you plan to use Gemini, be warned, here are the usual Big Tech dragons: Please don’t enter ...confidential info or any data... you wouldn’t want a reviewer to see or Google to use ... The full extract of the terms of usage: How human reviewers improve Google AI To help with quality and improve our products (such as the generative machine-learning models that power Gemini Apps), human reviewers (including third parties) read, annotate, and process your Gemini Apps conversations. We take steps to protect your privacy as part of this process. This includes disconnecting your conversations with Gemini Apps from your Google Account before reviewers see or annotate them. Please don’t enter confidential information in your conversations or any data you wouldn’t want a reviewer to see or Google to use to improve our products, services, and machine-learning technologies.
- greatgib 2y agoAnd even worse: Conversations that have been reviewed or annotated by human reviewers (and related data like your language, device type, location info, or feedback) are not deleted when you delete your Gemini Apps activity because they are kept separately and are not connected to your Google Account. Instead, they are retained for up to three years. Emphasis on "retained for up to three years" even if you delete it!!
- mastodon_acc 2y agoHow does it compare to OpenAI and anthropic’s user data retention policy?
- greatgib 2y agoIf i'm not wrong, Chatgpt states clearly that they don't use user data anymore by default. Also, maybe some services are doing "machine learning" training with user data, but it is the first time I see recent LLM service saying that you can feed your data to human reviewers at their will.
- cubefox 2y ago
- TheMagicHorsey 2y agoI tested out Gemini 2.5 and it failed miserably at calling into tools that we had defined for it. Also, it got into an infinite loop a number of times where it would just spit out the exact same line of text continuously until we hard killed the process. I really don't know how others are getting these amazing results. We had no problems using Claude or OpenAI models in the same scenario. Even Deepseek R1 works just fine.
- batata_frita 2y agoWhy do I have the feel that nobody is too much excited to google's models compared to other companies?
- Mond_ 2y agoGoogle is worse at marketing and hyping people up.
- dmix 2y agoThe internal incentives must not align with new things making money.
- bklyn11201 2y agoMost of us care only about coding performance, and Sonnet 3.5 has been such a giant winner that we don't get too excited about the latest model from Google.
- SamuelAdams 2y agoBecause it’s more likely to be sunsetted. https://killedbygoogle.com/ https://killedbygoogle.com/
- owlninja 2y agoYea I get a little bummed but I guess a lot of HNers have reasons to not like google. I've had a Google One membership forever so opted for the higher subscription with Gemini access since the beginning (plus a free year with new Pixel phone). and I think it is awesome.
- ipsum2 2y agoThey're not good models. They over fit to LMArena leaderboard, but perform worse in real life scenarios compared to their competitors. The exceptions are auto regressive image generation and audio models.
- Davidzheng 2y agoI feel like Google intentionally don't want people to be as excited. This is a very good model. Definitely the best available model today.
- asah 2y agoIt nailed my two hard reasoning+linguistic+math questions in one shot, both the kinds of things that LLM struggle but humans do well. (DM me for the questions)
- rodolphoarruda 2y agoI've been trying to use Gemini 2.0 Flash, but I don't think it's possible. The model still thinks it's running the 1.5 Pro model. Reference: https://rodolphoarruda.pro.br/wp-content/uploads/image-14.png https://rodolphoarruda.pro.br/wp-content/uploads/image-14.pn...
- strstr 2y agoIt's a lot better at my standard benchmark "Magic: The Gathering" rules puzzles. Gets the answers right (both the outcome and rationale).
- strstr 2y agoOoof, it failed my "Wheel of Potential" bug finding question, and got aggressive about asserting it was correct.
- Davidzheng 2y agoOn initial thoughts, I think this might be the first AI model to be reliably helpful as a research assistant in pure mathematics (o3-mini-high can be helpful but is more prone to hallucinations)
- kadushka 2y agoHave you tried o1-pro?
- simonw 2y agoI'm impressed by this one. I tried it on audio transcription with timestamps and speaker identification (over a 10 minute MP3) and drawing bounding boxes around creatures in a complex photograph and it did extremely well on both of those. Plus it drew me a very decent pelican riding a bicycle. Notes here: https://simonwillison.net/2025/Mar/25/gemini/ https://simonwillison.net/2025/Mar/25/gemini/
- kridsdale3 2y agoI just tried your trademark benchmark on the new 4o Image Output, though it's not the same test: https://imgur.com/a/xuPn8Yq https://imgur.com/a/xuPn8Yq
- Unroasted6154 2y agoWas that an actual SVG?
- simonw 2y agoNo that's GPT-4o native image output.
- sebzim4500 2y agoI wonder how far away we are from models which, given this prompt, generate that image in the first step in their chain-of-thought and then use it as a reference to generate SVG code. It could be useful for much more than just silly benchmarks, there's a reason why physics students are taught to draw a diagram before attempting a problem.
- simonw 2y agoSomeone managed to get ChatGPT to render the image using GPT-4o, then save that image to a Code Interpreter container and run Python code with OpenCV to trace the edges and produce an SVG: https://bsky.app/profile/btucker.net/post/3lla7extk5c2u https://bsky.app/profile/btucker.net/post/3lla7extk5c2u
- famouswaffles 2y agoOne of the biggest problems with hands off LLM writing (for long horizon stuff like novels) is that you can't really give them any details of your story because they get absolutely neurotic with it. Imagine for instance you give the LLM the profile of the love interest for your epic fantasy, it will almost always have the main character meeting them within 3 pages (usually page 1) which is of course absolutely nonsensical pacing. No attempt to tell it otherwise changes anything. This is the first model that after 19 pages generated so far resembles anything like normal pacing even with a TON of details. I've never felt the need to generate anywhere near this much. Extremely impressed. Edit: Sharing it - https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221OgLP7Wfj3zP3-_bhc6U0GhtfJRSuDqqH%22%5D,%22action%22:%22open%22,%22userId%22:%22115416429261485925946%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%... with pastebin - https://pastebin.com/aiWuYcrF https://pastebin.com/aiWuYcrF
- fshr 2y agoI think you would be better off having the LLM help you build up the plot with high level chapter descriptions and then have it dig into each chapter or arc. Or start by giving it the beats before you ask it for help with specifics. That'd be better at keeping it on rails.
- famouswaffles 2y agoI don't disagree. Like with almost anything else involving LLMs, getting hands on produces better results but because in this instance, i much prefer to be the reader than the author or editor, it's really important to me that a LLM is capable of pacing long form writing properly on its own.
- deleted 2y ago[deleted]
- saberience 2y agoRandom question, if you don't care about being a creator yourself, why do you even want to read long form writing written by an LLM? There are literally 10000s of actual human written books out there all of them better than anything an LLM can write, why not read them?
- mclau156 2y agoGenerated 1000 lines of turn based combat with shop, skills, stats, elements, enemy types, etc. with this one
- nikcub 2y agoImpressive model - but I'm confused by the knowledge cutoff. AI Studio says it is January 2025 (which would be impressive) but querying it for anything early 2025 or mid/late 2024 and it self-reports that it's cutoff is in 2023 (which can't be right). This is most evident when querying about fast-moving dev tools like uv or bun. It seems to only know the original uv options like pip and tools, while with bun it is unfamiliar with bun outdated (from Aug 2024), bun workspaces (from around that time?) but does know how to install bun on windows (April 2024). You'll still need to provide this model with a lot of context to use it with any tooling or libraries with breaking changes or new features from the past ~year - which seems to contradict the AI Studio reported knowledge cutoff. Were I developing models - I'd prioritise squeezing in the most recent knowledge of popular tools and libraries since development is such a popular (and revenue generating) use case.
- foota 2y agoMaybe less has been written about these newer things, even if they had technically been released?
- joshdavham 2y agoWhen these companies release a model “2.5”, are they using some form of semver? Where are these numbers coming from?
- eru 2y agoMarketing.
- zone411 2y agoScores 54.1 on the Extended NYT Connections Benchmark, a large improvement over Gemini 2.0 Flash Thinking Experimental 01-21 (23.1). 1 o1-pro (medium reasoning) 82.3 2 o1 (medium reasoning) 70.8 3 o3-mini-high 61.4 4 Gemini 2.5 Pro Exp 03-25 54.1 5 o3-mini (medium reasoning) 53.6 6 DeepSeek R1 38.6 7 GPT-4.5 Preview 34.2 8 Claude 3.7 Sonnet Thinking 16K 33.6 9 Qwen QwQ-32B 16K 31.4 10 o1-mini 27.0 https://github.com/lechmazur/nyt-connections/ https://github.com/lechmazur/nyt-connections/
- slama 2y agoInterestingly, the model hallucinated the ability to use a search tool when I was playing around with it
- freediver 2y agoTops our benchmark in an unprecedented way. https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html High quality, to the point. Bit on the slow side. Indeed a very strong model. Google is back in the game big time.
- aoeusnth1 2y agoIt should be in the "reasoning" category, right? (still topping the charts there)
- utopcell 2y agoThat is some wide gap!
- causal 2y agoRemarkable how few tokens it needed to get a much better score than other reasoning models. Any chance of contamination?
- 85392_school 2y agoIt makes me wonder how the token counting was implemented and if it missed the (not sent in API) reasoning.
- freediver 2y agoVaild concern, most likely thinking tokens were not counted due to API reporting changes.
- anotherpaulg 2y agoGemini 2.5 Pro set the SOTA on the aider polyglot coding leaderboard [0] with a score of 73%. This is well ahead of thinking/reasoning models. A huge jump from prior Gemini models. The first Gemini model to effectively use efficient diff-like editing formats. [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- aoeusnth1 2y agoAm I correct in assuming that accuracy < using correct edit format? i.e. it made mistakes in 27% of the problems, 11% of which were due to (at least) messing up the diff format? In which case, google should be working on achieving better output format following, as Claude and R1 are able to hit nearly 100% accuracy on the format.
- anotherpaulg 2y agoIt does have fairly low adherence to the edit format, compared to the other frontier models. But it is much better than any previous Gemini model in this regard. Aider automatically asks models to retry malformed edits, so it recovers. And goes on to produce a SOTA score.
- aoeusnth1 2y agoOk, thanks for clearing that up.
- sagarpatil 2y agoThe only benchmark I care about. Thanks!
- mindwok 2y agoJust adding to the praise: I have a little test case I've used lately which was to identify the cause of a bug in a Dart library I was encountering by providing the LLM with the entire codebase and description of the bug. It's about 360,000 tokens. I tried it a month ago on all the major frontier models and none of them correctly identified the fix. This is the first model to identify it correctly.
- ripped_britches 2y agoWow holy smokes that is exciting
- kungfufrog 2y agoHow do you use the model so quickly? Google AI Studio? Maybe I've missed how powerful that is.. I didn't see any easy way to pass it a whole code base!
- mindwok 2y agoYep! AI studio I think is the only way you can actually use it right now and AFAIK it's free.
- weatherlite 2y ago360k tokens = how many lines of code approximately ? and also, if its an open source lib are you sure there's no mentions of this bug anywhere on the web?
- mindwok 2y agoNot a huge library, around 32K LoC and no mention of the bug on the web - I was the first to encounter it (it’s since been fixed) unless the training data is super recent.
- weatherlite 2y agoImpressive. I tend to think it managed to find the bug by itself which is pretty crazy without being able to debug anything. Then again I haven't seen the bug description, perhaps the description makes it super obvious where the problem lies.
- simonw 2y agoHere's a Gemini 2.5 provided summary of this Hacker News thread as of the moment when it had 269 comments: https://gist.github.com/simonw/3efa62d917370c5038b7acc24b7c786e https://gist.github.com/simonw/3efa62d917370c5038b7acc24b7c7... I ran this command to create it: curl -s "https://hn.algolia.com/api/v1/items/43473489" | \ jq -r 'recurse(.children[]) | .author + ": " + .text' | \ llm -m "gemini-2.5-pro-exp-03-25" -s \ 'Summarize the themes of the opinions expressed here. For each theme, output a markdown header. Include direct "quotations" (with author attribution) where appropriate. You MUST quote directly from users when crediting them, with double quotes. Fix HTML entities. Output markdown. Go long. Include a section of quotes that illustrate opinions uncommon in the rest of the piece' Using this script: https://til.simonwillison.net/llms/claude-hacker-news-themes https://til.simonwillison.net/llms/claude-hacker-news-themes
- billforsternz 2y agoI know next to nothing about AI, but I just experienced an extraordinary hallucination in a google AI search (presumably an older Gemini model right?) as I elaborated in detail in another HN thread. It might be a good test question. https://news.ycombinator.com/item?id=43477710 https://news.ycombinator.com/item?id=43477710
- testycool 2y agoIt feels like Gemini 2.0 Pro + Reasoning. I also see Gemini 2.0 Pro has been replaced completely in AI Studio.
- jszymborski 2y agoGemini refuses to answer any questions on poprtional swing models or anything related to psephology on the grounds that it has to do with elections. Neither Claude nor ChatGPT nor Mistral/Le Chat are that neutered.
- eru 2y agoI assume Gemini would be less neutered in this regard, if it wasn't developed by Google.
- ofermend 2y agoThis model is quite impressive. Not just useful for math/research with great reasoning, it also maintained a very low hallucination rate of 1.1% on Vectara Hallucination Leaderboard: https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard
- Medicineguy 2y agoWhile I'm sure the new Gemini model has made improvements, I feel like the user experience outside of the model itself is stagnating. I think OpenAI's interfaces, both web app and mobile app, are quite a bit more polished currently. For example, Gemini's speech recognition struggles with longer pauses and often enough cuts me off mid-sentence. Also, OpenAIs whisper model understands more context (for instance, saying “[...] plex, emby and Jellyfin [...]” is usually understood in whisper, but less often in Gemini) The Gemini web app lacks keyboard shortcuts for basic actions like opening a new chat or toggling the sidebar (good for privacy friendly pair programming). Last point off the top of my head would be the ability to edit messages beyond just the last one. That's possible in ChatGPT, but not in Gemini. Googlers are spending so much money for model training, I would appreciate spending some for making it fun to use :)
- skinkestek 2y agoCan it now generate images of soldier in typical uniforms from 1940s Germany without having to throw in a few token ethnicities? Or generate images of the founding fathers of US that at least to some degree resemble the actual ones?
- t_minus_40 2y agoi have asked the direction of friction on ball rolling either up or down on an inclined plan - it gave wrong answer and was adamant about it. Surprisingly, similar to o1. gave a problem which sounds like monty hall problem but a simple probability question and it nailed it. asked to tell a joke - horrible joke ever. much better than o1 but still no where near agi. it has been optimized for logic and reasoning at best.
- ascorbic 2y agoIt can answer my favourite riddle for LLMs: "Anna, Becca and Clare go to the play park. There is nobody else there. Anna is playing on the see-saw, Becca is playing on the swings. What is Clare doing?" (Sometimes I ask similar questions with the same structure and assumptions but different activities) About a year ago none of them could answer it. All the latest models can pass it if I tell them to think hard, but previously Gemini could rarely answer it without that extra hint. Gemini 2.5 caveats its answer a bit, but does get it correct. Interestingly GPT-4o initially suggests it will give a wrong answer without thinking, but recognises it's a riddle, so decides to think harder and gets it right.
- f1shy 2y agoOne test I always do is ask for an absolutely minimal language interpreter with TCO. This is part of the code output (after several interactions of it not returning actual code): // Tail Call Optimization (very basic) if(func->type == VAL_FUNCTION){ return apply(func, args, env); //no stack growth. } else{ return apply(func, args, env); } I'm not very impressed. I pointed out that part of the code, and answered: You've correctly pointed out that the TCO implementation in the provided C code snippet is essentially a no-op. The if and else blocks do the same thing: they both call apply(func, args, env). This means there's no actual tail call optimization happening; it's just a regular function call. But then follows with even worst code. It does not even compile!
- andai 2y agoI asked it for suggestions for a project, and it was the only model that correctly pointed out serious flaws in the existing proposal. So far so good!
- DaveMcMartin 2y agoI love to see this competition between companies trying to get the best LLM, and also, the fact that they’re trying to make them useful as tools, focusing on math, science, coding, and so on
- curtisszmania 2y ago[dead]
- marcus_holmes 2y agoI tried the beta version of this model to write a business plan (long story). I was impressed at first. Then it got really hung up on the financial model, and I had to forcibly move it on. After that it wrote a whole section in Indonesian, which I don't speak, and then it crashed. I'd not saved for a while (ever since the financial model thing), and ended up with an outline and a couple of usable sections. I mean, yes, this is better than nothing. It's impressive that we made a pile of sand do this. And I'm aware that my prompt engineering could improve a lot. But also, this isn't a usable tool yet. I'm curious to try again, but wary of spending too much time "playing" here.
- blinding-streak 2y agoLOL the random Indonesian section. That's incredible and so strange.
- Alifatisk 2y agoCan't wait for the benchmark at artificialanalysis.ai
- theragra 2y agoYeah, and then it says that call of duty is pronounced call of dah-tee when I speak in Russian. Chatgpt pronounced correctly
- meerab 2y agoThe Gemini 2.5 model is truly impressive, especially with its multimodal capability. Its ability to understand audio and video content is amazing—truly groundbreaking. I spent some time experimenting with Gemini 2.5, and its reasoning abilities blew me away. Here are few standout use cases that showcase its potential: 1. Counting Occurrences in a Video In one experiment, I tested Gemini 2.5 with a video of an assassination attempt on then-candidate Donald Trump. Could the model accurately count the number of shots fired? This task might sound trivial, but earlier AI models often struggled with simple counting tasks (like identifying the number of "R"s in the word "strawberry"). Gemini 2.5 nailed it! It correctly identified each sound, outputted the timestamps where they appeared, and counted eight shots, providing both visual and audio analysis to back up its answer. This demonstrates not only its ability to process multimodal inputs but also its capacity for precise reasoning—a major leap forward for AI systems. 2. Identifying Background Music and Movie Name Have you ever heard a song playing in the background of a video and wished you could identify it? Gemini 2.5 can do just that! Acting like an advanced version of Shazam, it analyzes audio tracks embedded in videos and identifies background music. I am also not a big fan of people posting shorts without specifying the movie name. Gemini 2.5 solves that problem for you - no more searching for movie name! 3. OCR Text Recognition Gemini 2.5 excels at Optical Character Recognition (OCR), making it capable of extracting text from images or videos with precision. I asked the model to output one of Khan Academy's handwritten visuals into a nice table format - and the text was precisely copied from video into a neat little table! 4. Listen to Foreign News Media The model can translate text from one language to another and give a good translation. I tested the recent official statement from Thai officials about an earthquake in Bangkok, and the latest news from a Marathi news channel. The model was correctly able to translate and output the news synopsis in the language of your choice. 5. Cricket Fans? Sports fans and analysts alike will appreciate this use case! I tested Gemini 2.5 on an ICC T20 World Cup cricket match video to see how well it could analyze gameplay data. The results were incredible: the model accurately calculated scores, identified the number of fours and sixes, and even pinpointed key moments—all while providing timestamps for each event. 7. Webinar - Generate Slides from Video Now this blew my mind - video webinars are generated by slide decks and a person talking about the slides. Can we reverse the process? Given a video, can we ask AI to output the slide deck? Google Gemini 2.5 outputted 41 slides for a Stanford webinar! Bonus: Humor Test Finally, I put Gemini 2.5 through a humor test using a PG-13 joke from one of my favorite YouTube channels, Mike and Joelle. I wanted to see if the model could understand adult humor and infer punchlines. At first, the model hesitated to spell out the punchline (perhaps trying to stay appropriate?), but eventually, it got there—and yes, it understood the joke perfectly! https://videotobe.com/blog/googles-gemini-25 https://videotobe.com/blog/googles-gemini-25