17 ms·

Gemini 2.5 Flash

1076 points by meetpateltech 2y ago

piizeus 2y ago
I don't think it is as good as deepseek v3 for updating coding tasks. I use 2.5 pro as architect with deepseek v3 as editor engineer. This combination work almost so perfect. Flash has advantages of great context size though. gpt 4.1 is also great.
xnx 2y ago
50% price increase from Gemini 2.0 Flash. That sounds like a lot, but Flash is still so cheap when compared to other models of this (or lesser) quality. https://developers.googleblog.com/en/start-building-with-gemini-25-flash/ https://developers.googleblog.com/en/start-building-with-gem...
- akudha 2y ago
  Is this cheaper than DeepSeek? Am I reading this right?
  - vdfs 2y ago
    Only if you don't use reasoning
- Tiberium 2y ago
  del
  - Havoc 2y ago
    You may want to consult Gemini on those percentage calcs .10 to .15 is not 25%
- swyx 2y ago
  done pretty much inline with the price elo pareto frontier https://x.com/swyx/status/1912959140743586206/photo/1 https://x.com/swyx/status/1912959140743586206/photo/1
  - xnx 2y ago
    Love that chart! Am I imagining that I saw a version of that somewhere that even showed how the boundary has moved out over time?
    - swyx 2y ago
      https://x.com/swyx/status/1882933368444309723 https://x.com/swyx/status/1882933368444309723 https://x.com/swyx/status/1830866865884991999 https://x.com/swyx/status/1830866865884991999 (scroll up)
  - oezi 2y ago
    So if I see it right flash 2.5 doesn't push the pareto front forward, right? It just sits between 2.5 pro and 2.0 flash. https://storage.googleapis.com/gweb-developer-goog-blog-assets/images/Gemini-2.5-Flash-price-to-performance-compariso.original.png https://storage.googleapis.com/gweb-developer-goog-blog-asse...
    - swyx 2y ago
      yeah but 1) its useful to have the point there on the curve if you need it, 2) intelligence is multidimensional, maybe in 2.5 flash you get qualitatively a better set of capabilities for your needs than 2.5 pro
    - jsnell 2y ago
      It does, that point in the tradeoff space was not available until now. Any model that's not dominated by at least one model on both axes will push forward the frontier. (The actual frontier isn't actually a straight line between the points on the frontier like visualized there. It's a step function.)
- onlyrealcuzzo 2y ago
  Why isn't Phi-3, Llama 3, or Mistral in the comparison? Aren't there a lot of hosted options? How do they compare in terms of cost?
byefruit 2y ago
It's interesting that there's a price nearly 6x price difference between reasoning and no reasoning. This implies it's not a hybrid model that can just skip reasoning steps if requested. Anyone know what else they might be doing? Reasoning means contexts will be longer (for thinking tokens) and there's an increase in cost to inference with a longer context but it's not going to be 6x. Or is it just market pricing?
- vineyardmike 2y ago
  Based on their graph, it does look explicitly priced along their “Pareto Frontier” curve. I’m guessing that is guiding the price more than their underlying costs. It’s smart because it gives them room to drop prices later and compete once other company actually get to a similar quality.
- jsnell 2y ago
  > This implies it's not a hybrid model that can just skip reasoning steps if requested. It clearly is, since most of the post is dedicated to the tunability (both manual and automatic) of the reasoning budget. I don't know what they're doing with this pricing, and the blog post does not do a good job explaining. Could it be that they're not counting thinking tokens as output tokens (since you don't get access to the full thinking trace anyway), and this is the basically amortizing the thinking tokens spend over the actual output tokens? Doesn't make sense either, because then the user has no incentive to use anything except 0/max thinking budgets.
- deleted 2y ago
  [deleted]
- RobinL 2y ago
  Does anyone know how this pricing works? Supposing I have a classification prompt where I need the response to be a binary yes/no. I need one token of output, but reasoning will obviously add far more than 6 additional tokens. Is it still a 6x price multiplier? That doesn't seem to make sense, but not does paying 6x more for every token including reasoning ones
  - 2y ago
punkpeye 2y ago
This is cool, but rate limits on all of these preview models are PITA
- Layvier 2y ago
  Agreed, it's not even possible to run an eval dataset. If someone from google see this please at least increase the burst rate limit
  - punkpeye 2y ago
    It is not without rate limits, but we do have elevated limits for our accounts through: https://glama.ai/models/gemini-2.5-flash-preview-04-17 https://glama.ai/models/gemini-2.5-flash-preview-04-17 So if you just want to run evals, that should do it. Though the first couple of days after a model comes out are usually pretty rough because everyone try to run their evals.
    - punkpeye 2y ago
      What I am noticing with every new Gemini model that comes out is that the time to first token (TTFT) is not great. I guess it is because they gradually transfer computer power from old models to new models as the demand increases.
      - Filligree 2y ago
        If you’re imagining that 2.5Pro gets dynamically loaded during the time to first token, then you’re vastly overestimating what’s physically possible. It’s more likely a latency-throughput tradeoff. Your query might get put inside a large batch, for example.
    - Layvier 2y ago
      That's very interesting, thanks for sharing!
arnaudsm 2y ago
Gemini flash models have the least hype, but in my experience in production have the best bang for the buck and multimodal tooling. Google is silently winning the AI race.
- belter 2y ago
  > Google is silently winning the AI race. That is what we keep hearing here...The last Gemini I cancelled the account, and can't help notice the new one they are offering for free...
  - arnaudsm 2y ago
    Sorry I was talking of B2B APIs for my YC startup. Gemini is still far behind for consumers indeed.
    - JeremyNT 2y ago
      I use Gemini almost exclusively as a normal user. What am I missing out on that they are far behind on? It seems shockingly good and I've watched it get much better up to 2.5 Pro.
      - arnaudsm 2y ago
        Mostly brand recognition and the earlier Geminis had more refusals. As a consumer, I also really miss the Advanced voice mode of ChatGPT, which is the most transformative tech in my daily life. It's the only frontier model with true audio-to-audio.
        wavewrangler 2y ago
        What do you mean miss? You don’t have the budget to keep something you truly miss for $20? What am in missing here / I don’t mean to criticize I am just curious is all. I would reword but I have to go
        what_ever 2y ago
transformi 2y ago
Bad day is going on google. First the decleration of illegal monopoly.. and now... Google’s latest innovation: programmable overthinking. With Gemini 2.5 Flash, you too can now set a thinking_budget—because nothing says "state-of-the-art AI" like manually capping how long it’s allowed to reason. Truly the dream: debugging a production outage at 2am wondering if your LLM didn’t answer correctly because you cheaped out on tokens. lol. “Turn thinking off for better performance.” That’s not a model config, that’s a metaphor for Google’s entire AI strategy lately. At this point, Gemini isn’t an AI product—it’s a latency-cost-quality compromise simulator with a text interface. Meanwhile, OpenAI and Anthropic are out here just… cooking the benchmarks
- danielbln 2y ago
  Google's Gemini 2.5 pro model is incredibly strong, it's en par and at times better than Claude 3.7 in coding performance, being able to ingest entire videos into the context is something I haven seen elsewhere either. Google AI products have been anywhere between bad (Bard) to lackluster (Gemini 1.5), but 2.5 is a contender, in all dimensions. Google is also the only player that owns the entire stack, from research, software , data, compute hardware. I think they were slow to start but they've closed the gap since.
- bsmith 2y ago
  Using AI to debug code at 2am sounds like pure insanity.
  - mring33621 2y ago
    the new normal
  - spiderice 2y ago
    They're suggesting you'll be up at 2am debugging code because your AI code failed. Not that you'll be using AI to do the debugging.
- SoftMachine 2y ago
  [dead]
hmaxwell 2y ago
I did some testing this morning: Prompt: "can you find any mistakes on my codebase? I put one in there on purpose" + 70,000 tokens of codebase where in one line I have an include for a non-existent file. Claude 3.7 Sonnet (Try 1 - new chat) Claimed it Found a race condition in the api of the admin interface that would be triggered if two admins were to change the room order at the same time. Claude suggested I group all sql queries in a single transaction. I looked at the code and found that it already used a transaction for all queries. I said: the order_update api is already done with a transaction. Claude replied: "You're absolutely right, and I apologize for my mistake. I was incorrect to claim there was a race condition issue. The transaction ensures atomicity and consistency of the updates, and the SQL queries are properly structured for their intended purpose." Claude 3.7 Sonnet (Try 2 - new chat) Found a comment in the admin ui javascript code that suggested a potential alternative to event handler cleanup that was not implemented because I decided to go with a cleaner route. Then asked "Is this the issue you intentionally included, or would you like me to look for other potential problems?" I said: "The comment merely suggests an alternative, right?" claude said: "Yes, you're absolutely right. The comment is merely suggesting an alternative approach that isn't being used in the code, rather than indicating a mistake. So there's no actual bug or mistake in this part of the code - just documentation of different possible approaches. I apologize for misinterpreting this as an issue!" Claude 3.7 Sonnet (Try 3 - new chat) When processing items out of the database to generate QR codes in the admin interface, Claude says that my code both attempts to generate QR codes with undefined data AS WELL AS saying that my error handling skips undefined data. Claude contradicts itself within 2 sentences. When asking about clarification Claude replies: Looking at the code more carefully, I see that the code actually has proper error handling. I incorrectly stated that it "still attempts to call generateQRCode()" in the first part of my analysis, which was wrong. The code properly handles the case when there's no data-room attribute. Gemnini Advanced 2.5 Pro (Try 1 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase. Gemnini Advanced 2.5 Pro (Try 2 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase. Gemnini Advanced 2.5 Pro (Try 3 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase. o4-mini-high and o4-mini and o3 and 4.5 and 4o - "The message you submitted was too long, please reload the conversation and submit something shorter."
Workaccount2 2y ago
OpenAI might win the college students but it looks like Google will lock in enterprise.
- xnx 2y ago
  ChatGPT seems to have a name recognition / first-mover advantage with college students now, but is there any reason to think that will stick when today's high school students are using Gemini on their Chromebooks?
- gundmc 2y ago
  Funny you should say that. Google just announced today that they are giving all college students one year of free Gemini advanced. I wonder how much that will actually move the needle among the youth.
  - Workaccount2 2y ago
    My guess is that they will use it and still call it "ChatGPT"...
    - xnx 2y ago
      Chat Gemini Pretrained Transformer
    - tantalor 2y ago
      Pass the Kleenex. Can I get a Band-Aid? Here's a Sharpie. I need a Chapstick. Let me Xerox that. Toss me that Frisbee.
      - drob518 2y ago
        Exactly.
      - esafak 2y ago
        Do you prefer those brands or just use their names? I google stuff on Kagi...
statements 2y ago
Interesting to note that this might be the only model with knowledge cut off as recent as 2025 January
- Tiberium 2y ago
  Gemini 2.5 Pro has the same knowledge cutoff specified, but in reality on more niche topics it's still limited to ~middle of 2024.
- brightball 2y ago
  Isn't Grok 3 basically real time now?
  - Tiberium 2y ago
    That's the web version (which has tools like search plugged in), other models in their official frontends (Gemini on gemini.google.com, GPT/o models on chatgpt.com) are also "real time". But when served over API, most of those models are just static.
  - bearjaws 2y ago
    No LLM is real time, and in fact, even a 2025 cut off isn't entirely realistic. Without guidance to say, a new version of a framework it will frequently "reference" documentation from old versions and use that. It's somewhat real time when it searches the web, of course that data is getting populated into context rather than in training.
    - arizen 1y ago
      You can query Grok on recent news and it will deliver the very recent updates (1 day old or probably even less)
  - jiocrag 2y ago
    Not at all. The model weights and training data remain the same, it's just RAG'ing real-time twitter data into its context window when returning results. It's like a worse version of Perplexity.
deleted 2y ago
[deleted]
ein0p 2y ago
Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).
- gundmc 2y ago
  It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.
- rfw300 2y ago
  o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.
  - ein0p 2y ago
    These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.
    - overfeed 2y ago
      > These small models only make sense with "thinking" enabled This entirely depends on your use-cases.
- vessenes 2y ago
  o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.
  - mccraveiro 2y ago
    2.5 flash with reasoning is just 20% cheaper than o4-mini
xbmcuser 2y ago
For a non programmer like me google is becoming shockingly good. It is giving working code the first time. I was playing around with it asked it to write code to scrape some data of a website to analyse. I was expecting it to write something that would scrape the data and later I would upload the data to it to analyse. But it actually wrote code that scraped and analysed the data. It was basic categorizing and counting of the data but I was not expecting it to do that.
- ant6n 2y ago
  Last time I tried Gemini, it messed with my google photo data plan and family sharing. I wish I could try the AI separate from my Google account.
  - jsnell 2y ago
    > I wish I could try the AI separate from my Google account. If that's a concern, just create another account. Doesn't even require using a separate browser profile, you can be logged into multiple accounts at once and use the account picker in the top right of most their apps to switch.
- kccqzy 2y ago
  That's the opposite experience of my wife who's in tech but also a non programmer. She wanted to ask Gemini to write code to do some basic data analysis things in a more automated way than Excel. More than once, Gemini wrote a long bash script where some sed invocations are just plain wrong. More than once I've had to debug Gemini-written bash scripts. As a programmer I knew how bash scripts aren't great for readability so I told my wife to ask Gemini to write Python. It resulted in higher code quality, but still contained bugs that are impossible for a non programmer to fix. Sometimes asking a follow up about the bugs would cause Gemini to fix it, but doing so repeatedly will result in Gemini forgetting what's being asked or simply throwing an internal error. Currently IMO you have to be a programmer to use Gemini to write programs effectively.
  - sbarre 2y ago
__alexs 2y ago
Does billing for the API actually work properly yet?
alecco 2y ago
Gemini models are very good but in my experience they tend to overdo the problems. When I give it things for context and something to rework, Gemini often reworks the problem. For software it is barely useful because you want small commits for specific fixes not a whole refactor/rewrite. I tried many prompts but it's hard. Even when I give it function signatures of the APIs the code I want to fix uses, Gemini rewrites the API functions. If anybody knows a prompt hack to avoid this, I'm all ears. Meanwhile I'm staying with Claude Pro.
- byearthithatius 2y ago
  Yes, it will add INSANE amounts of "robust error handling" to quick scripts where I can be confident about assumptions. This turns my clean 40 lines of Python where I KNOW the JSONL I am parsing is valid into 200+ lines filled with ten new try except statements. Even when I tell it not to do this, it loves to "find and help" in other ways. Quite annoying. But overall it is pretty dang good. It even spotted a bug I missed the other day in a big 400+ line complex data processing file.
  - zhengyi13 2y ago
    I wonder how much of that sort of thing is driven by having trained their models on their own internal codebases? Because if that's the case, careful and defensive being the default would be unsurprising.
  - stavros 2y ago
    I didn't realize this was a bigger trend, I asked it to write a simple testing script that POSTed a string to a local HTTP server as JSON, and it wrote a 40 line script, handling any possible error. I just wanted two lines.
    - jug 2y ago
      Yes, as late as earlier today, I asked it to provide "naive" code which helped a bit.
    - free_energy_min 2y ago
ks2048 2y ago
If this announcement is targeting people not up-to-date on the models available, I think they should say what "flash" means. Is there a "Gemini (non-flash)"? I see the 4 Google model names in the chart here. Are these 4 the main "families" of models to choose from? - Gemini-Pro-Preview - Gemini-Flash-Preview - Gemini-Flash - Gemini-Flash-Lite
- mwest217 2y ago
  Gemini has had 4 families of models, in order of decreasing size: - Ultra - Pro - Flash - Flash-Lite Versions with `-Preview` at the end haven't had their "official release" and are technically in some form of "early access" (though I'm not totally clear on exactly what that means given that they're fully available and as of 2.5 Pro Preview, have pricing attached to them - earlier versions were free during Preview but had pretty strict rate limiting but now it seems that Preview models are more or less fully usable).
  - drob518 2y ago
    Is GMail still in beta?
    - mring33621 2y ago
      so Sigma...
  - jsnell 2y ago
    The free-with-small-rate-limits designator was "experimental", not "preview". I think the distinction between preview and full release is that the preview models have no guarantees on how long they'll be available, the full release comes with a pre-set discontinuation date. So if want the stability for a production app, you wouldn't want to use a preview model.
deleted 2y ago
[deleted]
AStonesThrow 2y ago
I've been leveraging the services of 3 LLMs, mainly: Meta, Gemini, and Copilot. It depends on what I'm asking. If I'm looking for answers in the realm of history or culture, religion, or I want something creative such as a cute limerick, or a song or dramatic script, I'll ask Copilot. Currently, Copilot has two modes: "Quick Answer"; or "Think Deeply", if you want to wait about 30 seconds for a good answer. If I want info on a product, a business, an industry or a field of employment, or on education, technology, etc., I'll inquire of Gemini. Both Copilot and Gemini have interactive voice conversation modes. Thankfully, they will also write a transcript of what we said. They also eagerly attempt to engage the user with further questions and followups, with open questions such as "so what's on your mind tonight?" And if I want to know about pop stars, film actors, the social world or something related to tourism or recreation in general, I can ask Meta's AI through [Facebook] Messenger. One thing I found to be extremely helpful and accurate was Gemini's tax advice. I mean, it was way better than human beings at the entry/poverty level. Commercial tax advisors, even when I'd paid for the Premium Deluxe Tax Software from the Biggest Name, they just went to Google stuff for me. I mean, they didn't even seem to know where stuff was on irs.gov. When I asked for a virtual or phone appointment, they were no-shows, with a litany of excuses. I visited 3 offices in person; the first two were closed, and the third one basically served Navajos living off the reservation. So when I asked Gemini about tax information -- simple stuff like the terminology, definitions, categories of income, and things like that -- Gemini was perfectly capable of giving lucid answers. And citing its sources, so I could immediately go find the IRS.GOV publication and read it "from the horse's mouth". Oftentimes I'll ask an LLM just to jog my memory or inform me of what specific terminology I should use. Like "Hey Gemini, what's the PDU for Ethernet called?" and when Gemini says it's a "frame" then I have that search term I can plug into Wikipedia for further research. Or, for an introduction or overview to topics I'm unfamiliar with. LLMs are an important evolutionary step in the general-purpose "search engine" industry. One problem was, you see, that it was dangerous, annoying, or risky to go Googling around and click on all those tempting sites. Google knew this: the dot-com sites and all the SEO sites that surfaced to the top were traps, they were bait, they were sometimes legitimate scams. So the LLM providers are showing us that we can stay safe in a sandbox, without clicking external links, without coughing up information about our interests and setting cookies and revealing our IPv6 addresses: we can safely ask a local LLM, or an LLM in a trusted service provider, about whatever piques our fancy. And I am glad for this. I saw y'all complaining about how every search engine was worthless, and the Internet was clogged with blogspam, and there was no real information anymore. Well, perhaps LLMs, for now, are a safe space, a sandbox to play in, where I don't need to worry about drive-by-zero-click malware, or being inundated with Joomla ads, or popups. For now.
cynicalpeace 2y ago
1. The main transformative aspect of LLMs has been in writing code. 2. LLMs have had less transformative aspects in 2025 than we anticipated back in late 2022. 3. LLMs are unlikely to be very transformative to society, even as their intelligence increases, because intelligence is a minor changemaker in society. Bigger changemakers are motivation, courage, desire, taste, power, sex and hunger. 4. LLMs are unlikely to develop these more important traits because they are trained on text, not evolved in a rigamarole of ecological challenges.
charcircuit 2y ago
500 RPD for the free tier is good enough for my coding needs. Nice.
AbuAssar 2y ago
I noticed that OpenAI don't compare their models to third party models in their announcement posts, unlike google, meta and the others.
- jskherman 2y ago
  They're doing the Apple strategy. Less spotlight for other third parties, and less awareness how they're lagging behind so that those already ignorantly locked into OpenAI would not switch. But at this point why would anyone do that when switching costs are low?
mmaunder 2y ago
More great innovation from Google. OpenAI have two major problems. The first is Google's vertically integrated chip pipeline and deep supply chain and operational knowledge when it comes to creating AI chips and putting them into production. They have a massive cost advantage at every step. This translates into more free services, cheaper paid services, more capabilities due to more affordable compute, and far more growth. Second problem is data starvation and the unfair advantage that social media has when it comes to a source of continually refreshed knowledge. Now that the foundational model providers have churned through the common crawl and are competing to consume things like video and whatever is left, new data is becoming increasingly valuable as a differentiator, and more importantly, as a provider of sustained value for years to come. SamA has signaled both of these problems when he made noises about building a fab a while back and is more recently making noises about launching a social media platform off OpenAI. The smart money among his investors know these issues to be fundamental in deciding if OAI will succeed or not, and are asking the hard questions. If the only answer for both is "we'll build it from scratch", OpenAI is in very big trouble. And it seems that that is the best answer that SamA can come up with. I continue to believe that OpenAI will be the Netscape of the AI revolution. The win is Google's for the taking, if they can get out of their own way.
- jbverschoor 2y ago
  Except that they train their model even when you pay. So yeah.. I'd rather not use their "evil"
  - dayvigo 2y ago
    Source?
    - throwaway314155 2y ago
      It's right there in the comment.
  - mkl 2y ago
    This is false: https://ai.google.dev/gemini-api/terms https://ai.google.dev/gemini-api/terms
mark_l_watson 2y ago
Nice! Low price, even with reasoning enabled. I have been working on a short new book titled “Practical AI with Google: A Solo Knowledge Worker's Guide to Gemini, AI Studio, and LLM APIs” but with all of Google’s recent announcements it might not be a short book.
serjester 2y ago
Just ran it on one of our internal PDF (3 pages, medium difficulty) to json benchmarks: gemini-flash-2.0: 60 ish% accuracy 6,250 pages per dollar gemini-2.5-flash-preview (no thinking): 80 ish% accuracy 1,700 pages per dollar gemini-2.5-flash-preview (with thinking): 80 ish% accuracy (not sure what's going on here) 350 pages per dollar gemini-flash-2.5: 90 ish% accuracy 150 pages per dollar I do wish they separated the thinking variant from the regular one - it's incredibly confusing when a model parameter dramatically impacts pricing.
- ValveFan6969 2y ago
  I have been having similar performance issues, I believe they intentionally made a worse model (Gemini 2.5) to get more money out of you. However, there is a way where you can make money off of Gemini 2.5. If you set the thinking parameter lower and lower, you can make the model spew absolute nonsense for the first response. It costs 10 cents per input / output, and sometimes you get a response that was just so bad your clients will ask for more and more corrections.
  - mpalmer 2y ago
    Wow, what apps have you made so I know never to use them?
zoogeny 2y ago
Google making Gemini 2.5 Pro (Experimental) free was a big deal. I haven't tried the more expensive OpenAI models so I can't even compare, only to the free models I have used of theirs in the past. Gemini 2.5 Pro is so much of a step up (IME) that I've become sold on Google's models in general. It not only is smarter than me on most of the subjects I engage with it, it also isn't completely obsequious. The model pushes back on me rather than contorting itself to find a way to agree. 100% of my casual AI usage is now in Gemini and I look forward to asking it questions on deep topics because it consistently provides me with insight. I am building new tools with the mind to optimize my usage to increase it's value to me.
- PerusingAround 2y ago
  This comment is exactly my experience, I feel like as if I had wrote it myself.
- cjohnson318 2y ago
  Yeah, my wife pays for ChatGPT, but Gemini is fine enough for me.
  - qwertox 2y ago
    Just be aware that if you don't add a key (and set up billing) youre granting Google the right to train on your data. To have persons read them and decide how to use them for training.
    - energy123 2y ago
      I thought if you turn off App Activity then that's good enough to protect your data?
      - voxic11 2y ago
        Nope, not if you are in the US https://ai.google.dev/gemini-api/terms#data-use-unpaid https://ai.google.dev/gemini-api/terms#data-use-unpaid
minimaxir 2y ago
One hidden note from Gemini 2.5 Flash when diving deep into the documentation: for image inputs, not only can the model be instructed to generated 2D bounding boxes of relevant subjects, but it can also create segmentation masks! https://ai.google.dev/gemini-api/docs/image-understanding#segmentation https://ai.google.dev/gemini-api/docs/image-understanding#se... At this price point with the Flash model, creating segmentation masks is pretty nifty. The segmentation masks are a bit of a galaxy brain implementation by generating a b64 string representing the mask: https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Spatial_understanding.ipynb#scrollTo=WQJTJ8wdGOKx https://colab.research.google.com/github/google-gemini/cookb... I am trying to test it in AI Studio but it sometimes errors out, likely because it tries to decode the b64 lol.
- behnamoh 2y ago
  Wait, did they just kill YOLO, at least for time-insensitive tasks?
  - minimaxir 2y ago
    YOLO is probably still cheaper if bounding boxes are your main goal. Good segmentation models that work for arbitrary labels, however, are much more expensive to set up and run, so this type of approach could be an interesting alternative depending on performance.
  - daemonologist 2y ago
    No, the speed of YOLO/DETR inference makes it cheap as well - probably at least five or six orders of magnitude cheaper. Edit: After some experimentation, Gemini also seems to not perform nearly as well as a purpose-tuned detection model. It'll be interesting to test this capability and see how it evolves though. At some point you might be able use it as a "teacher" to generate training data for new tasks.
  - vunderba 2y ago
    Well no. You can run/host YOLO which means not having to submit potentially sensitive information to a company that generates a large amount of revenue from targeted advertising.
simonw 2y ago
I spotted something interesting in the Python API library code: https://github.com/googleapis/python-genai/blob/473bf4b6b5a69e5324a5d4bac0fe852351338c43/google/genai/types.py#L2070-L2082 https://github.com/googleapis/python-genai/blob/473bf4b6b5a6... class ThinkingConfig(_common.BaseModel): """The thinking features configuration.""" include_thoughts: Optional[bool] = Field( default=None, description="""Indicates whether to include thoughts in the response. If true, thoughts are returned only if the model supports thought and thoughts are available. """, ) thinking_budget: Optional[int] = Field( default=None, description="""Indicates the thinking budget in tokens. """, ) That thinking_budget thing is documented, but what's the deal with include_thoughts? It sounds like it's an option to have the API return the thought summary... but I can't figure out how to get it to work, and I've not found documentation or example code that uses it. Anyone managed to get Gemini to spit out thought summaries in its API using this option?
- phillypham 2y ago
  They removed the docs and support for it https://github.com/googleapis/python-genai/commit/af3b339a9d58e25cb3baa6fd3d0b1d078c620f9d https://github.com/googleapis/python-genai/commit/af3b339a9d.... You can see the thoughts in AI Studio UI as per https://ai.google.dev/gemini-api/docs/thinking#debugging-and-steering https://ai.google.dev/gemini-api/docs/thinking#debugging-and....
- lemming 2y ago
  I maintain an alternative client which I build from the API definitions at https://github.com/googleapis/googleapis https://github.com/googleapis/googleapis, which according to https://github.com/googleapis/python-genai/issues/345 https://github.com/googleapis/python-genai/issues/345 should be the right place. But neither the AI Studio nor the Vertex definitions even have ThinkingConfig yet - very frustrating. In general it's amazing how much API munging is required to get a working client from the public API definitions.
- qwertox 2y ago
  In AI Studio the flash moddels has two toggles: Enable thinking and Set thinking budget. If thinking budget is enabled, you can set tue max number of tokens it can use to think, else it's Auto.
deanmoriarty 2y ago
Genuine naive question: when it comes to Google HN has generally a negative view of it (pick any random story on Chrome, ads, search, web, working at faang, etc. and this should be obvious from the comments), yet when it comes to AI there is a somewhat notable “cheering effect” for Google to win the AI race that goes beyond a conventional appreciation of a healthy competitive landscape, which may appear as a bit of a double standard. Why is this? Is it because OpenAI is seen as such a negative player in this ecosystem that Google “gets a pass on this one”? And bonus question: what do people think will happen to OpenAI if Google wins the race? Do you think they’ll literally just go bust?
- antirez 2y ago
  Maybe because Google is largely responsible, paying for the research, of most of the results we are seeing now. I'm not a Google fan, in the web side, and in their idea of what software engineering is, but they deserve to win the AI race, because right now all the other players provided a lot less than what Google did as public research. Also, with Gemini 2.5 PRO, there was a big hype moment, because the model is of unseen ability.
  - wkat4242 2y ago
    Maybe they deserve it but it would be really bad for the world. Because they will enshittify the hell out of it once they're established. That's their MO. I don't want Google to have a stranglehold over yet another type of online service. So I avoid them. And things are going so fast now, whatever Google has today that might be better than the rest, in two months the rest will have it too. Of course Google will have something new again. But being 2 months behind isn't a huge deal. I don't have to have the 'winning' product. In fact most of my AI tasks go to an 8b llama 3.1 model. It's about on par with gpt 3.5 but that's fine.
    - visarga 2y ago
      The situation with LLMs is much different than search, Google doesn't have such a large lead here. LLMs are social things, they learn from each other, any provider with SOTA model will see its abilities leaked through synthetic training data. That's what GPT-4 did for a year, against the wishes of OpenAI, powering up millions of open model finetunes.
krembo 2y ago
How is this sustainable for Google from business POV? It feels like Google is shooting itself in the foot while "winning" the AI race.. From my experience I think Google lost 99% of the ads it used to show me before in the search engine.
- tomr75 2y ago
  someone else will do it if they don't
- aoeusnth1 2y ago
  Their inference costs are the lowest in the business.
zenGull 2y ago
I've been paying for googles pro llm for about six months. At 20 it feels steep considering the free version is very good. I'm a devops work, and it's been very helpful. Ive tried gpt, copilot, Mixtral, Claude, etc and Geminis 1.5 pro was what sold me. The new 2.0 stuff is even better. Anecdotally, Gemini seems to forget to add stuff but doesn't hallucinate as much. I've been doing some pretty complex scripting this last week purely on Gemini fast 2.0 and it's been really really good.
jdthedisciple 2y ago
Very excited to try it, but it is noteworthy that o4-mini is strictly better according to the very benchmarks shown by Google here. Of course it's about 4x as expensive too (I believe), but still, given the release of openai/codex as well, o4-mini will remain a strong competitor for now.
thimabi 2y ago
I find it baffling that Google offers such impressive models through the API and even the free AI Studio with fine-grained control, yet the models used in the Gemini app feel much worse. Over the past few weeks, I’ve been using Gemini Advanced on my Workspace account. There, the models think for shorter times, provide shorter outputs, and even their context window is far from the advertised 1 million tokens. It makes me think that Google is intentionally limiting the Gemini app. Perhaps the goal is to steer users toward the API or AI Studio, with the free tier that involves data collection for training purposes.
- Alifatisk 2y ago
  Google lack marketing for ai studio, it has only recently become widely known through word of mouth
  - thimabi 2y ago
    That does work in Google’s favor. Users who are technical enough to want a better model eventually learn about AI Studio, while the rest are none the wiser.
- _delirium 2y ago
  This might have changed after you posted your comment, but it looks like 2.5 Pro and 2.5 Flash are available in the Gemini app now, both web and mobile.
  - thimabi 2y ago
    Oh, I didn’t mean to say that these models were unavailable through the app or website. Rather, I’ve realized that using them through the API or AI Studio yields much better results — even in the free tier. You can check that by trying prompts with complex instructions and long inputs/outputs. For instance, ask Gemini to generate notes from a specific source (say, a book or class transcription). Or ask it to translate a long article, full of idiomatic expressions, while maintaining high fidelity to the source. You will see that the very same Gemini models are underutilized on the app or the website, while their performance is stellar on the API or AI Studio.
bingdig 2y ago
It appears that this impacted gemini-2.5-pro-preview-03-25 somehow? grounding with google search no longer works. I had a workflow running that would pull news articles from the past 24 hours. It now refuses to believe the current date is 2025-04-17. Even with search turned on and I ask it what the date is it and it always replies sometime in July 2024.
Alifatisk 2y ago
No matter how good the new Gemini models have become, my bad experience with early Gemini is still stuck with me and I am afraid I still suffer from confirmation bias. Whenever I just look at the Gemini app, I already assume it’s going to be a bad experience.
thallavajhula 2y ago
At this point, at the current pace of AI model development, I feel like I can't tell which one is better. I usually end up using multiple LLMs to get a task done to my taste. They're all equally good and bad. It's like using GCP vs AWS vs Azure all over again, except in the AI space.
simonw 2y ago
An often overlooked feature of the Gemini models is that they can write and execute Python code directly via their API. My llm-gemini plugin supports that: https://github.com/simonw/llm-gemini https://github.com/simonw/llm-gemini uv tool install llm llm install llm-gemini llm keys set gemini # paste key here llm -m gemini-2.5-flash-preview-04-17 \ -o code_excution 1 \ 'render a mandelbrot fractal in ascii art' I ran that just now and got this: https://gist.github.com/simonw/cb431005c0e0535343d6977a7c47050c https://gist.github.com/simonw/cb431005c0e0535343d6977a7c470... They don't charge anything extra for code execution, you just pay for input and output tokens. The above example used 10 input, 1,531 output which is $0.15/million for input and $3.50/million output for Gemini 2.5 Flash with thinking enabled, so 0.536 cents (just over half a cent) for this prompt.
- blahgeek 2y ago
  > An often overlooked feature of the Gemini models is that they can write and execute Python code directly via their API. Could you elaborate? I thought function calling is a common feature among models from different providers
  - WiSaGaN 2y ago
    This common feature requires the user of the API to implement the tool, in this case, the user is responsible to run the code the API outputs. The post you replied suggests that Gemini will run the code for the user behind the API call.
    - tempoponet 2y ago
      That was how I read it as well, as if it had a built-in lambda type service in the cloud. If we're just talking about some API support to call python scripts, that's pretty basic to wire up with any model that supports tool use.
  - simonw 2y ago
    The Gemini API runs the Python code for you as part of your single API call, without you having to handle the tool call request yourself.
lleymrl651 2y ago
good
djrj477dhsnv 2y ago
Why are most comments here only comparing to Claude and just a few to ChatGPT and none to Grok? Grok 3 has been my main LLM since its release. Is it not as good as I thought it was?
- jofzar 2y ago
  IMO I will not use Grok while it's owned and related to Elon, not only do I not trust their privacy and data usage (not that I "really" trust open AI/Google etc) I just despise him. It would have to be very significantly better for me to use it.
- dyauspitr 2y ago
  Grok just isn’t the best out there.
WiSaGaN 2y ago
Interesting that the output price per 1M tokens is $0.6 for non-reasoning, but $3.5 for reasoning. This seems to defy common assumption of how reasoning models work, and you tweak the <think> token probability to control how much thinking it does, but underlying it's the same model and the same inference code path.
michaelbrave 2y ago
Yesterday I started working through How to design programs, and set up a chat with Gemini 2.5 asking it to be my tutor as I go through it and to help answer my questions if I don't understand a part of the book. It has been knowledgeable, helpful and capable of breaking down complex things that I couldn't understand into understandable things. Fantastic all around.
throwawaytodey 2y ago
[dead]
zenkey 2y ago
Google is totally back in the game now, but it’s still going to take a lot more for them at this point to overcome OpenAI’s “first‑mover advantage” (clearly the favorite among younger users atm).
- sweca 2y ago
  Google Pixel marketing is doing wonders for Gemini in young populations. I have been seeing a lot more of their phones in my generation's hands.
uptownfunk 2y ago
[flagged]
sinuhe69 2y ago
I’m not familiar with Python internals, so when I tried to convert a public AI model (not a LLM) to run locally, I got some problems no other AI could help. Asked Gemini 2.5 and it pin pointed the problem immediately. It solution was not practical but I guess it also works.
ashu1461 2y ago
One place where I feel gemini models lag is function calling and predicting correct arguments to function calls, is there a benchmark which scores models on the basis of this ?
upmind 2y ago
It's a shame that Gemini doesn't seem to have as much hype as GPT, I hope they gain more market share.
menshiki 2y ago
As a person mostly using AI for everyday tasks and business-related research, it's very impressive how quickly they've progressed. I would consider all models before 2.0 totally unusable. Their web interface, however, is so much worse than that of the ChatGPT macOS app.
- gcbirzan 2y ago
  Some aren't even at 2.0, and the version numbers aren't related in any way to their... generation? Also, what is so good about the ChatGPT app, specifically on macOS that makes it better?
convivialdingo 2y ago
Dang - Google finally made a quality model that doesn’t make me want to throw my computer out a window. It’s honest, neutral and clearly not trained by the ideologically rabid anti-bias but actually super biased regime. Did I miss a revolt or something in googley land? A Google model saying “free speech is valuable and diverse opinions are good” is frankly bizarre to see.
- convivialdingo 2y ago
  Downvote me all you want - the fact remains that previous Google models were so riddled with guardrails and political correctness that it was practically impossible to use for anything besides code and clean business data. Random text and opinion would trigger a filter and shut down output. Even this model criticizes the failures of the previous models.
  - tempaccount420 2y ago
    Yes, something definitely changed. It's still a little biased, it's kind of like OpenAI before Trump became president.
camkego 2y ago
The pricing table image in the article really should have included Gemini 2.5 pro. Sure, it could be after Flash to the right, but it would help people understand the price performance benefits of 2.5 Flash.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
wanderr 2y ago
Gemini has the annoying habit of delegating tasks to me. Most recently I was trying to find out how to do something in FastRawViewer that I couldn't find a straightforward answer on. After hallucinating a bunch of settings and menus that don't exist, it told me to read the manual and check the user forums. So much for saving me time.
131hn 2y ago
If OpenAI offers Codex and Anthropic offers Claude Code, is there a CLI integration that Google recommends for using Gemini 2.5? That’s what’s keeping me, for now, with the other two.
hubraumhugo 2y ago
You can get your HN profile analyzed and roasted by it. It's pretty funny :) https://hn-wrapped.kadoa.com/ https://hn-wrapped.kadoa.com/ I'll add a selection for different models soon.
- CoolCold 2y ago
  Error An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance which may provide additional details about the nature of the error.
  - hubraumhugo 2y ago
    fixed!
- demaga 2y ago
  Didn't expect to be roasted by AI this morning. Nice one
- Alifatisk 2y ago
  How is this relevant to Gemini 2.5 Flash? I guess it's using it or something?
  - deleted 2y ago
    [deleted]
- few 2y ago
  This is cool. Does it only use a few recent comments or entire history? I'm trying to figure out where it figured out my city when I thought I was careful not to reveal it. I'm scrolling back pages without finding where I said it in the past. Could it have inferred it based on other information or hallucinated it? I wonder if there's a more opsec-focused version of this.
- x187463 2y ago
  Personal Projects Will finally implement that gravity in TTE, despite vowing not to. We all know how well developers keep promises. Knowledge Growth Will achieve enlightenment on the true meaning of 'enshittification', likely after attempting to watch a single YouTube video without Premium. I found these actually funny. Cool project.
yawaramin 2y ago
I just asked "why is 'Good Friday' so called?" and it got stuck. Flash 2.0 worked though.
uninformed-me00 2y ago
I want to think that this is all great, but the fact that this is also one of the best way to collect unsuspecting user data by default without explicit consent just doesn't feel right -- that applies to most people who would never have a chance reading this comment. I don't want to be angry but screw these default opt-in to have your privacy violated free stuff. Before you jump in to say you can pay to keep your privacy, stop and read again.
sgt 2y ago
How are they able to remain so competitive and will it last? The pricing almost seems too good to be true in terms of what they claim you get.
- sweca 2y ago
  Custom TPUs ftw
egorfine 2y ago
I am always overlooking anything Google due to the fact that they are the opposite of "Don't be evil" and because their developer's console (Google Cloud) is incredibly hostile to humans. Today I reluctantly clicked on their "AI Studio" link in the press-release and I was pleasantly surprised to discover that AI Studio has nothing in common with their typical UI/UX. It's nice and I love it!
- brap 2y ago
  To be fair the UX of all GCP/AWS/Azure is ass. If you don’t know exactly what you’re looking for, good luck navigating that mess.
techwiz137 2y ago
I had a heart attack moment thinking they were bringing some form of Adobe Flash back.
deleted 2y ago
[deleted]
latemedium 2y ago
There's an important difference between Gemini and Claude that I'm not sure how to quantify. I often use shell-connected LLMs (LLMs with a shell tool enabled) to take care of basic CSV munging / file-sorting tasks for me - I work in data science so there's a lot of this. When I ask Claude to do something, it carefully looks at all the directories and files before doing anything. Gemini, on the other hand, blindly jumps in and just starts moving stuff around. Claude executes more tools and is a little slower, but it almost always gets the right answer because it appropriately gathers the right context before really trying to solve the problem. Gemini doesn't seem to do this at all, but it makes a world of difference for my set of problems. Curious to see if others have had the same experience or if its just a quirk of my particular set of tasks
- energy123 2y ago
  What's a shell connected LLM and how to do that?
  - kmacdough 2y ago
    Look up Claude Code, Cursor, Aider and VSCode's agent integration. Generally, tools to use AI more actively for development. There are others as well. Plenty of info around. Here's not the place for a tutorial.
- Alifatisk 2y ago
  Claude has always been the best at coding, no matter what all the benchmarks says, the people have spoken and the consensus is that Claude is the best.
rjurney 2y ago
I am building a knowledge graph using BAML [baml-py] to extract documents [it's opinionated towards docs] and then PySpark to ETL the data into a node / edge list. GPT4o got few relations... Gemini 2.5 got so many it was nuts, all accurate but not all from the article! I had to reign it in and instruct it not to build so vast a graph. Really cool, it knows a LOT about semiconductors :)
profsummergig 2y ago
I tried this prompt in both Gemini 2.5 Pro, and in ChatGPT. "Draw me a timeline of all the dynasties of China. Imagine a horizontal line. Start from the leftmost point and draw segments for the start and end of each dynasty. For periods where multiple dynasties existed simultaneously draw parallel lines or boxes to represent the concurrent rule." Gemini's response: "I'm just a language model, so I can't help you with that." ChatGPT's response: an actual visual timeline.
- renewiltord 2y ago
  All the communities where people think LLMs are junk love Gemini. Makes me sceptical that the enthusiasm is useful signal. I found the full 2.0 useful for transcription of images. Very good OCR. But not a good assistant. Stalls often and once it has, loses context easily.
  - thegeomaster 2y ago
    Is it possible that a community of people who are constantly pushing LLMs to their limits would be most aware of their limitations, and so more inclined to think they are junk? In terms of business utility, Google has had great releases ever since the 2.0 family. Their models have never missed some mark --- either a good price/performance ratio, insane speeds, novel modalities (they still have the only API for autoregressive image generation atm), state-of-the-art long context support and coding ability (Gemini 2.5), etc. However, most average users are using these models through a chat-like UI, or via generic tools like Cursor, which don't really optimize their pipelines to capture the strengths of different models. This way, it's very difficult to judge a model objectively. Just look at the obscene sycophancy exhibited by chatgpt-4o-latest and how it lifted LMArena scores.
    - renewiltord 2y ago
      Just the fact that everyone on HN is always telling us how LLMs are useless but that Gemini is the best of them convinces me of the opposite. No one who can't find a use for this technology is really informed on the subject. Hard to take them seriously.
asim 2y ago
I just wish the whole industry would stop using terms like thinking and reasoning. This is not what's happening. If we could come up with more appropriate terms that don't treat these models like they're human then we'd be in a much better place. That aside, it's cool to see the advancement of Google's offering.
- dbbk 2y ago
  Thinking perhaps, but why not reasoning?
- lyu07282 2y ago
  Do you think any machine will ever be able to think and/or reason? Or is that a uniquely human thing? and do you have a rational standard to judge when something is reasoning or thinking, or just vibes? I'm asking because I wonder how much of that common attitude is just a sort of species-chauvinism. You are feeling anxious because machines getting smarter, you are feeling anger because "they" are taking your job away, but the machine doesn't do that, its people with an ideology that do that, you should be angry at that instead.
aerhardt 2y ago
I am only on OpenAI because they have a native Mac app. Call me old-school but my preferred workflow is still for the most part just asking narrow questions and copying-pasting back and forth. I've been playing with Junie (Jetbrain's AI agent) for a couple of days, but I still don't trust agents to run loose in my codebase for any sizeable amount of work. Does anyone know if Google is planning native apps? Or any wrapping interfaces that work well on a Mac?
- sweca 2y ago
  Raycast[0] has Gemini support in their AI offering and it's native, fast and intuitive. [0] https://raycast.com/ai https://raycast.com/ai
sweca 2y ago
Honestly, the best part about Gemini, especially as a consumer product, is their super lax, or lack thereof, ratelimits. They never have capacity issues, unlike Claude which always feels slow or sometimes outright rejects requests during peak hours. Gemini is constantly speedy and has extremely generous context window limits on the Gemini apps.
- onlyrealcuzzo 2y ago
  Interesting. I use Claude quite a bit, and haven't encountered this. Is this the free version of Claude or the paid version? When are peak hours typically (in what timezone)?
  - sweca 2y ago
    I have Claude Pro and peak hours are in the afternoon and at night for me in EST
bossyTeacher 2y ago
Is everyone on here solely evaluating the models on their programming capabilities? I understand this is HN but vibe coding LLM tools won't be able to sustain the LLM industry (let's not call it AI please)
barfingclouds 2y ago
I just need the Gemini app to allow push to talk :( Otherwise it’s not usable for me in the way I want it to be

1076 points by meetpateltech 2y ago

I don't think it is as good as deepseek v3 for updating coding tasks. I use 2.5 pro as architect with deepseek v3 as editor engineer. This combination work almost so perfect. Flash has advantages of great context size though. gpt 4.1 is also great.

xnx 2y ago

50% price increase from Gemini 2.0 Flash. That sounds like a lot, but Flash is still so cheap when compared to other models of this (or lesser) quality. https://developers.googleblog.com/en/start-building-with-gemini-25-flash/ https://developers.googleblog.com/en/start-building-with-gem...

akudha 2y ago
Is this cheaper than DeepSeek? Am I reading this right?
- vdfs 2y ago
  Only if you don't use reasoning
Tiberium 2y ago
del
- Havoc 2y ago
  You may want to consult Gemini on those percentage calcs .10 to .15 is not 25%
swyx 2y ago
done pretty much inline with the price elo pareto frontier https://x.com/swyx/status/1912959140743586206/photo/1 https://x.com/swyx/status/1912959140743586206/photo/1
- xnx 2y ago
  Love that chart! Am I imagining that I saw a version of that somewhere that even showed how the boundary has moved out over time?
  - swyx 2y ago
    https://x.com/swyx/status/1882933368444309723 https://x.com/swyx/status/1882933368444309723 https://x.com/swyx/status/1830866865884991999 https://x.com/swyx/status/1830866865884991999 (scroll up)
- oezi 2y ago
  So if I see it right flash 2.5 doesn't push the pareto front forward, right? It just sits between 2.5 pro and 2.0 flash. https://storage.googleapis.com/gweb-developer-goog-blog-assets/images/Gemini-2.5-Flash-price-to-performance-compariso.original.png https://storage.googleapis.com/gweb-developer-goog-blog-asse...
  - swyx 2y ago
    yeah but 1) its useful to have the point there on the curve if you need it, 2) intelligence is multidimensional, maybe in 2.5 flash you get qualitatively a better set of capabilities for your needs than 2.5 pro
  - jsnell 2y ago
    It does, that point in the tradeoff space was not available until now. Any model that's not dominated by at least one model on both axes will push forward the frontier. (The actual frontier isn't actually a straight line between the points on the frontier like visualized there. It's a step function.)
onlyrealcuzzo 2y ago
Why isn't Phi-3, Llama 3, or Mistral in the comparison? Aren't there a lot of hosted options? How do they compare in terms of cost?

byefruit 2y ago

It's interesting that there's a price nearly 6x price difference between reasoning and no reasoning. This implies it's not a hybrid model that can just skip reasoning steps if requested. Anyone know what else they might be doing? Reasoning means contexts will be longer (for thinking tokens) and there's an increase in cost to inference with a longer context but it's not going to be 6x. Or is it just market pricing?

vineyardmike 2y ago
Based on their graph, it does look explicitly priced along their “Pareto Frontier” curve. I’m guessing that is guiding the price more than their underlying costs. It’s smart because it gives them room to drop prices later and compete once other company actually get to a similar quality.
jsnell 2y ago
> This implies it's not a hybrid model that can just skip reasoning steps if requested. It clearly is, since most of the post is dedicated to the tunability (both manual and automatic) of the reasoning budget. I don't know what they're doing with this pricing, and the blog post does not do a good job explaining. Could it be that they're not counting thinking tokens as output tokens (since you don't get access to the full thinking trace anyway), and this is the basically amortizing the thinking tokens spend over the actual output tokens? Doesn't make sense either, because then the user has no incentive to use anything except 0/max thinking budgets.
deleted 2y ago
[deleted]
RobinL 2y ago
Does anyone know how this pricing works? Supposing I have a classification prompt where I need the response to be a binary yes/no. I need one token of output, but reasoning will obviously add far more than 6 additional tokens. Is it still a 6x price multiplier? That doesn't seem to make sense, but not does paying 6x more for every token including reasoning ones
- 2y ago

punkpeye 2y ago

This is cool, but rate limits on all of these preview models are PITA

Layvier 2y ago
Agreed, it's not even possible to run an eval dataset. If someone from google see this please at least increase the burst rate limit
- punkpeye 2y ago
  It is not without rate limits, but we do have elevated limits for our accounts through: https://glama.ai/models/gemini-2.5-flash-preview-04-17 https://glama.ai/models/gemini-2.5-flash-preview-04-17 So if you just want to run evals, that should do it. Though the first couple of days after a model comes out are usually pretty rough because everyone try to run their evals.
  - punkpeye 2y ago
    What I am noticing with every new Gemini model that comes out is that the time to first token (TTFT) is not great. I guess it is because they gradually transfer computer power from old models to new models as the demand increases.
    - Filligree 2y ago
      If you’re imagining that 2.5Pro gets dynamically loaded during the time to first token, then you’re vastly overestimating what’s physically possible. It’s more likely a latency-throughput tradeoff. Your query might get put inside a large batch, for example.
  - Layvier 2y ago
    That's very interesting, thanks for sharing!

arnaudsm 2y ago

Gemini flash models have the least hype, but in my experience in production have the best bang for the buck and multimodal tooling. Google is silently winning the AI race.

belter 2y ago
> Google is silently winning the AI race. That is what we keep hearing here...The last Gemini I cancelled the account, and can't help notice the new one they are offering for free...
- arnaudsm 2y ago
  Sorry I was talking of B2B APIs for my YC startup. Gemini is still far behind for consumers indeed.
  - JeremyNT 2y ago
    I use Gemini almost exclusively as a normal user. What am I missing out on that they are far behind on? It seems shockingly good and I've watched it get much better up to 2.5 Pro.
    - arnaudsm 2y ago
      Mostly brand recognition and the earlier Geminis had more refusals. As a consumer, I also really miss the Advanced voice mode of ChatGPT, which is the most transformative tech in my daily life. It's the only frontier model with true audio-to-audio.
      - wavewrangler 2y ago
        What do you mean miss? You don’t have the budget to keep something you truly miss for $20? What am in missing here / I don’t mean to criticize I am just curious is all. I would reword but I have to go
      - what_ever 2y ago

transformi 2y ago

Bad day is going on google. First the decleration of illegal monopoly.. and now... Google’s latest innovation: programmable overthinking. With Gemini 2.5 Flash, you too can now set a thinking_budget—because nothing says "state-of-the-art AI" like manually capping how long it’s allowed to reason. Truly the dream: debugging a production outage at 2am wondering if your LLM didn’t answer correctly because you cheaped out on tokens. lol. “Turn thinking off for better performance.” That’s not a model config, that’s a metaphor for Google’s entire AI strategy lately. At this point, Gemini isn’t an AI product—it’s a latency-cost-quality compromise simulator with a text interface. Meanwhile, OpenAI and Anthropic are out here just… cooking the benchmarks

danielbln 2y ago
Google's Gemini 2.5 pro model is incredibly strong, it's en par and at times better than Claude 3.7 in coding performance, being able to ingest entire videos into the context is something I haven seen elsewhere either. Google AI products have been anywhere between bad (Bard) to lackluster (Gemini 1.5), but 2.5 is a contender, in all dimensions. Google is also the only player that owns the entire stack, from research, software , data, compute hardware. I think they were slow to start but they've closed the gap since.
bsmith 2y ago
Using AI to debug code at 2am sounds like pure insanity.
- mring33621 2y ago
  the new normal
- spiderice 2y ago
  They're suggesting you'll be up at 2am debugging code because your AI code failed. Not that you'll be using AI to do the debugging.
SoftMachine 2y ago
[dead]

hmaxwell 2y ago

I did some testing this morning: Prompt: "can you find any mistakes on my codebase? I put one in there on purpose" + 70,000 tokens of codebase where in one line I have an include for a non-existent file. Claude 3.7 Sonnet (Try 1 - new chat) Claimed it Found a race condition in the api of the admin interface that would be triggered if two admins were to change the room order at the same time. Claude suggested I group all sql queries in a single transaction. I looked at the code and found that it already used a transaction for all queries. I said: the order_update api is already done with a transaction. Claude replied: "You're absolutely right, and I apologize for my mistake. I was incorrect to claim there was a race condition issue. The transaction ensures atomicity and consistency of the updates, and the SQL queries are properly structured for their intended purpose." Claude 3.7 Sonnet (Try 2 - new chat) Found a comment in the admin ui javascript code that suggested a potential alternative to event handler cleanup that was not implemented because I decided to go with a cleaner route. Then asked "Is this the issue you intentionally included, or would you like me to look for other potential problems?" I said: "The comment merely suggests an alternative, right?" claude said: "Yes, you're absolutely right. The comment is merely suggesting an alternative approach that isn't being used in the code, rather than indicating a mistake. So there's no actual bug or mistake in this part of the code - just documentation of different possible approaches. I apologize for misinterpreting this as an issue!" Claude 3.7 Sonnet (Try 3 - new chat) When processing items out of the database to generate QR codes in the admin interface, Claude says that my code both attempts to generate QR codes with undefined data AS WELL AS saying that my error handling skips undefined data. Claude contradicts itself within 2 sentences. When asking about clarification Claude replies: Looking at the code more carefully, I see that the code actually has proper error handling. I incorrectly stated that it "still attempts to call generateQRCode()" in the first part of my analysis, which was wrong. The code properly handles the case when there's no data-room attribute. Gemnini Advanced 2.5 Pro (Try 1 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase. Gemnini Advanced 2.5 Pro (Try 2 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase. Gemnini Advanced 2.5 Pro (Try 3 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase. o4-mini-high and o4-mini and o3 and 4.5 and 4o - "The message you submitted was too long, please reload the conversation and submit something shorter."

Workaccount2 2y ago

OpenAI might win the college students but it looks like Google will lock in enterprise.

xnx 2y ago
ChatGPT seems to have a name recognition / first-mover advantage with college students now, but is there any reason to think that will stick when today's high school students are using Gemini on their Chromebooks?
gundmc 2y ago
Funny you should say that. Google just announced today that they are giving all college students one year of free Gemini advanced. I wonder how much that will actually move the needle among the youth.
- Workaccount2 2y ago
  My guess is that they will use it and still call it "ChatGPT"...
  - xnx 2y ago
    Chat Gemini Pretrained Transformer
  - tantalor 2y ago
    Pass the Kleenex. Can I get a Band-Aid? Here's a Sharpie. I need a Chapstick. Let me Xerox that. Toss me that Frisbee.
    - drob518 2y ago
      Exactly.
    - esafak 2y ago
      Do you prefer those brands or just use their names? I google stuff on Kagi...

statements 2y ago

Interesting to note that this might be the only model with knowledge cut off as recent as 2025 January

Tiberium 2y ago
Gemini 2.5 Pro has the same knowledge cutoff specified, but in reality on more niche topics it's still limited to ~middle of 2024.
brightball 2y ago
Isn't Grok 3 basically real time now?
- Tiberium 2y ago
  That's the web version (which has tools like search plugged in), other models in their official frontends (Gemini on gemini.google.com, GPT/o models on chatgpt.com) are also "real time". But when served over API, most of those models are just static.
- bearjaws 2y ago
  No LLM is real time, and in fact, even a 2025 cut off isn't entirely realistic. Without guidance to say, a new version of a framework it will frequently "reference" documentation from old versions and use that. It's somewhat real time when it searches the web, of course that data is getting populated into context rather than in training.
  - arizen 1y ago
    You can query Grok on recent news and it will deliver the very recent updates (1 day old or probably even less)
- jiocrag 2y ago
  Not at all. The model weights and training data remain the same, it's just RAG'ing real-time twitter data into its context window when returning results. It's like a worse version of Perplexity.

deleted 2y ago

[deleted]

ein0p 2y ago

Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).

gundmc 2y ago
It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.
rfw300 2y ago
o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.
- ein0p 2y ago
  These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.
  - overfeed 2y ago
    > These small models only make sense with "thinking" enabled This entirely depends on your use-cases.
vessenes 2y ago
o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.
- mccraveiro 2y ago
  2.5 flash with reasoning is just 20% cheaper than o4-mini

xbmcuser 2y ago

For a non programmer like me google is becoming shockingly good. It is giving working code the first time. I was playing around with it asked it to write code to scrape some data of a website to analyse. I was expecting it to write something that would scrape the data and later I would upload the data to it to analyse. But it actually wrote code that scraped and analysed the data. It was basic categorizing and counting of the data but I was not expecting it to do that.

ant6n 2y ago
Last time I tried Gemini, it messed with my google photo data plan and family sharing. I wish I could try the AI separate from my Google account.
- jsnell 2y ago
  > I wish I could try the AI separate from my Google account. If that's a concern, just create another account. Doesn't even require using a separate browser profile, you can be logged into multiple accounts at once and use the account picker in the top right of most their apps to switch.
kccqzy 2y ago
That's the opposite experience of my wife who's in tech but also a non programmer. She wanted to ask Gemini to write code to do some basic data analysis things in a more automated way than Excel. More than once, Gemini wrote a long bash script where some sed invocations are just plain wrong. More than once I've had to debug Gemini-written bash scripts. As a programmer I knew how bash scripts aren't great for readability so I told my wife to ask Gemini to write Python. It resulted in higher code quality, but still contained bugs that are impossible for a non programmer to fix. Sometimes asking a follow up about the bugs would cause Gemini to fix it, but doing so repeatedly will result in Gemini forgetting what's being asked or simply throwing an internal error. Currently IMO you have to be a programmer to use Gemini to write programs effectively.
- sbarre 2y ago

__alexs 2y ago

Does billing for the API actually work properly yet?

alecco 2y ago

Gemini models are very good but in my experience they tend to overdo the problems. When I give it things for context and something to rework, Gemini often reworks the problem. For software it is barely useful because you want small commits for specific fixes not a whole refactor/rewrite. I tried many prompts but it's hard. Even when I give it function signatures of the APIs the code I want to fix uses, Gemini rewrites the API functions. If anybody knows a prompt hack to avoid this, I'm all ears. Meanwhile I'm staying with Claude Pro.

byearthithatius 2y ago
Yes, it will add INSANE amounts of "robust error handling" to quick scripts where I can be confident about assumptions. This turns my clean 40 lines of Python where I KNOW the JSONL I am parsing is valid into 200+ lines filled with ten new try except statements. Even when I tell it not to do this, it loves to "find and help" in other ways. Quite annoying. But overall it is pretty dang good. It even spotted a bug I missed the other day in a big 400+ line complex data processing file.
- zhengyi13 2y ago
  I wonder how much of that sort of thing is driven by having trained their models on their own internal codebases? Because if that's the case, careful and defensive being the default would be unsurprising.
- stavros 2y ago
  I didn't realize this was a bigger trend, I asked it to write a simple testing script that POSTed a string to a local HTTP server as JSON, and it wrote a 40 line script, handling any possible error. I just wanted two lines.
  - jug 2y ago
    Yes, as late as earlier today, I asked it to provide "naive" code which helped a bit.
  - free_energy_min 2y ago

ks2048 2y ago

If this announcement is targeting people not up-to-date on the models available, I think they should say what "flash" means. Is there a "Gemini (non-flash)"? I see the 4 Google model names in the chart here. Are these 4 the main "families" of models to choose from? - Gemini-Pro-Preview - Gemini-Flash-Preview - Gemini-Flash - Gemini-Flash-Lite

mwest217 2y ago
Gemini has had 4 families of models, in order of decreasing size: - Ultra - Pro - Flash - Flash-Lite Versions with `-Preview` at the end haven't had their "official release" and are technically in some form of "early access" (though I'm not totally clear on exactly what that means given that they're fully available and as of 2.5 Pro Preview, have pricing attached to them - earlier versions were free during Preview but had pretty strict rate limiting but now it seems that Preview models are more or less fully usable).
- drob518 2y ago
  Is GMail still in beta?
  - mring33621 2y ago
    so Sigma...
- jsnell 2y ago
  The free-with-small-rate-limits designator was "experimental", not "preview". I think the distinction between preview and full release is that the preview models have no guarantees on how long they'll be available, the full release comes with a pre-set discontinuation date. So if want the stability for a production app, you wouldn't want to use a preview model.

deleted 2y ago

[deleted]

AStonesThrow 2y ago

I've been leveraging the services of 3 LLMs, mainly: Meta, Gemini, and Copilot. It depends on what I'm asking. If I'm looking for answers in the realm of history or culture, religion, or I want something creative such as a cute limerick, or a song or dramatic script, I'll ask Copilot. Currently, Copilot has two modes: "Quick Answer"; or "Think Deeply", if you want to wait about 30 seconds for a good answer. If I want info on a product, a business, an industry or a field of employment, or on education, technology, etc., I'll inquire of Gemini. Both Copilot and Gemini have interactive voice conversation modes. Thankfully, they will also write a transcript of what we said. They also eagerly attempt to engage the user with further questions and followups, with open questions such as "so what's on your mind tonight?" And if I want to know about pop stars, film actors, the social world or something related to tourism or recreation in general, I can ask Meta's AI through [Facebook] Messenger. One thing I found to be extremely helpful and accurate was Gemini's tax advice. I mean, it was way better than human beings at the entry/poverty level. Commercial tax advisors, even when I'd paid for the Premium Deluxe Tax Software from the Biggest Name, they just went to Google stuff for me. I mean, they didn't even seem to know where stuff was on irs.gov. When I asked for a virtual or phone appointment, they were no-shows, with a litany of excuses. I visited 3 offices in person; the first two were closed, and the third one basically served Navajos living off the reservation. So when I asked Gemini about tax information -- simple stuff like the terminology, definitions, categories of income, and things like that -- Gemini was perfectly capable of giving lucid answers. And citing its sources, so I could immediately go find the IRS.GOV publication and read it "from the horse's mouth". Oftentimes I'll ask an LLM just to jog my memory or inform me of what specific terminology I should use. Like "Hey Gemini, what's the PDU for Ethernet called?" and when Gemini says it's a "frame" then I have that search term I can plug into Wikipedia for further research. Or, for an introduction or overview to topics I'm unfamiliar with. LLMs are an important evolutionary step in the general-purpose "search engine" industry. One problem was, you see, that it was dangerous, annoying, or risky to go Googling around and click on all those tempting sites. Google knew this: the dot-com sites and all the SEO sites that surfaced to the top were traps, they were bait, they were sometimes legitimate scams. So the LLM providers are showing us that we can stay safe in a sandbox, without clicking external links, without coughing up information about our interests and setting cookies and revealing our IPv6 addresses: we can safely ask a local LLM, or an LLM in a trusted service provider, about whatever piques our fancy. And I am glad for this. I saw y'all complaining about how every search engine was worthless, and the Internet was clogged with blogspam, and there was no real information anymore. Well, perhaps LLMs, for now, are a safe space, a sandbox to play in, where I don't need to worry about drive-by-zero-click malware, or being inundated with Joomla ads, or popups. For now.

cynicalpeace 2y ago

1. The main transformative aspect of LLMs has been in writing code. 2. LLMs have had less transformative aspects in 2025 than we anticipated back in late 2022. 3. LLMs are unlikely to be very transformative to society, even as their intelligence increases, because intelligence is a minor changemaker in society. Bigger changemakers are motivation, courage, desire, taste, power, sex and hunger. 4. LLMs are unlikely to develop these more important traits because they are trained on text, not evolved in a rigamarole of ecological challenges.

charcircuit 2y ago

500 RPD for the free tier is good enough for my coding needs. Nice.

AbuAssar 2y ago

I noticed that OpenAI don't compare their models to third party models in their announcement posts, unlike google, meta and the others.

jskherman 2y ago
They're doing the Apple strategy. Less spotlight for other third parties, and less awareness how they're lagging behind so that those already ignorantly locked into OpenAI would not switch. But at this point why would anyone do that when switching costs are low?

mmaunder 2y ago

More great innovation from Google. OpenAI have two major problems. The first is Google's vertically integrated chip pipeline and deep supply chain and operational knowledge when it comes to creating AI chips and putting them into production. They have a massive cost advantage at every step. This translates into more free services, cheaper paid services, more capabilities due to more affordable compute, and far more growth. Second problem is data starvation and the unfair advantage that social media has when it comes to a source of continually refreshed knowledge. Now that the foundational model providers have churned through the common crawl and are competing to consume things like video and whatever is left, new data is becoming increasingly valuable as a differentiator, and more importantly, as a provider of sustained value for years to come. SamA has signaled both of these problems when he made noises about building a fab a while back and is more recently making noises about launching a social media platform off OpenAI. The smart money among his investors know these issues to be fundamental in deciding if OAI will succeed or not, and are asking the hard questions. If the only answer for both is "we'll build it from scratch", OpenAI is in very big trouble. And it seems that that is the best answer that SamA can come up with. I continue to believe that OpenAI will be the Netscape of the AI revolution. The win is Google's for the taking, if they can get out of their own way.

jbverschoor 2y ago
Except that they train their model even when you pay. So yeah.. I'd rather not use their "evil"
- dayvigo 2y ago
  Source?
  - throwaway314155 2y ago
    It's right there in the comment.
- mkl 2y ago
  This is false: https://ai.google.dev/gemini-api/terms https://ai.google.dev/gemini-api/terms

mark_l_watson 2y ago

Nice! Low price, even with reasoning enabled. I have been working on a short new book titled “Practical AI with Google: A Solo Knowledge Worker's Guide to Gemini, AI Studio, and LLM APIs” but with all of Google’s recent announcements it might not be a short book.

serjester 2y ago

Just ran it on one of our internal PDF (3 pages, medium difficulty) to json benchmarks: gemini-flash-2.0: 60 ish% accuracy 6,250 pages per dollar gemini-2.5-flash-preview (no thinking): 80 ish% accuracy 1,700 pages per dollar gemini-2.5-flash-preview (with thinking): 80 ish% accuracy (not sure what's going on here) 350 pages per dollar gemini-flash-2.5: 90 ish% accuracy 150 pages per dollar I do wish they separated the thinking variant from the regular one - it's incredibly confusing when a model parameter dramatically impacts pricing.

ValveFan6969 2y ago
I have been having similar performance issues, I believe they intentionally made a worse model (Gemini 2.5) to get more money out of you. However, there is a way where you can make money off of Gemini 2.5. If you set the thinking parameter lower and lower, you can make the model spew absolute nonsense for the first response. It costs 10 cents per input / output, and sometimes you get a response that was just so bad your clients will ask for more and more corrections.
- mpalmer 2y ago
  Wow, what apps have you made so I know never to use them?

zoogeny 2y ago

Google making Gemini 2.5 Pro (Experimental) free was a big deal. I haven't tried the more expensive OpenAI models so I can't even compare, only to the free models I have used of theirs in the past. Gemini 2.5 Pro is so much of a step up (IME) that I've become sold on Google's models in general. It not only is smarter than me on most of the subjects I engage with it, it also isn't completely obsequious. The model pushes back on me rather than contorting itself to find a way to agree. 100% of my casual AI usage is now in Gemini and I look forward to asking it questions on deep topics because it consistently provides me with insight. I am building new tools with the mind to optimize my usage to increase it's value to me.

PerusingAround 2y ago
This comment is exactly my experience, I feel like as if I had wrote it myself.
cjohnson318 2y ago
Yeah, my wife pays for ChatGPT, but Gemini is fine enough for me.
- qwertox 2y ago
  Just be aware that if you don't add a key (and set up billing) youre granting Google the right to train on your data. To have persons read them and decide how to use them for training.
  - energy123 2y ago
    I thought if you turn off App Activity then that's good enough to protect your data?
    - voxic11 2y ago
      Nope, not if you are in the US https://ai.google.dev/gemini-api/terms#data-use-unpaid https://ai.google.dev/gemini-api/terms#data-use-unpaid

minimaxir 2y ago

One hidden note from Gemini 2.5 Flash when diving deep into the documentation: for image inputs, not only can the model be instructed to generated 2D bounding boxes of relevant subjects, but it can also create segmentation masks! https://ai.google.dev/gemini-api/docs/image-understanding#segmentation https://ai.google.dev/gemini-api/docs/image-understanding#se... At this price point with the Flash model, creating segmentation masks is pretty nifty. The segmentation masks are a bit of a galaxy brain implementation by generating a b64 string representing the mask: https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Spatial_understanding.ipynb#scrollTo=WQJTJ8wdGOKx https://colab.research.google.com/github/google-gemini/cookb... I am trying to test it in AI Studio but it sometimes errors out, likely because it tries to decode the b64 lol.

behnamoh 2y ago
Wait, did they just kill YOLO, at least for time-insensitive tasks?
- minimaxir 2y ago
  YOLO is probably still cheaper if bounding boxes are your main goal. Good segmentation models that work for arbitrary labels, however, are much more expensive to set up and run, so this type of approach could be an interesting alternative depending on performance.
- daemonologist 2y ago
  No, the speed of YOLO/DETR inference makes it cheap as well - probably at least five or six orders of magnitude cheaper. Edit: After some experimentation, Gemini also seems to not perform nearly as well as a purpose-tuned detection model. It'll be interesting to test this capability and see how it evolves though. At some point you might be able use it as a "teacher" to generate training data for new tasks.
- vunderba 2y ago
  Well no. You can run/host YOLO which means not having to submit potentially sensitive information to a company that generates a large amount of revenue from targeted advertising.

simonw 2y ago

I spotted something interesting in the Python API library code: https://github.com/googleapis/python-genai/blob/473bf4b6b5a69e5324a5d4bac0fe852351338c43/google/genai/types.py#L2070-L2082 https://github.com/googleapis/python-genai/blob/473bf4b6b5a6... class ThinkingConfig(_common.BaseModel): """The thinking features configuration.""" include_thoughts: Optional[bool] = Field( default=None, description="""Indicates whether to include thoughts in the response. If true, thoughts are returned only if the model supports thought and thoughts are available. """, ) thinking_budget: Optional[int] = Field( default=None, description="""Indicates the thinking budget in tokens. """, ) That thinking_budget thing is documented, but what's the deal with include_thoughts? It sounds like it's an option to have the API return the thought summary... but I can't figure out how to get it to work, and I've not found documentation or example code that uses it. Anyone managed to get Gemini to spit out thought summaries in its API using this option?

phillypham 2y ago
They removed the docs and support for it https://github.com/googleapis/python-genai/commit/af3b339a9d58e25cb3baa6fd3d0b1d078c620f9d https://github.com/googleapis/python-genai/commit/af3b339a9d.... You can see the thoughts in AI Studio UI as per https://ai.google.dev/gemini-api/docs/thinking#debugging-and-steering https://ai.google.dev/gemini-api/docs/thinking#debugging-and....
lemming 2y ago
I maintain an alternative client which I build from the API definitions at https://github.com/googleapis/googleapis https://github.com/googleapis/googleapis, which according to https://github.com/googleapis/python-genai/issues/345 https://github.com/googleapis/python-genai/issues/345 should be the right place. But neither the AI Studio nor the Vertex definitions even have ThinkingConfig yet - very frustrating. In general it's amazing how much API munging is required to get a working client from the public API definitions.
qwertox 2y ago
In AI Studio the flash moddels has two toggles: Enable thinking and Set thinking budget. If thinking budget is enabled, you can set tue max number of tokens it can use to think, else it's Auto.

deanmoriarty 2y ago

Genuine naive question: when it comes to Google HN has generally a negative view of it (pick any random story on Chrome, ads, search, web, working at faang, etc. and this should be obvious from the comments), yet when it comes to AI there is a somewhat notable “cheering effect” for Google to win the AI race that goes beyond a conventional appreciation of a healthy competitive landscape, which may appear as a bit of a double standard. Why is this? Is it because OpenAI is seen as such a negative player in this ecosystem that Google “gets a pass on this one”? And bonus question: what do people think will happen to OpenAI if Google wins the race? Do you think they’ll literally just go bust?

antirez 2y ago
Maybe because Google is largely responsible, paying for the research, of most of the results we are seeing now. I'm not a Google fan, in the web side, and in their idea of what software engineering is, but they deserve to win the AI race, because right now all the other players provided a lot less than what Google did as public research. Also, with Gemini 2.5 PRO, there was a big hype moment, because the model is of unseen ability.
- wkat4242 2y ago
  Maybe they deserve it but it would be really bad for the world. Because they will enshittify the hell out of it once they're established. That's their MO. I don't want Google to have a stranglehold over yet another type of online service. So I avoid them. And things are going so fast now, whatever Google has today that might be better than the rest, in two months the rest will have it too. Of course Google will have something new again. But being 2 months behind isn't a huge deal. I don't have to have the 'winning' product. In fact most of my AI tasks go to an 8b llama 3.1 model. It's about on par with gpt 3.5 but that's fine.
  - visarga 2y ago
    The situation with LLMs is much different than search, Google doesn't have such a large lead here. LLMs are social things, they learn from each other, any provider with SOTA model will see its abilities leaked through synthetic training data. That's what GPT-4 did for a year, against the wishes of OpenAI, powering up millions of open model finetunes.

krembo 2y ago

How is this sustainable for Google from business POV? It feels like Google is shooting itself in the foot while "winning" the AI race.. From my experience I think Google lost 99% of the ads it used to show me before in the search engine.

tomr75 2y ago
someone else will do it if they don't
aoeusnth1 2y ago
Their inference costs are the lowest in the business.

zenGull 2y ago

I've been paying for googles pro llm for about six months. At 20 it feels steep considering the free version is very good. I'm a devops work, and it's been very helpful. Ive tried gpt, copilot, Mixtral, Claude, etc and Geminis 1.5 pro was what sold me. The new 2.0 stuff is even better. Anecdotally, Gemini seems to forget to add stuff but doesn't hallucinate as much. I've been doing some pretty complex scripting this last week purely on Gemini fast 2.0 and it's been really really good.

jdthedisciple 2y ago

Very excited to try it, but it is noteworthy that o4-mini is strictly better according to the very benchmarks shown by Google here. Of course it's about 4x as expensive too (I believe), but still, given the release of openai/codex as well, o4-mini will remain a strong competitor for now.

thimabi 2y ago

I find it baffling that Google offers such impressive models through the API and even the free AI Studio with fine-grained control, yet the models used in the Gemini app feel much worse. Over the past few weeks, I’ve been using Gemini Advanced on my Workspace account. There, the models think for shorter times, provide shorter outputs, and even their context window is far from the advertised 1 million tokens. It makes me think that Google is intentionally limiting the Gemini app. Perhaps the goal is to steer users toward the API or AI Studio, with the free tier that involves data collection for training purposes.

Alifatisk 2y ago
Google lack marketing for ai studio, it has only recently become widely known through word of mouth
- thimabi 2y ago
  That does work in Google’s favor. Users who are technical enough to want a better model eventually learn about AI Studio, while the rest are none the wiser.
_delirium 2y ago
This might have changed after you posted your comment, but it looks like 2.5 Pro and 2.5 Flash are available in the Gemini app now, both web and mobile.
- thimabi 2y ago
  Oh, I didn’t mean to say that these models were unavailable through the app or website. Rather, I’ve realized that using them through the API or AI Studio yields much better results — even in the free tier. You can check that by trying prompts with complex instructions and long inputs/outputs. For instance, ask Gemini to generate notes from a specific source (say, a book or class transcription). Or ask it to translate a long article, full of idiomatic expressions, while maintaining high fidelity to the source. You will see that the very same Gemini models are underutilized on the app or the website, while their performance is stellar on the API or AI Studio.

bingdig 2y ago

It appears that this impacted gemini-2.5-pro-preview-03-25 somehow? grounding with google search no longer works. I had a workflow running that would pull news articles from the past 24 hours. It now refuses to believe the current date is 2025-04-17. Even with search turned on and I ask it what the date is it and it always replies sometime in July 2024.

Alifatisk 2y ago

No matter how good the new Gemini models have become, my bad experience with early Gemini is still stuck with me and I am afraid I still suffer from confirmation bias. Whenever I just look at the Gemini app, I already assume it’s going to be a bad experience.

thallavajhula 2y ago

At this point, at the current pace of AI model development, I feel like I can't tell which one is better. I usually end up using multiple LLMs to get a task done to my taste. They're all equally good and bad. It's like using GCP vs AWS vs Azure all over again, except in the AI space.

simonw 2y ago

An often overlooked feature of the Gemini models is that they can write and execute Python code directly via their API. My llm-gemini plugin supports that: https://github.com/simonw/llm-gemini https://github.com/simonw/llm-gemini uv tool install llm llm install llm-gemini llm keys set gemini # paste key here llm -m gemini-2.5-flash-preview-04-17 \ -o code_excution 1 \ 'render a mandelbrot fractal in ascii art' I ran that just now and got this: https://gist.github.com/simonw/cb431005c0e0535343d6977a7c47050c https://gist.github.com/simonw/cb431005c0e0535343d6977a7c470... They don't charge anything extra for code execution, you just pay for input and output tokens. The above example used 10 input, 1,531 output which is $0.15/million for input and $3.50/million output for Gemini 2.5 Flash with thinking enabled, so 0.536 cents (just over half a cent) for this prompt.

blahgeek 2y ago
> An often overlooked feature of the Gemini models is that they can write and execute Python code directly via their API. Could you elaborate? I thought function calling is a common feature among models from different providers
- WiSaGaN 2y ago
  This common feature requires the user of the API to implement the tool, in this case, the user is responsible to run the code the API outputs. The post you replied suggests that Gemini will run the code for the user behind the API call.
  - tempoponet 2y ago
    That was how I read it as well, as if it had a built-in lambda type service in the cloud. If we're just talking about some API support to call python scripts, that's pretty basic to wire up with any model that supports tool use.
- simonw 2y ago
  The Gemini API runs the Python code for you as part of your single API call, without you having to handle the tool call request yourself.

lleymrl651 2y ago

good

djrj477dhsnv 2y ago

Why are most comments here only comparing to Claude and just a few to ChatGPT and none to Grok? Grok 3 has been my main LLM since its release. Is it not as good as I thought it was?

jofzar 2y ago
IMO I will not use Grok while it's owned and related to Elon, not only do I not trust their privacy and data usage (not that I "really" trust open AI/Google etc) I just despise him. It would have to be very significantly better for me to use it.
dyauspitr 2y ago
Grok just isn’t the best out there.

WiSaGaN 2y ago

Interesting that the output price per 1M tokens is $0.6 for non-reasoning, but $3.5 for reasoning. This seems to defy common assumption of how reasoning models work, and you tweak the <think> token probability to control how much thinking it does, but underlying it's the same model and the same inference code path.

michaelbrave 2y ago

Yesterday I started working through How to design programs, and set up a chat with Gemini 2.5 asking it to be my tutor as I go through it and to help answer my questions if I don't understand a part of the book. It has been knowledgeable, helpful and capable of breaking down complex things that I couldn't understand into understandable things. Fantastic all around.

throwawaytodey 2y ago

[dead]

zenkey 2y ago

Google is totally back in the game now, but it’s still going to take a lot more for them at this point to overcome OpenAI’s “first‑mover advantage” (clearly the favorite among younger users atm).

sweca 2y ago
Google Pixel marketing is doing wonders for Gemini in young populations. I have been seeing a lot more of their phones in my generation's hands.

uptownfunk 2y ago

[flagged]

sinuhe69 2y ago

I’m not familiar with Python internals, so when I tried to convert a public AI model (not a LLM) to run locally, I got some problems no other AI could help. Asked Gemini 2.5 and it pin pointed the problem immediately. It solution was not practical but I guess it also works.

ashu1461 2y ago

One place where I feel gemini models lag is function calling and predicting correct arguments to function calls, is there a benchmark which scores models on the basis of this ?

upmind 2y ago

It's a shame that Gemini doesn't seem to have as much hype as GPT, I hope they gain more market share.

menshiki 2y ago

As a person mostly using AI for everyday tasks and business-related research, it's very impressive how quickly they've progressed. I would consider all models before 2.0 totally unusable. Their web interface, however, is so much worse than that of the ChatGPT macOS app.

gcbirzan 2y ago
Some aren't even at 2.0, and the version numbers aren't related in any way to their... generation? Also, what is so good about the ChatGPT app, specifically on macOS that makes it better?

convivialdingo 2y ago

Dang - Google finally made a quality model that doesn’t make me want to throw my computer out a window. It’s honest, neutral and clearly not trained by the ideologically rabid anti-bias but actually super biased regime. Did I miss a revolt or something in googley land? A Google model saying “free speech is valuable and diverse opinions are good” is frankly bizarre to see.

convivialdingo 2y ago
Downvote me all you want - the fact remains that previous Google models were so riddled with guardrails and political correctness that it was practically impossible to use for anything besides code and clean business data. Random text and opinion would trigger a filter and shut down output. Even this model criticizes the failures of the previous models.
- tempaccount420 2y ago
  Yes, something definitely changed. It's still a little biased, it's kind of like OpenAI before Trump became president.

camkego 2y ago

The pricing table image in the article really should have included Gemini 2.5 pro. Sure, it could be after Flash to the right, but it would help people understand the price performance benefits of 2.5 Flash.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

wanderr 2y ago

Gemini has the annoying habit of delegating tasks to me. Most recently I was trying to find out how to do something in FastRawViewer that I couldn't find a straightforward answer on. After hallucinating a bunch of settings and menus that don't exist, it told me to read the manual and check the user forums. So much for saving me time.

131hn 2y ago

If OpenAI offers Codex and Anthropic offers Claude Code, is there a CLI integration that Google recommends for using Gemini 2.5? That’s what’s keeping me, for now, with the other two.

hubraumhugo 2y ago

You can get your HN profile analyzed and roasted by it. It's pretty funny :) https://hn-wrapped.kadoa.com/ https://hn-wrapped.kadoa.com/ I'll add a selection for different models soon.

CoolCold 2y ago
Error An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance which may provide additional details about the nature of the error.
- hubraumhugo 2y ago
  fixed!
demaga 2y ago
Didn't expect to be roasted by AI this morning. Nice one
Alifatisk 2y ago
How is this relevant to Gemini 2.5 Flash? I guess it's using it or something?
- deleted 2y ago
  [deleted]
few 2y ago
This is cool. Does it only use a few recent comments or entire history? I'm trying to figure out where it figured out my city when I thought I was careful not to reveal it. I'm scrolling back pages without finding where I said it in the past. Could it have inferred it based on other information or hallucinated it? I wonder if there's a more opsec-focused version of this.
x187463 2y ago
Personal Projects Will finally implement that gravity in TTE, despite vowing not to. We all know how well developers keep promises. Knowledge Growth Will achieve enlightenment on the true meaning of 'enshittification', likely after attempting to watch a single YouTube video without Premium. I found these actually funny. Cool project.

yawaramin 2y ago

I just asked "why is 'Good Friday' so called?" and it got stuck. Flash 2.0 worked though.

uninformed-me00 2y ago

I want to think that this is all great, but the fact that this is also one of the best way to collect unsuspecting user data by default without explicit consent just doesn't feel right -- that applies to most people who would never have a chance reading this comment. I don't want to be angry but screw these default opt-in to have your privacy violated free stuff. Before you jump in to say you can pay to keep your privacy, stop and read again.

sgt 2y ago

How are they able to remain so competitive and will it last? The pricing almost seems too good to be true in terms of what they claim you get.

sweca 2y ago
Custom TPUs ftw

egorfine 2y ago

I am always overlooking anything Google due to the fact that they are the opposite of "Don't be evil" and because their developer's console (Google Cloud) is incredibly hostile to humans. Today I reluctantly clicked on their "AI Studio" link in the press-release and I was pleasantly surprised to discover that AI Studio has nothing in common with their typical UI/UX. It's nice and I love it!

brap 2y ago
To be fair the UX of all GCP/AWS/Azure is ass. If you don’t know exactly what you’re looking for, good luck navigating that mess.

techwiz137 2y ago

I had a heart attack moment thinking they were bringing some form of Adobe Flash back.

deleted 2y ago

[deleted]

latemedium 2y ago

There's an important difference between Gemini and Claude that I'm not sure how to quantify. I often use shell-connected LLMs (LLMs with a shell tool enabled) to take care of basic CSV munging / file-sorting tasks for me - I work in data science so there's a lot of this. When I ask Claude to do something, it carefully looks at all the directories and files before doing anything. Gemini, on the other hand, blindly jumps in and just starts moving stuff around. Claude executes more tools and is a little slower, but it almost always gets the right answer because it appropriately gathers the right context before really trying to solve the problem. Gemini doesn't seem to do this at all, but it makes a world of difference for my set of problems. Curious to see if others have had the same experience or if its just a quirk of my particular set of tasks

energy123 2y ago
What's a shell connected LLM and how to do that?
- kmacdough 2y ago
  Look up Claude Code, Cursor, Aider and VSCode's agent integration. Generally, tools to use AI more actively for development. There are others as well. Plenty of info around. Here's not the place for a tutorial.
Alifatisk 2y ago
Claude has always been the best at coding, no matter what all the benchmarks says, the people have spoken and the consensus is that Claude is the best.

rjurney 2y ago

I am building a knowledge graph using BAML [baml-py] to extract documents [it's opinionated towards docs] and then PySpark to ETL the data into a node / edge list. GPT4o got few relations... Gemini 2.5 got so many it was nuts, all accurate but not all from the article! I had to reign it in and instruct it not to build so vast a graph. Really cool, it knows a LOT about semiconductors :)

profsummergig 2y ago

I tried this prompt in both Gemini 2.5 Pro, and in ChatGPT. "Draw me a timeline of all the dynasties of China. Imagine a horizontal line. Start from the leftmost point and draw segments for the start and end of each dynasty. For periods where multiple dynasties existed simultaneously draw parallel lines or boxes to represent the concurrent rule." Gemini's response: "I'm just a language model, so I can't help you with that." ChatGPT's response: an actual visual timeline.

renewiltord 2y ago
All the communities where people think LLMs are junk love Gemini. Makes me sceptical that the enthusiasm is useful signal. I found the full 2.0 useful for transcription of images. Very good OCR. But not a good assistant. Stalls often and once it has, loses context easily.
- thegeomaster 2y ago
  Is it possible that a community of people who are constantly pushing LLMs to their limits would be most aware of their limitations, and so more inclined to think they are junk? In terms of business utility, Google has had great releases ever since the 2.0 family. Their models have never missed some mark --- either a good price/performance ratio, insane speeds, novel modalities (they still have the only API for autoregressive image generation atm), state-of-the-art long context support and coding ability (Gemini 2.5), etc. However, most average users are using these models through a chat-like UI, or via generic tools like Cursor, which don't really optimize their pipelines to capture the strengths of different models. This way, it's very difficult to judge a model objectively. Just look at the obscene sycophancy exhibited by chatgpt-4o-latest and how it lifted LMArena scores.
  - renewiltord 2y ago
    Just the fact that everyone on HN is always telling us how LLMs are useless but that Gemini is the best of them convinces me of the opposite. No one who can't find a use for this technology is really informed on the subject. Hard to take them seriously.

asim 2y ago

I just wish the whole industry would stop using terms like thinking and reasoning. This is not what's happening. If we could come up with more appropriate terms that don't treat these models like they're human then we'd be in a much better place. That aside, it's cool to see the advancement of Google's offering.

dbbk 2y ago
Thinking perhaps, but why not reasoning?
lyu07282 2y ago
Do you think any machine will ever be able to think and/or reason? Or is that a uniquely human thing? and do you have a rational standard to judge when something is reasoning or thinking, or just vibes? I'm asking because I wonder how much of that common attitude is just a sort of species-chauvinism. You are feeling anxious because machines getting smarter, you are feeling anger because "they" are taking your job away, but the machine doesn't do that, its people with an ideology that do that, you should be angry at that instead.

aerhardt 2y ago

I am only on OpenAI because they have a native Mac app. Call me old-school but my preferred workflow is still for the most part just asking narrow questions and copying-pasting back and forth. I've been playing with Junie (Jetbrain's AI agent) for a couple of days, but I still don't trust agents to run loose in my codebase for any sizeable amount of work. Does anyone know if Google is planning native apps? Or any wrapping interfaces that work well on a Mac?

sweca 2y ago
Raycast[0] has Gemini support in their AI offering and it's native, fast and intuitive. [0] https://raycast.com/ai https://raycast.com/ai

sweca 2y ago

Honestly, the best part about Gemini, especially as a consumer product, is their super lax, or lack thereof, ratelimits. They never have capacity issues, unlike Claude which always feels slow or sometimes outright rejects requests during peak hours. Gemini is constantly speedy and has extremely generous context window limits on the Gemini apps.

onlyrealcuzzo 2y ago
Interesting. I use Claude quite a bit, and haven't encountered this. Is this the free version of Claude or the paid version? When are peak hours typically (in what timezone)?
- sweca 2y ago
  I have Claude Pro and peak hours are in the afternoon and at night for me in EST

bossyTeacher 2y ago

Is everyone on here solely evaluating the models on their programming capabilities? I understand this is HN but vibe coding LLM tools won't be able to sustain the LLM industry (let's not call it AI please)

barfingclouds 2y ago

I just need the Gemini app to allow push to talk :( Otherwise it’s not usable for me in the way I want it to be