13 ms·
Two new Gemini models, reduced 1.5 Pro pricing, increased rate limits, and more
- thekevan 2y agoGoogle does not miss one single opportunity to miss an opportunity. They announced a price reduction but it "won't be available for a few days". By the time, the initial hype will be over and the consumer-use side of the opportunity to get new users will be lost in other news.
- deleted 2y ago[deleted]
- phren0logy 2y agoAs far as I can tell there's still no option for keeping data private?
- diggan 2y agoMakes sense, as soon as your data leaves your computer, it's safe to assume it's no longer private, no matter what promises a service gives you. You want guaranteed private data that won't be used for anything? Keep it on your own computer.
- 999900000999 2y agoNot sure why your getting down voted. Anything sent to an cloud hosted LLM is subject to be publicly released or used in training. Setting up a local LLM isn't that hard, although I'd probably air gap anything truly sensitive. I like ollama, but it wouldn't surprise me if it's phoning home.
- phren0logy 2y agoThis is just incorrect. The OpenAI models hosted though Azure are HIPAA-compliant, and Antropic will also sign a BAA.
- 999900000999 2y agoI'm open to being wrong. However for many industries your still running the risk of leaking data via a 3rd party service. You can run Llama3 on prem, which eliminates that risk. I try to reduce reliance on 3rd party services when possible. I still have PTSD from Saucelabs constantly going down and my manager berating me over it.
- caseyy 2y agoYou are not technically wrong because a statement "there is a risk of leaking data" is not falsifiable. But your comment is performative cynicism to display your own high standards. For the very vast majority of people and companies, privacy standards-compliant services (like HIPAA-compliant) are private enough.
- 999900000999 2y agoI know my company outright warns us to not share any sensitive information with LLMs, including ones that claim to not use customer data for training. I can flip your statement around. For the vast majority of use cases, LLAMA 3 can be hosted on prem and will have similar performance.
- spiralk 2y agoThis is not true. Both OpenAI and Google's LLM APIs have a policy of not using the data sent over them. Its no different than trusting Microsoft's or Google's cloud to store private data.
- phren0logy 2y agoCan you link to documentation for Google's LLMs? I searched long and hard when Gemma 2 came out, and all of the LLM offerings seemed specifically exempted. I'd love to know if that has changed.
- spiralk 2y agohttps://ai.google.dev/gemini-api/terms https://ai.google.dev/gemini-api/terms this?
- phren0logy 2y agoThanks very much! I think before I looked at docs for Google AI Studio, but also for Google Workspace, and both made no guarantees. From the linked document, so save someone else a click: > The terms in this "Paid Services" section apply solely to your use of paid Services ("Paid Services"), as opposed to any Services that are offered free of charge like direct interactions with Google AI Studio or unpaid quota in Gemini API ("Unpaid Services").
- Deathmax 2y agoThere's some possible confusion because of the Copilot problem where everything in the product stack is called Gemini. The Gemini API (or Generative Language API) as documented on https://ai.google.dev https://ai.google.dev uses https://ai.google.dev/gemini-api/terms https://ai.google.dev/gemini-api/terms for its terms. Paid usage, or usage from a UK/CH/EEA geolocated IP address will not be used for training. Then there's Google Cloud's Vertex AI Generative AI offering, which has https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance https://cloud.google.com/vertex-ai/generative-ai/docs/data-g.... Data is not used for training, and you can opt out of the 24 hour prompt cache to effectively be zero retention. And then there's all the different consumer facing Gemini things. The chatbot at https://gemini.google.com/ https://gemini.google.com/ (and the Gemini app) uses data for training by default: https://support.google.com/gemini/answer/13594961l https://support.google.com/gemini/answer/13594961l, unless you pay for Gemini Enterprise as part of Gemini for Workspace. Gemini in Chrome DevTools uses data for training (https://developer.chrome.com/docs/devtools/console/understand-messages?s=1#data-use https://developer.chrome.com/docs/devtools/console/understan...). Enterprise features like Gemini for Workspace (generative AI features in the office suite), Gemini for Google Cloud (generative AI features in GCP), Gemini Code Assist, Gemini in BigQuery/SecOps/etc do not use data for training.
- sweca 2y agoIf you are on the pay as you go model your data is exempted from training. > When you're using Paid Services, Google doesn't use your prompts (including associated system instructions, cached content, and files such as images, videos, or documents) or responses to improve our products, and will process your prompts and responses in accordance with the Data Processing Addendum for Products Where Google is a Data Processor. This data may be stored transiently or cached in any country in which Google or its agents maintain facilities. https://ai.google.dev/gemini-api/terms https://ai.google.dev/gemini-api/terms
- wildmXranat 2y agoInteresting. In essence, we could equate to paying for the Gemini Advanced or Pro as a way to avoid use of our data and prompts. https://ai.google.dev/gemini-api/terms#data-use-paid https://ai.google.dev/gemini-api/terms#data-use-paid
- kendallchuang 2y agoHas anyone used Gemini Code Assist? I'm curious how it compares with Github Copilot and Cursor.
- spotlmnop 2y agoGod, it sucks
- spiralk 2y agoThe Aider leaderboards seem like a good practical test of coding usefulness: https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/. I haven't tried Cursor personally but I am finding Aider with Sonnet more useful that Github Copilot and its nice to be able to pick any model API. Eventually even a local model may be viable. This new Gemini model does not rank very high unfortunately.
- kendallchuang 2y agoThanks for the link. That's unfortunate, though perhaps the benchmarks will be updated after this latest Gemini release. Cursor with Sonnet is great, I'll have to give Aider a try as well.
- spiralk 2y agoIt is updated actually, gemini-1.5-pro-002 is this new model.
- kendallchuang 2y agoThat was fast, I missed it!
- KaoruAoiShiho 2y agoIt's on the leaderboard, it's tied with qwen 2.5 72b and far below SOTA of o1, claude sonnet, and deepseek. (also below very old models like gpt-4-0314 lol)
- 2y ago
- TheAceOfHearts 2y agoI only use regular Gemini and the main feature I care about is absolutely terrible: summarizing YouTube videos. I'll ask for a breakdown or analysis of the video, and it'll give me a very high level overview. If I ask for timestamps or key points, it begins to hallucinate and make stuff up. It's incredibly disappointing that such a huge company with effectively unlimited access to both money and intellectual resources can't seem to implement a video analysis feature that doesn't suck. Part of me wonders if they're legitimately this incompetent or if they're deliberately not implementing good analysis features because it could eat into their views and advertisement opportunities.
- foota 2y agoI imagine you'd be paying more in ML costs than YouTube makes off your views.
- bahmboo 2y agoI've had it fail when a video does not have subtitles - I'm guessing that's what it uses. I have had good success having it answer the clickbait video titles like "is this the new best thing?" It's not watching the video as far as I can tell.
- msp26 2y agoHas anyone tried Google's context caching feature? The minimum caching window being 32k tokens seems crazy to me.
- jzebedee 2y agoAs someone who actually had to build on Gemini, it was so indefensibly broken that I couldn't believe Google really went to production with it. Model performance changes from day to day and production is completely unstable as Google will randomly decide to tweak things like safety filtering with no notice. It's also just plain buggy, as the agent scaffolding on top of Gemini will randomly fail or break their own internal parsing, generating garbage output for API consumers. Trying to build an actual product on top of it was an exercise in futility. Docs are flatly wrong, supposed features are vaporware (discovery engine querying, anybody?), and support is nonexistent. The only thing Google came back with was throwing more vendors at us and promising that bug fixes were "coming soon". With all the funded engagements and credits they've handed out, it's at the point where Google is paying us to use Gemini and it's _still_ not worth the money.
- wewtyflakes 2y ago> Docs are flatly wrong This +999; I couldn't believe how inconsistent and wrong the docs were. Not only that, but once I got something successfully integrated, it worked for a few weeks then the API was changed, so I was back to square one. I gave it a half-hearted try to fix it but ultimately said 'never again'! Their offering would have to be overwhelmingly better than Anthropic and OpenAI for me to consider using Gemini again.
- victor106 2y agoSame experience here. I had hopes of Google able to compete with Claude and OpenAI. But I don’t think that’s the case. Unless they come out with a product that’s 10x better in the next year or so I think they lost the AI race.
- ldjkfkdsjnv 2y agoThe engineers at google are bad, they keep hiring via pure leetcode. Cant ship working products
- deleted 2y ago[deleted]
- naiv 2y agoThis sounds interesting: "We will continue to offer a suite of safety filters that developers may apply to Google’s models. For the models released today, the filters will not be applied by default so that developers can determine the configuration best suited for their use case."
- ancorevard 2y agoThis is the most important update. Pricing and speed doesn't matter when your call fails because of "safety".
- CSMastermind 2y agoAlso Google's safety filters are absolutely awful. Beyond parody levels of bad. This is a query I did recently that got rejected for "safety" reasons: Who are the current NFL starting QBs? Controversial I know, I'm surprised I'd be willing to take the risk with submitting such a dangerous query to the model.
- elashri 2y agoNot stranger than my experience with openai. I got banned from DELL-3 access when it first came because I asked in the prompt about generating a particle moving in magnetic field of a forward direction and decays to two other particles with a kink angle between the particle and the charged daughter. I don't recall exact prompt but it should be something close to that. I really wonder what filters they had about kink tracks and why? Do they have a problem with Beyond standard model searches /s.
- CSMastermind 2y agoFor what it's worth I run every query I make through all the major models and Google's censorship is the only one I consistently hit. I think I bumped into Anthropics once? And I know I hit ChatGPTs a few months back but I don't even remember what the issue was. I hit Google's safety blocks at least a few times a week during the course of my regular work. It's actually crazy to me that they allowed someone to ship these restrictions. They must either think they will win the market no matter the product quality or just not care about winning it.
- stan_kirdey 2y agoGoogle should just offer llama3 405b, maybe slightly fine tuned. Geminis are unusable.
- tiborsaas 2y agoAI companies should not pick up naming models after astrological signs, after a while it will be hard to tell apart model reviews from horoscope.
- dcchambers 2y ago> Geminis are unusable how so?
- JimDabell 2y agoIn my experience, Gemini models are far worse than any other frontier model when it comes to hallucinations. They are also pretty bad at getting caught in loops where pointing out a mistake makes it flap between two broken solutions. And obviously the overzealous softly stuff that other people have mentioned.
- romland 2y agoPretty sure Google's got more than 700 million active users. In fact, Google is most likely _the_ target for that clause in the license.
- nmfisher 2y agoI've found Gemini Pro to be the most reliable for function calling.
- jmah 2y ago[flagged]
- deleted 2y ago[deleted]
- resource_waste 2y agoIts just not as smart as ChatGPT or LLAMA, its mind boggling Google fell so far behind.
- Mistletoe 2y agoHas it changed much since March when this was written? Gemini won, 6-4. And it matches my experiences of just using Gemini instead of ChatGPT because it gives me more useful responses, when I feel like I want an AI answer. https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt
- w23j 2y ago"For this initial test I’ll be comparing the free version of ChatGPT to the free version of Google Gemini, that is GPT-3.5 to Gemini Pro 1.0." The free version of ChatGPT is 4o now, isn't it? So maybe Gemini has not gotten worse, but the free alternatives are now better? When I compare ChatGPT-4o with Gemini-Advanced (wich is 1.5 Pro, I believe) the latter is just so much worse.
- ldjkfkdsjnv 2y agoThey have to drop the price because the model is bad. People will pay almost any cost for a model that is much better than the rest. How this company carries on the facade of competence is laughable. All the money on the planet, and they still cannot win on their core "competency".
- FergusArgyll 2y agoAny opinions on pro-002 vs pro-exp-0827 ? Unlike others here I really appreciate the gemini API, it's free and it works. I haven't done too many complicated things with it but I made a chatbot for the terminal, a forecasting agent (for metaculus challenge) and a yt-dlp auto namer of songs. The point for me isn't really how it compares to openAI/anthropic, it's a free API key and I wouldn't have made the above if I had to pay just to play around
- rkwasny 2y agoCan someone explain to me why there is COMPLETELY different pricing for models on Vertex AI, Google AI studio and also OpenRouter has another price ...
- hiddencost 2y agohttps://www.businessinsider.com/big-tech-org-charts-2011-6 https://www.businessinsider.com/big-tech-org-charts-2011-6 Google is now looking more like the Microsoft chart.
- sweca 2y agoNo Human eval benchmark result?
- kebsup 2y agoIs there a good benchmark comparing multilingual and/or translation abilities of most recent LLMs? GPT-4o struggles for some tasks in my language learning app.
- charlie0 2y agoAnyone want to take bets on how long it takes for this to hit the Google Graveyard?
- arresin 2y agoI’ve used it. The API is incredibly buggy and flakey. A particular pain point is the “recitation error” fiasco. If you’re developing a real world app this basically makes the Gemini api unusable. It strikes me as a kind of “Potemkin” service. Google is aware of the issue and it has been open on google's bug tracker since March 2024: https://issuetracker.google.com/issues/331677495 https://issuetracker.google.com/issues/331677495 There is also discussion on GitHub: https://github.com/google-gemini/generative-ai-js/issues/138 https://github.com/google-gemini/generative-ai-js/issues/138 It stems from something google added intentionally to prevent copyright material being returned verbatim (ala the NYT openai fiasco), so they dialled up the "recitation" control (the act of repeating training data—and maybe data they should not have legally trained on). Here are some quotes from the bug tracker page: > I got this error by just asking "Who is Google?" > We're encountering recitation errors even with basic tutorials on application development. When bootstrapping a Spring Boot app, we're flagged for the pom.xml being too similar to some blog posts. > This error is a deal breaker... It occurs hundreds of times a day for our users and massively degrades their UX.
- deleted 2y ago[deleted]
- screye 2y agoThe recitation error is a big deal. I was ready to champion gemini use across my organization, and the recitation issue curbed any enthusiasm I had. It's opaque and Google has yet to suggest a mitigation. Your comment is not hyperbole. It's a genuine expression of how angry many customers are.
- 3choff 2y agoIt seems the recitation problem has been fixed with the latest models. In my tests, the answer generation no longer stops prematurely. Before I resume a project that was on hold due to this issue, I'm gathering feedback from other users about their experiences with this and the new models. Are you still having this problem?
- anotherpaulg 2y agoThe new Gemini models perform basically the same as the previous versions on aider's code editing benchmark. The differences seem within the margin of error. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- ramshanker 2y agoOne company buying expensive NVIDIA hardware vs another using in-house chips. Google got a huge advantage here. They could really undercut OpenAI.
- rty32 2y agoPeople have said that for many years. Very few companies are choosing Google's TPUs. Everyone wants H100s.
- deleted 2y ago[deleted]
- accumulator 2y agoCool, now all Google has to do is make it easier to onboard new GCP customers and more people will probably use it...its comical how hard it is to create a new GCP organization & billing account. Also I think more Workspace customers would probably try Gemini if it was a usage-based trial as opposed to clicking a "Try for 14 days" CTA to activate a new subscription.
- pacoverdi 2y agoWhy the ** do they have to use a 8.9MiB png as hero image? I guess it was generated by AI?
- mnicky 2y agoI think the sentiment here is not fully objective, there are nice improvements in benchmarks (and even more so when accounting for the price): https://imgur.com/a/K3tVPEw https://imgur.com/a/K3tVPEw Also, this model shouldn't be compared to the CoT o1, I think. That is something different (also in price and speed).
- deleted 2y ago[deleted]
- mixtureoftakes 2y agoTLDR - 2x cheaper, slightly smarter, and they only compare those new models to their own old ones. Does google have moat?
- usaar333 2y agoThe math score exceeds o1-preview (though not mini or o1 full) fwiw.
- re-thc 2y ago> Does google have moat? Potentially (depends if the EU cares)... E.g. integration with Google search (instead of ChatGPT's Bing search), providing map data, android integration, etc...
- ianbicking 2y agoTheir Android integration certainly isn't on track to earn them any moats... https://hachyderm.io/@ianbicking/113099247306589777 https://hachyderm.io/@ianbicking/113099247306589777
- deleted 2y ago[deleted]
- cj 2y agoMoat could be things like direct integration into Gmail (ask it to find your last 5 receipts from Amazon), Drive (chat with PDF), Slides (create images / flow charts), etc. Not sure if their models are the moat. But they definitely have an opportunity from the productization perspective. But so does Microsoft.
- svara 2y agoHave you tried the Gemini Gmail integration? I have that enabled in my GSuite account. It's incredible how bad it is. I've seen it claim I've never received mail from a certain person, while the email was open right next to the chat widget. I've seen it tell me to use the standard search tool, when that wasn't suitable for the query. I've literally never had it find anything that wouldn't have been easier to find with the regular search. I mean, it's a really obvious thing for them to do, I'm genuinely confused why they released it like that.
- pie420 2y ago[flagged]
- deleted 2y ago[deleted]
- frankdenbow 2y agoInterview with the product lead: https://x.com/rowancheung/status/1838611170061918575 https://x.com/rowancheung/status/1838611170061918575?
- throwup238 2y ago> What makes the new model so unique?" >> Yeah It's a good question. I think it's maybe less so of what makes it unique and more so the general trajectory of the trend that we're on.* Disappointing.
- hadlock 2y agoThey put a 42 minute video on twitter? That's brave.
- mh- 2y agohttps://www.youtube.com/watch?v=WQvMdmk8IkM https://www.youtube.com/watch?v=WQvMdmk8IkM
- summerlight 2y agoLooks like they are more focused on the economical aspect of those large models? Like 90~95% performance of other frontier models at 50%~70% price.
- TIPSIO 2y agoI do like the trend. Imagine if Anthropic or someone eventually release a Claude 3.5 but at like a whopping 10x its current speed. Would be incredibly more useful and game changing than a slow o1 model that may or not be x percent smarter.
- sgt101 2y agoWe might see that with the inference ASICs later this year I guess?
- xendipity 2y agoOoh, what are these ASICs you're talking about? My understanding was that we'll see AMD/Nvidia gpus continue to be pushed and very competitive as well as have new system architectures like cerebras or grok. I haven't heard about new compute platforms framed as ASICs.
- Workaccount2 2y agoCerebras has ridiculously large LLM ASICs that can hit crazy speeds. You can try it with llama 8B and 70B: https://inference.cerebras.ai/ https://inference.cerebras.ai/ It's pretty fast, but my understanding is that it is still too expensive even accounting for the speed-up.
- throwup238 2y agoIs Cerebras an integrated circuit or more an integrated wafer? :-) And yeah their cost is ridiculous, on the order for high 6 to low 7 figures per wafer. The rack alone looks several times more expensive than the 8x NVIDIA pods [1] [1] https://web.archive.org/web/20230812020202/https://www.youtube.com/watch?v=pzyZpauU3Ig https://web.archive.org/web/20230812020202/https://www.youtu...
- simonw 2y agoThis price drop is significant. For <128,000 tokens they're dropping from $3.50/million to $1.25/million, and output from $10.50/million to $2.50/million. For comparison, GPT-4o is currently $5/million input and $15/million output and Claude 3.5 Sonnet is $3/million input and $15/million output. Gemini 1.5 Pro was already the cheapest of the frontier models and now it's even cheaper.
- GaggiX 2y agoGPT-4o is 2.5/10$. Unless you look at an old checkpoint. GPT-4o was the cheapest frontier model before.
- simonw 2y agoI can’t see that price on https://openai.com/api/pricing/ https://openai.com/api/pricing/ - it’s listing $5/m input and $15/m output for GPT-4o right now. No wait, correction: That’s confusing: it lists 4o first and then lists gpt-4o-2024-08-06 as $2.50/$10.
- jeffharris 2y agoapologies: it's taken us a minute to switch the default `gpt-4o` pointer to the newest snapshot we're planning on doing that default change next week (October 2nd). And you can get the lower prices now (and the structured outputs feature) by manually specify `gpt-4o-2024-08-06`
- jiggawatts 2y ago> “You can” No, “I” can’t. Open AI has always trickled out model access, putting their customers into “tiers” of access. I’m not sufficiently blessed by the great Sam to have immediate access. On, and Azure Open AI especially likes to drag their feet both consistently, and also on a per-region basis. I live in a “no model for you” region. Open AI says: “Wait your turn, peasant” while claiming to be about democratising access. Google and everyone else just gives access, no gatekeeping.
- serjester 2y agoGemini feels like an abusive relationship — every few months, they announce something exciting, and I’m hopeful that this time will be different, that they’ve finally changed for the better, but every time, I’m left regretting having spent any time with them. Their docs are awful, they have multiple unusable SDK's and the API is flaky. For example, I started bumping into "Recitation" errors - ie they issue a flat out refusal if your response resembles anything in the training data. There's a GitHub issue with hundreds of upvotes and they still haven't published formal guidance on preventing this. Good luck trying to use the 1M context window. Everything is built the "Google" way. It's genuinely unusable unless you're a total masochist and want to completely lock yourself into the Google ecosystem. The only thing they can compete on is price.