4 ms·
> They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. You h
by jjani 2y ago
> They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions.
You haven't been keeping up. Less than 2 weeks ago, Google released a model that has crushed the competition, clearly being SotA while currently effectively free for personal use.
Gemini 2.0 was already good, people just weren't paying attention. In fact 1.5 pro was already good, and ironically remains the #1 model at certain very specific tasks, despite being set for deprecation in September.
Google just suffered from their completely botched initial launch way back when (remember Bard?), rushed before the product was anywhere near ready, making them look lile a bunch of clowns compared to e.g. OpenAI. That left a lasting impression on those who don't devote significant time to keeping up with newer releases.
- codingwagie 2y agogemini 2.5 pro isnt good, and if you think it is, you arent using LLMs correctly. The model gets crushed by o1 pro and sonnet 3.7 thinking. Build a large contextual prompt ( > 50k tokens) with a ton of code, and see how bad it is. I cancelled my gemini subscription
- lerchmo 2y agohttps://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ your experience doesn't align with my experience or this benchmark. o1 pro is good but I would rather do 20 cycles on gemini 2.5 rather than wait for Pro to return.
- jjani 2y agoI have, dozens of times, and it's generally better than 3.7. Especially with more context it's less forgetful. o1-pro is absurdly expensive and slow, good luck using that with tools. Virtually all benchmarks, including less gamed ones such as Aider's, show the same. WebLM still has 3.7 ahead, with Sonnet always having been particularly strong at web development, but even on there 2.5 Pro is miles in front of any OpenAI model. Gemini subscription? Surely if you're "using LLMs correctly" you'd have been using the APIs for everything anyway. Subscriptions are generally for non-techy consumers. In any case, just straight up saying "it isn't good" is absurd, even if you personally prefer others.
- int_19h 2y agoI have just watched Sonnet 3.7 vs Gemini 2.5 solving the same task (fix a bug end-to-end) side by side, and Sonnet hallucinated far worse and repeatedly got stuck in dead-ends requiring manual rescue. OTOH Gemini understood the problem based on bug description and code from the get go, and required minimal guidance to come up with a decent solution and implement it.