5 ms·
Has anyone (outside of Google) gotten to play with Gemini Ultra yet? Been hearing a lot about Pro, but I'd be interested in seeing whether Ultra is really close
by paxys 3y ago
Has anyone (outside of Google) gotten to play with Gemini Ultra yet? Been hearing a lot about Pro, but I'd be interested in seeing whether Ultra is really close to as capable as they claim.
Also very interesting that Mixtral 8x7B ranks in the same neighborhood as Gemini Pro/GPT 3.5 Turbo/Claude 2.1 while being fully open source and Apache 2.0 licensed.
- niklasrde 3y agoLooks like Microsoft people have had access to it to benchmark promptbase: https://github.com/microsoft/promptbase https://github.com/microsoft/promptbase
- mkmk 3y agoMixtral is a mystery to me. How in the world is that team on par with/beating GOOGLE, who presumably have all the resources in the world to throw at this?
- carterschonwald 3y agoMy understanding, however fuzzy, is that all the safety/politeness tuning results in models that are at times less likely to give accurate responses. That said, I suspect that either way both types of models largely give similar answers for soft questions aside from those politeness and safety things
- n2d4 3y agoMixtral is on-par with Gemini Pro, not Gemini Ultra (and even there it is further behind Gemini Pro than Gemini Pro is behind GPT 3.5). But to directly answer your question, they are quite well-funded, having raised over $700mil to date. I definitely wouldn't count them out.
- euazOn 3y agoGemini Ultra is not out yet. With the same logic, you could compare an unreleased Mistral model with Gemini Ultra.
- sp332 3y agoMistral “Medium” is available (in beta, via API) and should give better results than the “Small” mixtral model.
- n2d4 3y agoRight. I'm just pointing out that comparing one model with a distilled version of another and then making broad statements about the companies behind them isn't really useful. Surely you could make a comparison of two unreleased models, but it wouldn't be interesting because you don't have any real data (and benchmarks don't really mean anything).
- jstummbillig 3y agoDebating the usefulness of hn commentary is a somewhat philosophical issue, but I think it's entirely fair to draw parallels between what is, not what might be. Gemini Ultra is self-evidently not ready for production. What the issues are? Who knows, but in a game that as of right now is mostly about reducing the amount of brute force required, something as "simple" as not being efficient enough is actually not something to gloss over. If your engines entire stick is having the greatest graphics but you can't make it run at acceptable fps, well, then it's not actually a usable product. A LLM that is not actually released could very well be in a comparably dire state and fixing it while also delivering on the promised performance might be entirely non-trivial.
- 3y ago
- bitshiftfaced 3y agoI wonder if Google sees MoE as a sort of local maxima, and so they tried a different path hoping it might outperform it.
- deleted 3y ago[deleted]
- moffkalast 3y agoMistral.AI was founded by three people from Deepmind, they're beating Google because Google no longer has them.
- guyomes 3y agoSlight correction, Mistral.AI was funded by two people from Meta (Guillaume Lample, Timothée Lacroix) and one from Deepmind (Arthur Mensch). For new technologies, what matters most might be the universities where people are from, rather than the companies. The founders of Google graduated from Stanford. The founders of Mistra AI graduated from École Polytechnique and École Normale Supérieure, that are renowned in France, notably for their scientific formations.
- paxys 3y agoSame as OpenAI, Anthropic, Cohere, Adept and hundreds of other small-mid sized AI startups. When the dust settles and the space gets more mature the exodus from Google Brain/Deepmind over the last few years will be considered this generation's Fairchild moment.
- cfiggers 3y agoThere's a survivorship bias going on here. You've never heard of the thousands of teams out there that are Mistral's size but AREN'T getting results that compete on the global stage, but they do exist. But you've heard of Google, whether they're getting it right or not.
- pixl97 3y agoI wonder how much GPU compute time is effectively being discarded by these companies as the results turn out to be garbage?
- paxys 3y ago"Thousands of teams" is a vast exaggeration. A tiny handful of companies out there have received funding to the tune of a billion dollars for model training like Mixtral. All of them have researchers with loaded resumes, and most are producing stuff of value. The thousands of other startups in the ecosystem are then taking these APIs and adding trivial abstractions on top.
- SeanAnderson 3y agoThe hosts of the All In podcast have used it, but they're billionaires. They think highly of Ultra. Early on they just talk about the paper that's released, but then they drop that they've used Ultra. https://www.youtube.com/watch?v=IeKUcpU5-Xk&t=3667s https://www.youtube.com/watch?v=IeKUcpU5-Xk&t=3667s
- paxys 3y agoStarted watching the video wondering when it would turn into a rant about "wokeness" and "cancel culture", and it happened about 30 seconds in. Glad to see these guys haven't changed.