6 ms·
This is extremely impressive if benchmarks translate to real-world performance [1]. The mistral-medium beats GPT3.5 and also Gemini Pro (Google's best available
by Palmik 3y ago
This is extremely impressive if benchmarks translate to real-world performance [1]. The mistral-medium beats GPT3.5 and also Gemini Pro (Google's best available model) with a huge margin on all available comparable benchmarks: https://screenbud.com/shot/c0d904e3-24a3-4c23-a1e4-2f18bc0215cf/image.png https://screenbud.com/shot/c0d904e3-24a3-4c23-a1e4-2f18bc021...
[1] I would expect real world-performance gap to be even larger if Mistral 7B is anything to go by. The fact that safety filters are opt-in is a huge benefit (even for safe applications).
- monlockandkey 3y agoThis is crazy. If they release this new model as open source, that would be very exciting!
- Aerbil313 3y agoThey already did, weights on Twitter.
- JimDabell 3y agoNo, they’ve released the weights for Mistral-small. They haven’t released the weights for Mistral-medium.
- samtheprogram 3y agoThey might’ve been referring to mistral-medium, the larger version, which is not open source. The open-sourced version was the small variant.
- usaar333 3y agoHigher Winogrande score than GPT-4. Surprising given others are more GPT-3ish
- logicchains 3y ago>also Gemini Pro (Google's best available model) Pretty sad for Google if their next big AI thing is already being beaten by a small company with a tiny fraction of their resources.
- margorczynski 3y agoGoogle in general looks like they're lost their touch and are kinda on a downward spiral. Trying to squeeze out as much as they can from Youtube is I think a major sign of that, usually attempts like that start to pop up in burnt-out companies that are heading to their end. As for AI they have a lot of world-class talent but they somehow cannot transform that into a coherent product even though they did amazing stuff on the research side. It has been brought up many times on HN on how Google has a product-design problem and it seems to be true.
- tomjen3 3y agoGoogle is conflicted about AI. Having AI do searches for you and summerize the results is both way more expensive than normal searches and means way fewer ads are shown. Google can't do anything about it either - consumers will prefer the chat approach and will go where it is offered. Google has to offer it, or people will go to ChatGpt and Bing, but at the same time they lose a lot of money each time people opt for the chat summary. It is way expensive for Bing too, but they don't care because their market is growing and since the only thing people used to search Bing for was how to download Chrome and how to change the default browser, they aren't losing a lot this way.
- AaronFriel 3y agoImagine a search engine that tries to persuade you a sponsored product is the answer to your problems, and it can use its knowledge about you to do so. I think search products will be OK - it will just be stranger than we expect.
- jamiek88 3y agoBig was used for porn. Don’t forget porn. Torrents too. Bing going legit and genuinely threatening google wasn’t on my bingo card for sure.
- eastbound 3y agoAnd honest content. One order of magnitude more honest at least; Google is like asking the Kremlin for truth.
- willsmith72 3y ago> Gemini Pro (Google's best available model) It's not really "available" though, is it? I won't buy any PR benchmarks until the models are publicly available, there's too much variation based on how the models need to toned back for safety reasons before being released to the public
- ioulaum 3y agoI just saw a video, and Mixtral itself performed amazingly well on a wide variety of tasks - where GPT-4 doesn't always do as well.