8 ms·
I appreciate the honesty in the marketing materials. Showing the product scoring below the market leader in a big benchmark is better than the Google way of che
by speedgoose 3y ago
I appreciate the honesty in the marketing materials. Showing the product scoring below the market leader in a big benchmark is better than the Google way of cherry picking benchmarks.
- breadwinner 3y ago[flagged]
- a_vanderbilt 3y agoWhich size model are you using? Large isn't terribly good, but Next is alright. It's not close at all to GPT-4, but I can see some use cases I'd try it for (and will be).
- Takennickname 3y agoI tried it. Seems fine. What prompts gave you nonsense?
- breadwinner 3y agoI tried what Google would call "long tail" queries. Chat GPT-4 gave me accurate answers, but Mistral gave me nonsensical answers. I can't share the exact prompts because they are personally identifying.
- Takennickname 3y agoAnd you can't share them and replace the personal information with xxx?
- syntaxing 3y agoAm I using these wrong? I asked a couple git and python questions and it answered it about the same as GPT-4 Turbo (or whatever ChatGPT uses nowadays). The answer was slightly better than GPT-3.5 Turbo in the sense there was a lot of fluff in the GPT-3.5 Turbo's answer.
- pseudosavant 3y agoI haven't tried this model yet (they seem to be under high load). Mistral's previous top model (Mixtral 8x7B) wasn't quite as good as GPT-4, but it seemed close. Clearly way better than GPT 3.5. Many Mixtral responses were better than the GPT-4 ones.
- onlyrealcuzzo 3y agoThey compare to Gemini Pro 1.0... Seems intentionally misleading.
- speedgoose 3y agoRight. Gemini Pro 1.5 scores 81.9% on MMLU and is also above in a few other benchmarks.
- onlyrealcuzzo 3y agoWhich - importantly - is better than Mistral at 81.2% ... Gemini Ultra scored 90% which is better than GPT-4. This reads like a paid-for press release from Microsoft to pretend like they're almighty and Google is incompetent.
- 7moritz7 3y agoDidn't Gemini 1.0 Ultra turn out to be worse than GPT 4 Turbo in practice? Synthetic benchmarks aren't a perfect representation of anything.
- maleldil 3y agoI think they're close enough that it depends on your use case. IME, which is mostly generating Python and shell code, Gemini Advanced is faster, less lazy and generates overall better solutions than GPT-4. It also seems to be more up-to-date with libraries and stuff. It can also directly go to URLs (e.g. summarise a paper), which GPT-4 refuses to do.
- Nidhug 3y agoDoesn't MMLU have like 2-3% wrong answers anyways ?
- jug 3y agoWith 128K context (1M paid) compared to 32K. Man, Google is going to be a game changer for especially free AI tiers. Edit: BTW, more Mistral benchmarks here: https://docs.mistral.ai/platform/endpoints/ https://docs.mistral.ai/platform/endpoints/ TIL Mistral Small outperforms Mixtral 8x7B.
- IncreasePosts 3y ago[flagged]
- autokad 3y agoit sounds like they are trying to be clear they aren't stepping on chatgpt's (openai) toes edit: not sure why I am being downvoted. I am 100% sure the way they structured it was meant to say "we are doing great, but not as great as openAI's work, which we are not trying to compete against". I guarantee there were discussions on how to make it look as to not appear that way.