4 ms·
Right. Gemini Pro 1.5 scores 81.9% on MMLU and is also above in a few other benchmarks.
by speedgoose 3y ago
Right. Gemini Pro 1.5 scores 81.9% on MMLU and is also above in a few other benchmarks.
- onlyrealcuzzo 3y agoWhich - importantly - is better than Mistral at 81.2% ... Gemini Ultra scored 90% which is better than GPT-4. This reads like a paid-for press release from Microsoft to pretend like they're almighty and Google is incompetent.
- 7moritz7 3y agoDidn't Gemini 1.0 Ultra turn out to be worse than GPT 4 Turbo in practice? Synthetic benchmarks aren't a perfect representation of anything.
- maleldil 3y agoI think they're close enough that it depends on your use case. IME, which is mostly generating Python and shell code, Gemini Advanced is faster, less lazy and generates overall better solutions than GPT-4. It also seems to be more up-to-date with libraries and stuff. It can also directly go to URLs (e.g. summarise a paper), which GPT-4 refuses to do.
- Nidhug 3y agoDoesn't MMLU have like 2-3% wrong answers anyways ?
- jug 3y agoWith 128K context (1M paid) compared to 32K. Man, Google is going to be a game changer for especially free AI tiers. Edit: BTW, more Mistral benchmarks here: https://docs.mistral.ai/platform/endpoints/ https://docs.mistral.ai/platform/endpoints/ TIL Mistral Small outperforms Mixtral 8x7B.