5 ms·

They compare to Gemini Pro 1.0... Seems intentionally misleading.

by onlyrealcuzzo 3y ago

speedgoose 3y ago
Right. Gemini Pro 1.5 scores 81.9% on MMLU and is also above in a few other benchmarks.
- onlyrealcuzzo 3y ago
  Which - importantly - is better than Mistral at 81.2% ... Gemini Ultra scored 90% which is better than GPT-4. This reads like a paid-for press release from Microsoft to pretend like they're almighty and Google is incompetent.
  - 7moritz7 3y ago
    Didn't Gemini 1.0 Ultra turn out to be worse than GPT 4 Turbo in practice? Synthetic benchmarks aren't a perfect representation of anything.
    - maleldil 3y ago
      I think they're close enough that it depends on your use case. IME, which is mostly generating Python and shell code, Gemini Advanced is faster, less lazy and generates overall better solutions than GPT-4. It also seems to be more up-to-date with libraries and stuff. It can also directly go to URLs (e.g. summarise a paper), which GPT-4 refuses to do.
- Nidhug 3y ago
  Doesn't MMLU have like 2-3% wrong answers anyways ?
- jug 3y ago
  With 128K context (1M paid) compared to 32K. Man, Google is going to be a game changer for especially free AI tiers. Edit: BTW, more Mistral benchmarks here: https://docs.mistral.ai/platform/endpoints/ https://docs.mistral.ai/platform/endpoints/ TIL Mistral Small outperforms Mixtral 8x7B.
cyrusradfar 3y ago
1.5 was probably released too late to be tested.
- miohtama 3y ago
  Is 1.5 even publicly available yet?
  - jensensbutton 3y ago
    Yes
    - renewiltord 3y ago
      How do I use it? I only have access to 1.0 Ultra through Gemini Advanced.
      - irthomasthomas 3y ago
        Me too, I don't think it's public yet.
      - mewpmewp2 3y ago
        And that doesn't count. Only pro 1.0 available for me through APIs. I need to be able to test for myself the capabilities. As it stands best LLM available by API by Google is far behind GPT4.
    - klohto 3y ago
      it’s not public, there is waitlist
- vy007vikas 3y ago
  But Ultra 1.0 is available to compare against, right?
  - mewpmewp2 3y ago
    How can I get access to it? The best I can compare against is pro 1.0. Most of my usecases are logic based on embedded content in the prompt and nothing available to me beats GPT-4 there.
lox 3y ago
From the article: > generally available through an API (next to GPT-4)

They compare to Gemini Pro 1.0... Seems intentionally misleading.

speedgoose 3y ago

Right. Gemini Pro 1.5 scores 81.9% on MMLU and is also above in a few other benchmarks.

onlyrealcuzzo 3y ago
Which - importantly - is better than Mistral at 81.2% ... Gemini Ultra scored 90% which is better than GPT-4. This reads like a paid-for press release from Microsoft to pretend like they're almighty and Google is incompetent.
- 7moritz7 3y ago
  Didn't Gemini 1.0 Ultra turn out to be worse than GPT 4 Turbo in practice? Synthetic benchmarks aren't a perfect representation of anything.
  - maleldil 3y ago
    I think they're close enough that it depends on your use case. IME, which is mostly generating Python and shell code, Gemini Advanced is faster, less lazy and generates overall better solutions than GPT-4. It also seems to be more up-to-date with libraries and stuff. It can also directly go to URLs (e.g. summarise a paper), which GPT-4 refuses to do.
Nidhug 3y ago
Doesn't MMLU have like 2-3% wrong answers anyways ?
jug 3y ago
With 128K context (1M paid) compared to 32K. Man, Google is going to be a game changer for especially free AI tiers. Edit: BTW, more Mistral benchmarks here: https://docs.mistral.ai/platform/endpoints/ https://docs.mistral.ai/platform/endpoints/ TIL Mistral Small outperforms Mixtral 8x7B.

cyrusradfar 3y ago

1.5 was probably released too late to be tested.

miohtama 3y ago
Is 1.5 even publicly available yet?
- jensensbutton 3y ago
  Yes
  - renewiltord 3y ago
    How do I use it? I only have access to 1.0 Ultra through Gemini Advanced.
    - irthomasthomas 3y ago
      Me too, I don't think it's public yet.
    - mewpmewp2 3y ago
      And that doesn't count. Only pro 1.0 available for me through APIs. I need to be able to test for myself the capabilities. As it stands best LLM available by API by Google is far behind GPT4.
  - klohto 3y ago
    it’s not public, there is waitlist
vy007vikas 3y ago
But Ultra 1.0 is available to compare against, right?
- mewpmewp2 3y ago
  How can I get access to it? The best I can compare against is pro 1.0. Most of my usecases are logic based on embedded content in the prompt and nothing available to me beats GPT-4 there.

lox 3y ago

From the article: > generally available through an API (next to GPT-4)