4 ms·

Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is

by nomilk 2y ago

Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is wrong much more often, meaning Gemini power users are at a significant disadvantage relative to if they used ChatGPT instead. I know the benchmarks say they’re vaguely similar, but benchmarks said the same of Llama II, and it was almost useless compared to ChatGPT. (so I don’t quite trust benchmarks, more interested in how these tools perform in real use cases)

abdullin 2y ago
It all depends on the benchmark and the use case. If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark. Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.
- benterix 2y ago
  > It all depends on the benchmark and the use case. That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.
  - abdullin 2y ago
    Yep, exactly! My approach is similar - closed source benchmarks with prompts and tests from real LLM-driven products (mostly around boring business automation and enterprise workflows). Although it would be neat to upgrade the setup to work on the synthetic data. This will at least make benchmarks shareable publicly (not just the results)
davidguetta 2y ago
I only use gpt4 but to be fair ive read that some people prefer gemini for coding tasks. Ive used gemini only for vision tasks on document and it was absolute CATASTROPHY (hallucinating the content instead of actually reading it) The woke part is also extra unhinged on gemini (hating white people is fine but hating other is racist etc...) Thats not the point of my message tho, the problems could be corrected later. Its more about the product vision

by nomilk 2y ago

abdullin 2y ago

It all depends on the benchmark and the use case. If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark. Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.

benterix 2y ago
> It all depends on the benchmark and the use case. That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.
- abdullin 2y ago
  Yep, exactly! My approach is similar - closed source benchmarks with prompts and tests from real LLM-driven products (mostly around boring business automation and enterprise workflows). Although it would be neat to upgrade the setup to work on the synthetic data. This will at least make benchmarks shareable publicly (not just the results)

davidguetta 2y ago

I only use gpt4 but to be fair ive read that some people prefer gemini for coding tasks. Ive used gemini only for vision tasks on document and it was absolute CATASTROPHY (hallucinating the content instead of actually reading it) The woke part is also extra unhinged on gemini (hating white people is fine but hating other is racist etc...) Thats not the point of my message tho, the problems could be corrected later. Its more about the product vision