4 ms·

I think they are actually being too timid. ChatGPT is already replacing search in many professions starting with SWE. IMO they should embrace it 100% and make

I think they are actually being too timid. ChatGPT is already replacing search in many professions starting with SWE. IMO they should embrace it 100% and make Gemini a full 50% of the search page or a new button. Just making it a sad widget gives the signal its not the place to be to get AI ar all, not to mentio the subpar results. Maybe sergey and larry are not taking the reins now. Sundar was a good peacetime CEO but maybe Google need some deep reorg of their product which only an original foubder can have the credential to lead

nomilk 2y ago
Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is wrong much more often, meaning Gemini power users are at a significant disadvantage relative to if they used ChatGPT instead. I know the benchmarks say they’re vaguely similar, but benchmarks said the same of Llama II, and it was almost useless compared to ChatGPT. (so I don’t quite trust benchmarks, more interested in how these tools perform in real use cases)
- abdullin 2y ago
  It all depends on the benchmark and the use case. If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark. Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.
  - benterix 2y ago
    > It all depends on the benchmark and the use case. That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.
    - abdullin 2y ago
      Yep, exactly! My approach is similar - closed source benchmarks with prompts and tests from real LLM-driven products (mostly around boring business automation and enterprise workflows). Although it would be neat to upgrade the setup to work on the synthetic data. This will at least make benchmarks shareable publicly (not just the results)
- davidguetta 2y ago
  I only use gpt4 but to be fair ive read that some people prefer gemini for coding tasks. Ive used gemini only for vision tasks on document and it was absolute CATASTROPHY (hallucinating the content instead of actually reading it) The woke part is also extra unhinged on gemini (hating white people is fine but hating other is racist etc...) Thats not the point of my message tho, the problems could be corrected later. Its more about the product vision
benterix 2y ago
Unless they split the screen into two, it makes no sense at all. These are two very different activities: when I search for "how to implement self-balancing binary search trees in C" it means one thing, when I search for "german history wiki" I mean something completely different and I don't need hallucinating chatbot standing on my way. The way for Google to beat OpenAI is to recognize AI chatbot usage is going to make a dent in web search usage and further develop Gemini to make it jump from the current GPT-3.5 to 4.0. When combined with a huge context window, they could quickly make people switch and own both web search and AI search. With the current strategy, they just make people migrate away from their web search.
- davidguetta 2y ago
  I dont disagree but i think they are currently SUPER LUCKY that open ai isnt going harder on making chatgpt available for more of the general public. There are still 60 70% people who havent tried it like my parents. They still have a window to stay relevant one way or the other
- rsynnott 2y ago
  > when I search for "how to implement self-balancing binary search trees in C" it means one thing I mean, even then you’d surely prefer to be pointed to an article by an actual human than getting whatever an LLM spits out?

nomilk 2y ago

Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is wrong much more often, meaning Gemini power users are at a significant disadvantage relative to if they used ChatGPT instead. I know the benchmarks say they’re vaguely similar, but benchmarks said the same of Llama II, and it was almost useless compared to ChatGPT. (so I don’t quite trust benchmarks, more interested in how these tools perform in real use cases)

abdullin 2y ago
It all depends on the benchmark and the use case. If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark. Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.
- benterix 2y ago
  > It all depends on the benchmark and the use case. That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.
  - abdullin 2y ago
    Yep, exactly! My approach is similar - closed source benchmarks with prompts and tests from real LLM-driven products (mostly around boring business automation and enterprise workflows). Although it would be neat to upgrade the setup to work on the synthetic data. This will at least make benchmarks shareable publicly (not just the results)
davidguetta 2y ago
I only use gpt4 but to be fair ive read that some people prefer gemini for coding tasks. Ive used gemini only for vision tasks on document and it was absolute CATASTROPHY (hallucinating the content instead of actually reading it) The woke part is also extra unhinged on gemini (hating white people is fine but hating other is racist etc...) Thats not the point of my message tho, the problems could be corrected later. Its more about the product vision

benterix 2y ago

Unless they split the screen into two, it makes no sense at all. These are two very different activities: when I search for "how to implement self-balancing binary search trees in C" it means one thing, when I search for "german history wiki" I mean something completely different and I don't need hallucinating chatbot standing on my way. The way for Google to beat OpenAI is to recognize AI chatbot usage is going to make a dent in web search usage and further develop Gemini to make it jump from the current GPT-3.5 to 4.0. When combined with a huge context window, they could quickly make people switch and own both web search and AI search. With the current strategy, they just make people migrate away from their web search.

davidguetta 2y ago
I dont disagree but i think they are currently SUPER LUCKY that open ai isnt going harder on making chatgpt available for more of the general public. There are still 60 70% people who havent tried it like my parents. They still have a window to stay relevant one way or the other
rsynnott 2y ago
> when I search for "how to implement self-balancing binary search trees in C" it means one thing I mean, even then you’d surely prefer to be pointed to an article by an actual human than getting whatever an LLM spits out?