3 ms·

CEO of Google Says It Has No Solution for Its AI Providing Incorrect Info

27 points by ColinWright 2y ago

ggm 2y ago
So.. turn it off. It's not going to increase revenue or even maintain it, instead it will undermine core brand value.
- ugjka 2y ago
  Someone will get hurt and sue
- greg_V 2y ago
  Wait until Monday and see how the stock reacts
  - grumpopotamus 2y ago
    I predict that the stock will not react at all on Monday.
    - BXlnt2EachOther 2y ago
      probably a bit of a "that's the joke" question, but is that due to the market being closed Monday? or is that it didn't adjust on Friday and this isn't significant enough to adjust the next time the market is open (Tuesday)?
      - grumpopotamus 2y ago
        Yes, I was making a joke about Monday being a trading holiday.
    - rsynnott 2y ago
      Alphabet is quoted on some non-US markets, so it is tradeable today.
- alephnerd 2y ago
  This is why GOOG didn't release a dedicated LLM product earlier despite building the core models for it almost a decade ago (eg. BERT). This is also why MS and other companies are leveraging OpenAI, as they can offload liability onto them. Emily Bender is correct about how LLMs are just "extruded text machines".
  - grugagag 2y ago
    Google should have released a disableable feature leveraging Llms for whoever wanted to use it not shove it in their search results. Perhaps Pichai needs to step down and allow a more capable CEO take over.
    - alephnerd 2y ago
      > Perhaps Pichai needs to step down and allow a more capable CEO take over. At a Google scale company (and any public company for that matter), CEO's job is not to drive product decisions. That comes down to product leadership. The CEO's job is longer term strategy (as in these are the types of categories we need to build features for, these are the the kinds of customers we need to target, these are the core financial metrics we need to hit). > Google should have released a disableable feature leveraging Llms for whoever wanted to use it not shove it in their search results The PM that owned this feature definitely has UX A/B tested it and found that the majority of users just don't care. For example, I don't see auto-generated LLM results when searching on mobile which is the majority of Google Search impressions.
- rsynnott 2y ago
  I mean, I assume that once the fad dies away a bit, they will. _Wouldn’t exactly be the first time, for Google._ However, today, the markets, which can remain irrational longer than anyone can remain solvent, would punish Google for this; it has to go along pretending this is a real thing for a bit longer.
jqpabc123 2y ago
A LLM provides a statistical approximation that sometimes appears intelligent --- and sometimes not. The question is, do the results justify the cost?
- coldtea 2y ago
  Instead of it being used as an "oracle" / "encyclopedia" it should beused as a glorified reference / helper system, with varieties optimized in different domains (say Java coding, history, and so on) as part of a bigger system with built-in guards
- returnInfinity 2y ago
  And according to Ilya, when this statistical approximation gets crazy good, its equivalent to AGI. AFAIK! We need Ilya himself to confirm again.
bell-cot 2y ago
BSaaS? Judging by how much crap there already is on the web these days, there would seem to be a market for it.
- mattashii 2y ago
  I think it is less of a service than a disservice (I certainly didn't ask, want, or expect misinformation in my search engine results), so BSaaDS would be more appropriate.
shortsunblack 2y ago
Should have weathered the storm and realized LLM craze is at its peak, and will die down. We are at Juicero juicing phase. Instead, they decided to kill their brand value.
- abdullin 2y ago
  Most of the blame that LLMs get is because they are used wrong. LLM is a good information transformation engine. But if I supply it with messy or wrong information in the context, then I’ll end up with hallucinations. This is why most of the AI systems tend to produce junk. LLM part is good, but the underlying system just provides it with noisy and meaningless data.
danielovichdk 2y ago
I believe that over time OpenAI will eat a considerate amount of Googles Search users. Search as we have known it for 25 years is becoming irrelevant because of LLMs and I don't think Google will be the one to counteract OpenAI. No search users no ad income. LMMs has totally removed the use for ads for revenue. Its amazing how well Microsoft and OpenAI has created a brand new way to interact with the Internet. I actually believe that Google will not be around as we know it in 10-15 years. And I do think OpenAI and Microsoft is playing a major part of that.
aldarisbm 2y ago
Wait but when did he say that?
davidguetta 2y ago
I think they are actually being too timid. ChatGPT is already replacing search in many professions starting with SWE. IMO they should embrace it 100% and make Gemini a full 50% of the search page or a new button. Just making it a sad widget gives the signal its not the place to be to get AI ar all, not to mentio the subpar results. Maybe sergey and larry are not taking the reins now. Sundar was a good peacetime CEO but maybe Google need some deep reorg of their product which only an original foubder can have the credential to lead
- nomilk 2y ago
  Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is wrong much more often, meaning Gemini power users are at a significant disadvantage relative to if they used ChatGPT instead. I know the benchmarks say they’re vaguely similar, but benchmarks said the same of Llama II, and it was almost useless compared to ChatGPT. (so I don’t quite trust benchmarks, more interested in how these tools perform in real use cases)
  - abdullin 2y ago
    It all depends on the benchmark and the use case. If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark. Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.
    - benterix 2y ago
      > It all depends on the benchmark and the use case. That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.
dangtimewaste 2y ago
[dead]
ChrisArchitect 2y ago
[dupe] More discussion: https://news.ycombinator.com/item?id=40468436 https://news.ycombinator.com/item?id=40468436 (Actual article https://news.ycombinator.com/item?id=40415846 https://news.ycombinator.com/item?id=40415846) https://news.ycombinator.com/item?id=40448074 https://news.ycombinator.com/item?id=40448074 https://news.ycombinator.com/item?id=40461580 https://news.ycombinator.com/item?id=40461580
djaouen 2y ago
Wow, AI sure is turning out great! /s
rezonant 2y ago
My problem with this is that clearly a big portion of AI Overview's issue is bad training. That is a problem with a straightforward solution. A lot of the publicized cases of bad information actually do have a source on the web, indicating they are not hallucinations.

27 points by ColinWright 2y ago

ggm 2y ago

So.. turn it off. It's not going to increase revenue or even maintain it, instead it will undermine core brand value.

ugjka 2y ago
Someone will get hurt and sue
greg_V 2y ago
Wait until Monday and see how the stock reacts
- grumpopotamus 2y ago
  I predict that the stock will not react at all on Monday.
  - BXlnt2EachOther 2y ago
    probably a bit of a "that's the joke" question, but is that due to the market being closed Monday? or is that it didn't adjust on Friday and this isn't significant enough to adjust the next time the market is open (Tuesday)?
    - grumpopotamus 2y ago
      Yes, I was making a joke about Monday being a trading holiday.
  - rsynnott 2y ago
    Alphabet is quoted on some non-US markets, so it is tradeable today.
alephnerd 2y ago
This is why GOOG didn't release a dedicated LLM product earlier despite building the core models for it almost a decade ago (eg. BERT). This is also why MS and other companies are leveraging OpenAI, as they can offload liability onto them. Emily Bender is correct about how LLMs are just "extruded text machines".
- grugagag 2y ago
  Google should have released a disableable feature leveraging Llms for whoever wanted to use it not shove it in their search results. Perhaps Pichai needs to step down and allow a more capable CEO take over.
  - alephnerd 2y ago
    > Perhaps Pichai needs to step down and allow a more capable CEO take over. At a Google scale company (and any public company for that matter), CEO's job is not to drive product decisions. That comes down to product leadership. The CEO's job is longer term strategy (as in these are the types of categories we need to build features for, these are the the kinds of customers we need to target, these are the core financial metrics we need to hit). > Google should have released a disableable feature leveraging Llms for whoever wanted to use it not shove it in their search results The PM that owned this feature definitely has UX A/B tested it and found that the majority of users just don't care. For example, I don't see auto-generated LLM results when searching on mobile which is the majority of Google Search impressions.
rsynnott 2y ago
I mean, I assume that once the fad dies away a bit, they will. _Wouldn’t exactly be the first time, for Google._ However, today, the markets, which can remain irrational longer than anyone can remain solvent, would punish Google for this; it has to go along pretending this is a real thing for a bit longer.

jqpabc123 2y ago

A LLM provides a statistical approximation that sometimes appears intelligent --- and sometimes not. The question is, do the results justify the cost?

coldtea 2y ago
Instead of it being used as an "oracle" / "encyclopedia" it should beused as a glorified reference / helper system, with varieties optimized in different domains (say Java coding, history, and so on) as part of a bigger system with built-in guards
returnInfinity 2y ago
And according to Ilya, when this statistical approximation gets crazy good, its equivalent to AGI. AFAIK! We need Ilya himself to confirm again.

bell-cot 2y ago

BSaaS? Judging by how much crap there already is on the web these days, there would seem to be a market for it.

mattashii 2y ago
I think it is less of a service than a disservice (I certainly didn't ask, want, or expect misinformation in my search engine results), so BSaaDS would be more appropriate.

shortsunblack 2y ago

Should have weathered the storm and realized LLM craze is at its peak, and will die down. We are at Juicero juicing phase. Instead, they decided to kill their brand value.

abdullin 2y ago
Most of the blame that LLMs get is because they are used wrong. LLM is a good information transformation engine. But if I supply it with messy or wrong information in the context, then I’ll end up with hallucinations. This is why most of the AI systems tend to produce junk. LLM part is good, but the underlying system just provides it with noisy and meaningless data.

danielovichdk 2y ago

I believe that over time OpenAI will eat a considerate amount of Googles Search users. Search as we have known it for 25 years is becoming irrelevant because of LLMs and I don't think Google will be the one to counteract OpenAI. No search users no ad income. LMMs has totally removed the use for ads for revenue. Its amazing how well Microsoft and OpenAI has created a brand new way to interact with the Internet. I actually believe that Google will not be around as we know it in 10-15 years. And I do think OpenAI and Microsoft is playing a major part of that.

aldarisbm 2y ago

Wait but when did he say that?

davidguetta 2y ago

I think they are actually being too timid. ChatGPT is already replacing search in many professions starting with SWE. IMO they should embrace it 100% and make Gemini a full 50% of the search page or a new button. Just making it a sad widget gives the signal its not the place to be to get AI ar all, not to mentio the subpar results. Maybe sergey and larry are not taking the reins now. Sundar was a good peacetime CEO but maybe Google need some deep reorg of their product which only an original foubder can have the credential to lead

nomilk 2y ago
Not meant as a slight or troll, is gemini anywhere near ChatGPT yet? My understanding from limited experience is that they function very similarly but Gemini is wrong much more often, meaning Gemini power users are at a significant disadvantage relative to if they used ChatGPT instead. I know the benchmarks say they’re vaguely similar, but benchmarks said the same of Llama II, and it was almost useless compared to ChatGPT. (so I don’t quite trust benchmarks, more interested in how these tools perform in real use cases)
- abdullin 2y ago
  It all depends on the benchmark and the use case. If we are talking about the ability of models to follow instructions and carry out concrete tasks (as in products or inside RAG systems), then Gemini Pro 1.5 is currently on the eighth place in our benchmark. Academic benchmarks, HF Leaderboards or LMSYS Chat arena will have different numbers.
  - benterix 2y ago
    > It all depends on the benchmark and the use case. That's why I have my own set of simple benchmarks that I'm not going to publish. Everybody can easily prepare such a set - in my case they are various programming tasks that should generate determined output. It is not easy to automatically qualify the quality of code, but at the very least you can filter the results by invalid outputs. With reasonably high number of tasks and their complexity, this can be a fair estimator - provided that it's never published publicly.

dangtimewaste 2y ago

[dead]

ChrisArchitect 2y ago

[dupe] More discussion: https://news.ycombinator.com/item?id=40468436 https://news.ycombinator.com/item?id=40468436 (Actual article https://news.ycombinator.com/item?id=40415846 https://news.ycombinator.com/item?id=40415846) https://news.ycombinator.com/item?id=40448074 https://news.ycombinator.com/item?id=40448074 https://news.ycombinator.com/item?id=40461580 https://news.ycombinator.com/item?id=40461580

djaouen 2y ago

Wow, AI sure is turning out great! /s

rezonant 2y ago

My problem with this is that clearly a big portion of AI Overview's issue is bad training. That is a problem with a straightforward solution. A lot of the publicized cases of bad information actually do have a source on the web, indicating they are not hallucinations.