5 ms·

A couple passages of note: > 112. The integration of generative AI is perhaps the clearest example of competition advancing search quality. Google accelerated

by jd3 2y ago

A couple passages of note: > 112. The integration of generative AI is perhaps the clearest example of competition advancing search quality. Google accelerated and launched its public piloting of Bard one day before Microsoft announced BingChat, the integration of ChatGPT’s generative AI technology into Bing to deliver answers to queries. Id. at 8272:4-7 (Reid); id. at 2670:10–2671:9 (Parakhin). (describing BingChat). Perhaps a normative assertion on my part, but AI results have not "advanc[ed] search quality" by any metric that I am familiar with; in fact, AI results in Google mark the first time I have ever encountered incorrect or patently untrue information at the top of a Google query. > This quality-reduction experiment correlated with only a 0.66– 0.99% decline in global search revenue. UPX1082 at 294. In short, this study demonstrates that a significant quality depreciation by Google would not result in a significant loss of revenues.

larubbio 2y ago
I remember this article from 2017 about the google snippet for how long it takes to caramelize an onion. https://gizmodo.com/googles-algorithm-is-lying-to-you-about-onions-and-blam-1793057789 https://gizmodo.com/googles-algorithm-is-lying-to-you-about-... The gist of the story is the author wrote an article debunking the recipe myth that you only need 5 minutes. In their tests it was 25 - 45. Google snippet would report "about 5 minutes" and link to the article as a reference. I think the information summaries at the top of searches have been wrong for a while.
- pradn 2y ago
  It's certainly wrong, but its not out of place w/ the sheer number of recipes and blogs saying the same thing. It's parroting "common wisdom" which is wrong. It's not as egregious as some of the generative AI results spewed out by the lowest-cost-inference LLMs they hooked up to Google search a few months back.
  - nox101 2y ago
    It was still a common occurrence for google to display incorrect results that weren't common wisdom. I had tons of such experiences. One I remember was searching for "healthy body weight for x of age y" where x was man,woman and y was my age. Google said 50lbs higher than I expected. I clicked through to the article. The article said the average was this high number, not a healthy weight.
    - pradn 2y ago
      Yes, totally. Automated, fallible systems. Ultimately it's a cost issue. If they wanted to, they could fact check the top million question queries - but they don't want to spend on that.
      - kevindamm 2y ago
        How familiar are you with the knowledge graph update process? There is actually a significant amount of spend going into reviewing the wrong answers to top queries and updating to fix/tweak them.
        pradn 2y ago
        Yes, there's humans in the loop for the knowledge graph. I'm unsure if snippets are reviewed in the same way.
  - refulgentis 2y ago
    Your assumption basis is completely wrong, not only on when it came about. Bizarrely, Google spelled out that the AI answers weren't from an LLM in post-debacle PR, like it was a good thing
- daedrdev 2y ago
  I think I actually ran into that exact search issue and was confused why my onions were not cooking well, lmao
fshbbdssbbgdd 2y ago
The idea that a 0.66% decline in revenue would be insignificant made me snort. Increasing revenue that much would make many careers - it would justify the existence of a substantial origination. That’s billions of dollars annually.
- vdfs 2y ago
  "It's only a $2B decline"
- singron 2y ago
  In a very competitive market, reducing quality slightly could drastically reduce revenue. E.g. if there were two brands of apple at the grocery store and one started putting a tiny worm in each apple, you would expect the worm apples to lose nearly 100% of their revenue.
  - otterley 2y ago
    That's an interesting example of a "slight" quality reduction. In apples, a slight quality reduction might be a bit of discoloration or a small bruise. The presence of a worm, on the other hand, causes the apple to be perceived as unsafe or inedible.
    - seoulmetro 2y ago
      Yeah, a worm in Apple the company would be their phones blowing up randomly.
  - deleted 2y ago
    [deleted]
asddubs 2y ago
google had plenty of incorrect information at the top of the page before then, just instead of AI it was some algorithm pulling sentence fragments out of a page where the context was often the opposite of what you're asking
Sakos 2y ago
I think there is a clearly anti-competitive component in this, demonstrated by the recent AI training deal between Reddit and Google. The one that went hand in hand with preventing any search engine but Google from indexing Reddit: https://news.ycombinator.com/item?id=41057033 https://news.ycombinator.com/item?id=41057033
jcfrei 2y ago
Google's main game is not serving better and better results: It's generating more clicks for advertisers on the first couple of results. The antithesis to Google's business model is someone asking an AI a question and receiving a bunch of direct answers for a monthly fee. Google's business model is people trying a bunch of queries (essentially shopping around), clicking on a whole set of links (each time costing the advertiser money without leading to a sale) until they get what they were looking for. It's the modern equivalent of confusing supermarket layouts with all the essential products stowed away in the back.
- ajross 2y ago
  (So, disclosure that I work at Google, but nowhere near anything to do with search or ads.) That is wildly distorting the way online advertisement works, mostly by a sideways denial that "advertising works" at all. The business model is that people who sell "X" want a link to their product displayed when potential customers consume content related to "X". There's nothing unclear or weird about this, and it's not fundamentally any different form print ads in the 1930's. So to the extent that "Google's main game is not serving better results", that's true, in the same way that Apple's game is not selling a better mobile operating system. iOS and search both generate revenue by making the product they're actually selling more effective.
  - ameister14 2y ago
    Totally agree with you about online ads, I worked in that space for years and that's not the goal with Ads. Years ago it became clear that Google was shifting towards taking over the lower level advertiser markets and providing services and automated systems to use Ads without requiring a marketing agency while providing higher levels of data analysis and tools for agencies to tap that market. Has nothing to do with stealing from advertisers, and a lot of Google's internal programs will likely drive more money to these smaller players. > iOS and search both generate revenue by making the product they're actually selling more effective In the abstract, sure. The problem starts with separating financial incentives internally from product improvement and aligning it with short term revenue increases. Then you start separating the financial side from all other aspects of the business entirely, a la Jack Welch. That's where a lot of major US corporations are right now. When that happens, you end up with a conflict between what is best for the company and what is best for the people running the company, and you end up with product quality decreasing. Right now from an outside perspective it looks like there was an internal conflict in the Search team at Google and the more product oriented leaders were pushed out in favor of the finance guys over the last year or two, with a side effect being lower search quality.
mullingitover 2y ago
> AI results in Google mark the first time I have ever encountered incorrect or patently untrue information at the top of a Google query. Google search ads have been rife with scams for a while[1]. I think that counts as patently untrue information at the top of a google query. [1] https://searchengineland.com/google-search-ads-brands-fraud-436871 https://searchengineland.com/google-search-ads-brands-fraud-...
deleted 2y ago
[deleted]

by jd3 2y ago

larubbio 2y ago

I remember this article from 2017 about the google snippet for how long it takes to caramelize an onion. https://gizmodo.com/googles-algorithm-is-lying-to-you-about-onions-and-blam-1793057789 https://gizmodo.com/googles-algorithm-is-lying-to-you-about-... The gist of the story is the author wrote an article debunking the recipe myth that you only need 5 minutes. In their tests it was 25 - 45. Google snippet would report "about 5 minutes" and link to the article as a reference. I think the information summaries at the top of searches have been wrong for a while.

pradn 2y ago
It's certainly wrong, but its not out of place w/ the sheer number of recipes and blogs saying the same thing. It's parroting "common wisdom" which is wrong. It's not as egregious as some of the generative AI results spewed out by the lowest-cost-inference LLMs they hooked up to Google search a few months back.
- nox101 2y ago
  It was still a common occurrence for google to display incorrect results that weren't common wisdom. I had tons of such experiences. One I remember was searching for "healthy body weight for x of age y" where x was man,woman and y was my age. Google said 50lbs higher than I expected. I clicked through to the article. The article said the average was this high number, not a healthy weight.
  - pradn 2y ago
    Yes, totally. Automated, fallible systems. Ultimately it's a cost issue. If they wanted to, they could fact check the top million question queries - but they don't want to spend on that.
    - kevindamm 2y ago
      How familiar are you with the knowledge graph update process? There is actually a significant amount of spend going into reviewing the wrong answers to top queries and updating to fix/tweak them.
      - pradn 2y ago
        Yes, there's humans in the loop for the knowledge graph. I'm unsure if snippets are reviewed in the same way.
- refulgentis 2y ago
  Your assumption basis is completely wrong, not only on when it came about. Bizarrely, Google spelled out that the AI answers weren't from an LLM in post-debacle PR, like it was a good thing
daedrdev 2y ago
I think I actually ran into that exact search issue and was confused why my onions were not cooking well, lmao

fshbbdssbbgdd 2y ago

The idea that a 0.66% decline in revenue would be insignificant made me snort. Increasing revenue that much would make many careers - it would justify the existence of a substantial origination. That’s billions of dollars annually.

vdfs 2y ago
"It's only a $2B decline"
singron 2y ago
In a very competitive market, reducing quality slightly could drastically reduce revenue. E.g. if there were two brands of apple at the grocery store and one started putting a tiny worm in each apple, you would expect the worm apples to lose nearly 100% of their revenue.
- otterley 2y ago
  That's an interesting example of a "slight" quality reduction. In apples, a slight quality reduction might be a bit of discoloration or a small bruise. The presence of a worm, on the other hand, causes the apple to be perceived as unsafe or inedible.
  - seoulmetro 2y ago
    Yeah, a worm in Apple the company would be their phones blowing up randomly.
- deleted 2y ago
  [deleted]

asddubs 2y ago

google had plenty of incorrect information at the top of the page before then, just instead of AI it was some algorithm pulling sentence fragments out of a page where the context was often the opposite of what you're asking

Sakos 2y ago

I think there is a clearly anti-competitive component in this, demonstrated by the recent AI training deal between Reddit and Google. The one that went hand in hand with preventing any search engine but Google from indexing Reddit: https://news.ycombinator.com/item?id=41057033 https://news.ycombinator.com/item?id=41057033

jcfrei 2y ago

Google's main game is not serving better and better results: It's generating more clicks for advertisers on the first couple of results. The antithesis to Google's business model is someone asking an AI a question and receiving a bunch of direct answers for a monthly fee. Google's business model is people trying a bunch of queries (essentially shopping around), clicking on a whole set of links (each time costing the advertiser money without leading to a sale) until they get what they were looking for. It's the modern equivalent of confusing supermarket layouts with all the essential products stowed away in the back.

ajross 2y ago
(So, disclosure that I work at Google, but nowhere near anything to do with search or ads.) That is wildly distorting the way online advertisement works, mostly by a sideways denial that "advertising works" at all. The business model is that people who sell "X" want a link to their product displayed when potential customers consume content related to "X". There's nothing unclear or weird about this, and it's not fundamentally any different form print ads in the 1930's. So to the extent that "Google's main game is not serving better results", that's true, in the same way that Apple's game is not selling a better mobile operating system. iOS and search both generate revenue by making the product they're actually selling more effective.
- ameister14 2y ago
  Totally agree with you about online ads, I worked in that space for years and that's not the goal with Ads. Years ago it became clear that Google was shifting towards taking over the lower level advertiser markets and providing services and automated systems to use Ads without requiring a marketing agency while providing higher levels of data analysis and tools for agencies to tap that market. Has nothing to do with stealing from advertisers, and a lot of Google's internal programs will likely drive more money to these smaller players. > iOS and search both generate revenue by making the product they're actually selling more effective In the abstract, sure. The problem starts with separating financial incentives internally from product improvement and aligning it with short term revenue increases. Then you start separating the financial side from all other aspects of the business entirely, a la Jack Welch. That's where a lot of major US corporations are right now. When that happens, you end up with a conflict between what is best for the company and what is best for the people running the company, and you end up with product quality decreasing. Right now from an outside perspective it looks like there was an internal conflict in the Search team at Google and the more product oriented leaders were pushed out in favor of the finance guys over the last year or two, with a side effect being lower search quality.

mullingitover 2y ago

> AI results in Google mark the first time I have ever encountered incorrect or patently untrue information at the top of a Google query. Google search ads have been rife with scams for a while[1]. I think that counts as patently untrue information at the top of a google query. [1] https://searchengineland.com/google-search-ads-brands-fraud-436871 https://searchengineland.com/google-search-ads-brands-fraud-...

deleted 2y ago

[deleted]