3 ms·

I don't know, feels like a paper titled "Is Google Getting Worse" could have benefited from actually looking at Google results rather than only results of other

by jsnell 3y ago

I don't know, feels like a paper titled "Is Google Getting Worse" could have benefited from actually looking at Google results rather than only results of other search engines. Edit: This got downvoted to hell, so let me be more explicit. This study did not look at Google results, the title is pure clickbait. They used Startpage results as a proxy for Google results. I don't think that's a valid assumption, even if Startpage is using Google's index.

deleted 3y ago
[deleted]
j2kun 3y ago
Using a reasonable proxy is not "pure clickbait", though it may be misleading. It adds an additional assumption about the Google results not being tampered with by Startpage, but it seems like a reasonable one, compared to the alternative of Google doctoring the search results when detecting a scraper and/or doing personalization. If the authors have done their due diligence and confirmed the results from Startpage are actually Google results, then I don't see why they couldn't claim so in their title.
- jsnell 3y ago
  The authors did not do any work to establish it as a reasonable proxy. We know that's the case, because they don't describe any of that work in the paper. they just state that they're using this proxy measure, and then seem to be using Startpage and Google interchangably for the rest of the paper. Sure, if they'd actually done the searches on Google, the title would not be outright dishonest. But they didn't, so this defense of yours is just absurd. It's pretty obvious why the results won't be the same: the feature sets of the search engines are different despite being based on the same index. Startpage's own documentation even has a page on how and why the results are different! (Personalization is a part of the Google feature set, and has to be taken into account when considering the results. It's also a part of the Bing feature set. This didn't stop them from reporting the results of their testing on Bing in detail.)
  - phoerious 3y ago
    Hi, I'm lead author of the study, so I can give some background on this. This is certainly valid criticism and we could have addressed our reasoning for using Startpage in more detail, but it just didn't fit in the paper anymore. The reason we used Startpage is simply that it's much easier to scrape. We started off checking only Startpage (as proxy for Google) and DuckDuckGo (as proxy for Bing), since they are both simple to scrape and produce stable rankings. Plain Bing, on the other hand, is a lot trickier. You often get different results for the same query and you get blocked much more easily if you send too many. The only stable way to use Bing is via the (certainly not very cheap) API, though even that wouldn't necessarily guarantee the same user experience as the web frontend. We did notice, however, that DDG (despite being mostly Bing) did deviate quite a bit in their results, so we started scraping Bing as well for a fairer comparison. As for Startpage, we did check it initially and we found the results to be virtually identical, except for a few minor rank differences here and there (which are probably just geo personalisation). The differences may have become larger now that Startpage also taps Bing to a certain amount, though when I do spot checks, the results are still sufficiently similar and they are also sufficiently different from what actual Bing gives you. Most prominently, Startpage/Google give you a lot more YouTube results, which we did another small spin-off study on (to be published at CHIIR this year https://downloads.webis.de/publications/papers/bevendorff_2024c.pdf https://downloads.webis.de/publications/papers/bevendorff_20...). Moreover, we could also measure certain immediate effects of Google's ranker updates in Startpage, which weren't as apparent in Bing. So we are confident that Startpage is a reasonable proxy for Google, though it's certainly something we will keep in mind for follow-up studies.
    - jsnell 3y ago
      Hey, I'm sure that using these sites was the thing for your research for all kinds of practical reasons, and you don't need to justify that :) My actual complaint was about the misleading title, which you did not address. Note that the clickbait title actually hijacked your research; there's like three comments out of 240 here that are about your paper. So while it probably feels good to have received all this attention, it's just bogus engagement. I guess it's pretty meta for a paper about search result spam to do blackhat social media engagement optimization though.

by jsnell 3y ago

deleted 3y ago

[deleted]

j2kun 3y ago

Using a reasonable proxy is not "pure clickbait", though it may be misleading. It adds an additional assumption about the Google results not being tampered with by Startpage, but it seems like a reasonable one, compared to the alternative of Google doctoring the search results when detecting a scraper and/or doing personalization. If the authors have done their due diligence and confirmed the results from Startpage are actually Google results, then I don't see why they couldn't claim so in their title.

jsnell 3y ago
The authors did not do any work to establish it as a reasonable proxy. We know that's the case, because they don't describe any of that work in the paper. they just state that they're using this proxy measure, and then seem to be using Startpage and Google interchangably for the rest of the paper. Sure, if they'd actually done the searches on Google, the title would not be outright dishonest. But they didn't, so this defense of yours is just absurd. It's pretty obvious why the results won't be the same: the feature sets of the search engines are different despite being based on the same index. Startpage's own documentation even has a page on how and why the results are different! (Personalization is a part of the Google feature set, and has to be taken into account when considering the results. It's also a part of the Bing feature set. This didn't stop them from reporting the results of their testing on Bing in detail.)
- phoerious 3y ago
  Hi, I'm lead author of the study, so I can give some background on this. This is certainly valid criticism and we could have addressed our reasoning for using Startpage in more detail, but it just didn't fit in the paper anymore. The reason we used Startpage is simply that it's much easier to scrape. We started off checking only Startpage (as proxy for Google) and DuckDuckGo (as proxy for Bing), since they are both simple to scrape and produce stable rankings. Plain Bing, on the other hand, is a lot trickier. You often get different results for the same query and you get blocked much more easily if you send too many. The only stable way to use Bing is via the (certainly not very cheap) API, though even that wouldn't necessarily guarantee the same user experience as the web frontend. We did notice, however, that DDG (despite being mostly Bing) did deviate quite a bit in their results, so we started scraping Bing as well for a fairer comparison. As for Startpage, we did check it initially and we found the results to be virtually identical, except for a few minor rank differences here and there (which are probably just geo personalisation). The differences may have become larger now that Startpage also taps Bing to a certain amount, though when I do spot checks, the results are still sufficiently similar and they are also sufficiently different from what actual Bing gives you. Most prominently, Startpage/Google give you a lot more YouTube results, which we did another small spin-off study on (to be published at CHIIR this year https://downloads.webis.de/publications/papers/bevendorff_2024c.pdf https://downloads.webis.de/publications/papers/bevendorff_20...). Moreover, we could also measure certain immediate effects of Google's ranker updates in Startpage, which weren't as apparent in Bing. So we are confident that Startpage is a reasonable proxy for Google, though it's certainly something we will keep in mind for follow-up studies.
  - jsnell 3y ago
    Hey, I'm sure that using these sites was the thing for your research for all kinds of practical reasons, and you don't need to justify that :) My actual complaint was about the misleading title, which you did not address. Note that the clickbait title actually hijacked your research; there's like three comments out of 240 here that are about your paper. So while it probably feels good to have received all this attention, it's just bogus engagement. I guess it's pretty meta for a paper about search result spam to do blackhat social media engagement optimization though.