7 ms·

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to restrict bots, it would be much easier to restrict googlebot because they actually follow the rules. I don't disagree in principle that there should be an open index of the web, but for once I don't see Google as a bad actor here.

zmarty 6y ago
A lot of news websites restrict any crawler other than Google. And this does not happen only via robots.txt.
- simias 6y ago
  Indeed, years ago I had scripts to automatically fetch URLs from IRC and I quickly realized that if I didn't spoof the user agent of a proper web browser many websites would reject the query. Googlebot's UA worked just fine however.
  - judge2020 6y ago
    > Googlebot's UA worked just fine however They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan). 0: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
throwaway_uat 6y ago
LinkedIn profile/Quora answer are accessible by Google bot without signin
malf 6y ago
What do you think this is used for? https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
calimac 6y ago
The studies and data to support their claim is in the first paragraph of the article you "read" before posting the question.
Lammy 6y ago
Spoofing your user-agent as googlebot is a common way to bypass paywalls, is (was?) a way to read Quora without creating an account, etc. Publishers obviously need to send their page/article to Google if they want it to be indexed but may not want to send the same page content to a normal user: https://www.256kilobytes.com/content/show/1934/spoofing-your-user-agent-as-googlebot-avoid-paywalls-circumvent-log-in-requirements-and-exploit-other-quality-loopholes https://www.256kilobytes.com/content/show/1934/spoofing-your... This was common even back in the mid-2000s: https://www.avivadirectory.com/bethebot/ https://www.avivadirectory.com/bethebot/ https://developers.google.com/search/blog/2006/09/how-to-verify-googlebot https://developers.google.com/search/blog/2006/09/how-to-ver...
soheil 6y ago
It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.
- deleted 6y ago
  [deleted]
- WesolyKubeczek 6y ago
  I don't scrape a website often, but when I do, I'm using a user agent of a major browser.
- Mauricebranagh 6y ago
  I have never had that problem running screaming frog on big brand sites apart from one or two times.
- dheera 6y ago
  Do any of them intersect with Google Cloud IP addresses? If so set up a VPN server on Google Cloud.
- tedunangst 6y ago
  I don't whitelist googlebot, but I don't block them either because their crawler is fairly slow and unobtrusive. Other crawlers seem determined to download the entire site in 60 seconds, and then download it again, and again, until they get banned.
- quitethelogic 6y ago
  > Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them. Google says[1] they do not do this: "Google doesn't post a public list of IP addresses for website owners to allowlist." [1]https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
suicas 6y ago
A company I worked for ~7 years ago ran its own focused web crawler (fetching ~10-100m pages per month, targeting certain sections of the web). There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time. We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was allowed to crawl them (though having no robots.txt configured to say so).
- luckylion 6y ago
  I usually recommend setting only Google/Bing/Yandex/Baidu etc to Allow and everything else to Disallow. Yes, the bad bots don't give a fuck, but even the non-malicious bots (ahrefs, moz, some university's search engine etc) don't bring any value to me as a site owner, take up band width and resources and fill up logs. If you can remove them with three lines in your robots.txt, that's less noise. Especially universities do, in my opinion, often behave badly and are uncooperative when you point out their throttling does not work and they're hammering your server. Giving them a "Go Away, You Are Not Wanted Here" in a robots.txt works for most, and the rest just gets blocked.
  - d110af5ccf 6y ago
    > they're hammering your server Why can't you just ratelimit IPs that are "too active" for your server to handle?
    - MinorTom 6y ago
      CGNAT
      - d110af5ccf 6y ago
        Now that's a really good point. I wonder why there isn't a standard protocol for signalling upstream that a particular connection is abusive and to please rate limit the path at the source on your behalf? It would certainly add complexity, but the current situation is hardly better.
burkaman 6y ago
See figure I.4 on page 24 of this UK government report: https://assets.publishing.service.gov.uk/media/5efb1db6e90e075c5674db35/Appendix_I_-_search_quality_v.3.pdf https://assets.publishing.service.gov.uk/media/5efb1db6e90e0... Additional evidence here: https://knuckleheads.club/the-evidence-we-found-so-far/ https://knuckleheads.club/the-evidence-we-found-so-far/
jameshart 6y ago
When you operate commercial sites at scale, bots are a real thing you spend real engineering hours thinking about and troubleshooting and coding to solve for. And yes, that means google gets special treatment. Think about the model for a site like stackoverflow. The longest of long tail questions on that site: what’s the actual lifecycle of that question? - posted by a random user - scraped by google, bing, et al - visited by someone who clicked on a search result on google - eventually, answered - hopefully, reindexed by google, bing et al - maybe never visited again because the answer now shows up on the google SERP In the lifetime of that question how many times is it accessed by a human, compared to the number of times it’s requested and rerequested by an indexing bot? What would be the impact on your site of three more bots as persistent as google bot? Why should you bother with their requests? So yes, sites care about bot traffic and they care about google in particular.
noxvilleza 6y ago
Google aren't the bad actor in the sense that they are actively doing something wrong, but they are definitely benefiting from the monopoly that they created and work on maintaining. If this continues then nobody will really ever be able to challenge them, which means possibly "better" products will fail to penetrate the market.
ehsankia 6y ago
> but for once I don't see Google as a bad actor here. As inflammatory as the headline of the page looks, they literally admit it's not google's fault in the smaller text lower down: "This isn’t illegal and it isn’t Google’s fault, but"

zmarty 6y ago

A lot of news websites restrict any crawler other than Google. And this does not happen only via robots.txt.

simias 6y ago
Indeed, years ago I had scripts to automatically fetch URLs from IRC and I quickly realized that if I didn't spoof the user agent of a proper web browser many websites would reject the query. Googlebot's UA worked just fine however.
- judge2020 6y ago
  > Googlebot's UA worked just fine however They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan). 0: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...

throwaway_uat 6y ago

LinkedIn profile/Quora answer are accessible by Google bot without signin

malf 6y ago

What do you think this is used for? https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...

calimac 6y ago

The studies and data to support their claim is in the first paragraph of the article you "read" before posting the question.

Lammy 6y ago

Spoofing your user-agent as googlebot is a common way to bypass paywalls, is (was?) a way to read Quora without creating an account, etc. Publishers obviously need to send their page/article to Google if they want it to be indexed but may not want to send the same page content to a normal user: https://www.256kilobytes.com/content/show/1934/spoofing-your-user-agent-as-googlebot-avoid-paywalls-circumvent-log-in-requirements-and-exploit-other-quality-loopholes https://www.256kilobytes.com/content/show/1934/spoofing-your... This was common even back in the mid-2000s: https://www.avivadirectory.com/bethebot/ https://www.avivadirectory.com/bethebot/ https://developers.google.com/search/blog/2006/09/how-to-verify-googlebot https://developers.google.com/search/blog/2006/09/how-to-ver...

soheil 6y ago

It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

deleted 6y ago
[deleted]
WesolyKubeczek 6y ago
I don't scrape a website often, but when I do, I'm using a user agent of a major browser.
Mauricebranagh 6y ago
I have never had that problem running screaming frog on big brand sites apart from one or two times.
dheera 6y ago
Do any of them intersect with Google Cloud IP addresses? If so set up a VPN server on Google Cloud.
tedunangst 6y ago
I don't whitelist googlebot, but I don't block them either because their crawler is fairly slow and unobtrusive. Other crawlers seem determined to download the entire site in 60 seconds, and then download it again, and again, until they get banned.
quitethelogic 6y ago
> Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them. Google says[1] they do not do this: "Google doesn't post a public list of IP addresses for website owners to allowlist." [1]https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...

suicas 6y ago

A company I worked for ~7 years ago ran its own focused web crawler (fetching ~10-100m pages per month, targeting certain sections of the web). There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time. We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was allowed to crawl them (though having no robots.txt configured to say so).

luckylion 6y ago
I usually recommend setting only Google/Bing/Yandex/Baidu etc to Allow and everything else to Disallow. Yes, the bad bots don't give a fuck, but even the non-malicious bots (ahrefs, moz, some university's search engine etc) don't bring any value to me as a site owner, take up band width and resources and fill up logs. If you can remove them with three lines in your robots.txt, that's less noise. Especially universities do, in my opinion, often behave badly and are uncooperative when you point out their throttling does not work and they're hammering your server. Giving them a "Go Away, You Are Not Wanted Here" in a robots.txt works for most, and the rest just gets blocked.
- d110af5ccf 6y ago
  > they're hammering your server Why can't you just ratelimit IPs that are "too active" for your server to handle?
  - MinorTom 6y ago
    CGNAT
    - d110af5ccf 6y ago
      Now that's a really good point. I wonder why there isn't a standard protocol for signalling upstream that a particular connection is abusive and to please rate limit the path at the source on your behalf? It would certainly add complexity, but the current situation is hardly better.

burkaman 6y ago

See figure I.4 on page 24 of this UK government report: https://assets.publishing.service.gov.uk/media/5efb1db6e90e075c5674db35/Appendix_I_-_search_quality_v.3.pdf https://assets.publishing.service.gov.uk/media/5efb1db6e90e0... Additional evidence here: https://knuckleheads.club/the-evidence-we-found-so-far/ https://knuckleheads.club/the-evidence-we-found-so-far/

jameshart 6y ago

When you operate commercial sites at scale, bots are a real thing you spend real engineering hours thinking about and troubleshooting and coding to solve for. And yes, that means google gets special treatment. Think about the model for a site like stackoverflow. The longest of long tail questions on that site: what’s the actual lifecycle of that question? - posted by a random user - scraped by google, bing, et al - visited by someone who clicked on a search result on google - eventually, answered - hopefully, reindexed by google, bing et al - maybe never visited again because the answer now shows up on the google SERP In the lifetime of that question how many times is it accessed by a human, compared to the number of times it’s requested and rerequested by an indexing bot? What would be the impact on your site of three more bots as persistent as google bot? Why should you bother with their requests? So yes, sites care about bot traffic and they care about google in particular.

noxvilleza 6y ago

Google aren't the bad actor in the sense that they are actively doing something wrong, but they are definitely benefiting from the monopoly that they created and work on maintaining. If this continues then nobody will really ever be able to challenge them, which means possibly "better" products will fail to penetrate the market.

ehsankia 6y ago

> but for once I don't see Google as a bad actor here. As inflammatory as the headline of the page looks, they literally admit it's not google's fault in the smaller text lower down: "This isn’t illegal and it isn’t Google’s fault, but"