4 ms·

From a linked blog post of the author's: > Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather

From a linked blog post of the author's: > Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgentMaybeTooClever https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...) Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.

shagie 4y ago
The sequence likely went "site is not crawalble by anything" (because there were so many badly behaved crawlers back in the day). Later the webmaster wanted it to show up in searches and so specifically opened up the robots.txt from "no body no way at all ever" to "ok, google can search these pages because they're not being evil" (this was back in the day). It wasn't so much a "ok, I want only google to search this and not anyone else", but rather at the time, there wasn't anyone else (that behaved). So now, you've got a website that is only searchable by google, and the webmaster retired and you've got this robots.txt that says you can't do it - and even some active defenses so that when non-google hits the page, they get tarpited. What are competing (reasonably well behaved) web spiders to do? How does Bing get in there to be able to compete now? Its not that Bing should or shouldn't be doing this - but there's a fairly reasonably way that we got to the point where are now without any nefarious behavior from the various participants.
- akerl_ 4y ago
  If I'm worried about badly behaved crawlers... what's robots.txt doing for me?
  - jakopo87 4y ago
    Years ago i added a url to not be crawled nor indexed to a web app I worked on to block bad crawlers, so I disallowed crawling through robots.txt and added a hidden link in the footer with the "noindex" attribute. Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page.
  - rezonant 4y ago
    (On that note) Fun fact, there are a lot of sites out there that have User-Agent: * Disallow / User-Agent: Googlebot Allow / -- and yet those sites are in the Bing index. Do I blame Bing for that? Absolutely not. Having a silly robots.txt is one thing- going to the lengths of blocking all bots that aren't in Google's IP space is what I'm worried about. If the whole world did that, nothing but Google could survive.
    - shagie 4y ago
      Those are absolutely problems. I suspect they're artifacts in configurations from when it was believed that Google did no evil and the next biggest company trying to get in on the web search was Microsoft and leaving them in wasn't a great loss. Now roles are reversed (or at least, equal - though if DDG consuming bing data is considered to be on the side of good, an argument could be made for reversed) and yet those configurations persist because they were forgotten about. My post above is a just so story ( https://en.wikipedia.org/wiki/Just-so_story https://en.wikipedia.org/wiki/Just-so_story ) for why they are the way they are now. Old systems that work tend to be forgotten until a security fix needs to go in and Chesterton's fences in config files tend to be left. With todays resources, I would not advocate indiscriminate tcp tarpits for things that might be badly behaved crawlers until they demonstrate that they are indeed badly behaved crawlers or scanners searching for exploits.
      - brazzledazzle 4y ago
        Unless I missed or forgot some major events I don’t think it’s fair to say that Microsoft and Google are on the same level of “evil”. Microsoft willfully broke the law to destroy their competition. On multiple occasions. They had well funded initiatives to destroy Linux and open source in general. They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. Microsoft was (and is) ruthless because Bill Gates was a fucking corporate terminator.
        saagarjha 4y ago
        > They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. …something which Google has never done, I'm sure.

shagie 4y ago

The sequence likely went "site is not crawalble by anything" (because there were so many badly behaved crawlers back in the day). Later the webmaster wanted it to show up in searches and so specifically opened up the robots.txt from "no body no way at all ever" to "ok, google can search these pages because they're not being evil" (this was back in the day). It wasn't so much a "ok, I want only google to search this and not anyone else", but rather at the time, there wasn't anyone else (that behaved). So now, you've got a website that is only searchable by google, and the webmaster retired and you've got this robots.txt that says you can't do it - and even some active defenses so that when non-google hits the page, they get tarpited. What are competing (reasonably well behaved) web spiders to do? How does Bing get in there to be able to compete now? Its not that Bing should or shouldn't be doing this - but there's a fairly reasonably way that we got to the point where are now without any nefarious behavior from the various participants.

akerl_ 4y ago
If I'm worried about badly behaved crawlers... what's robots.txt doing for me?
- jakopo87 4y ago
  Years ago i added a url to not be crawled nor indexed to a web app I worked on to block bad crawlers, so I disallowed crawling through robots.txt and added a hidden link in the footer with the "noindex" attribute. Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page.
- rezonant 4y ago
  (On that note) Fun fact, there are a lot of sites out there that have User-Agent: * Disallow / User-Agent: Googlebot Allow / -- and yet those sites are in the Bing index. Do I blame Bing for that? Absolutely not. Having a silly robots.txt is one thing- going to the lengths of blocking all bots that aren't in Google's IP space is what I'm worried about. If the whole world did that, nothing but Google could survive.
  - shagie 4y ago
    Those are absolutely problems. I suspect they're artifacts in configurations from when it was believed that Google did no evil and the next biggest company trying to get in on the web search was Microsoft and leaving them in wasn't a great loss. Now roles are reversed (or at least, equal - though if DDG consuming bing data is considered to be on the side of good, an argument could be made for reversed) and yet those configurations persist because they were forgotten about. My post above is a just so story ( https://en.wikipedia.org/wiki/Just-so_story https://en.wikipedia.org/wiki/Just-so_story ) for why they are the way they are now. Old systems that work tend to be forgotten until a security fix needs to go in and Chesterton's fences in config files tend to be left. With todays resources, I would not advocate indiscriminate tcp tarpits for things that might be badly behaved crawlers until they demonstrate that they are indeed badly behaved crawlers or scanners searching for exploits.
    - brazzledazzle 4y ago
      Unless I missed or forgot some major events I don’t think it’s fair to say that Microsoft and Google are on the same level of “evil”. Microsoft willfully broke the law to destroy their competition. On multiple occasions. They had well funded initiatives to destroy Linux and open source in general. They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. Microsoft was (and is) ruthless because Bill Gates was a fucking corporate terminator.
      - saagarjha 4y ago
        > They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. …something which Google has never done, I'm sure.