3 ms·
I disagree on the "natural" part. Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years, some websit
by lazyjones 6y ago
I disagree on the "natural" part. Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years, some websites supporting only Netscape and MSIE, a direct consequence of Google's current market share and might change once there is a good reason (like DDG growing into a significant player).
If a collection like commoncrawl with bulk downloads was more useful and thus used more often, even Google would have a good reason to use it.
- octoberfranklin 6y ago> Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years It's not just robots.txt, it's also cloudflare and IP-based throttling. And it is very, very commonplace: http://gigablast.com/blog.html http://gigablast.com/blog.html
- ColinHayhurst 6y agoNot commonplace https://news.ycombinator.com/item?id=25373909 https://news.ycombinator.com/item?id=25373909