5 ms·
Makes sense until you realize you'd have to prohibit everyone else's own crawling as well (can't single out and put Google at a disadvantage) and then you'd jus
by lazyjones 6y ago
Makes sense until you realize you'd have to prohibit everyone else's own crawling as well (can't single out and put Google at a disadvantage) and then you'd just have turned Google's de facto-monopoly into an actual, government-mandated crawling monopoly.
I'd rather make http://commoncrawl.org http://commoncrawl.org more current, accessible and more commonly used so website publishers can see a benefit in actively supporting it to lighten the load on their servers.
- octoberfranklin 6y agoIt's a natural monopoly. It has already de facto prohibited everyone else's own crawling, as the linked article demonstrates. When it had a monopoly, AT&T was forbidden from selling software.
- lazyjones 6y agoI disagree on the "natural" part. Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years, some websites supporting only Netscape and MSIE, a direct consequence of Google's current market share and might change once there is a good reason (like DDG growing into a significant player). If a collection like commoncrawl with bulk downloads was more useful and thus used more often, even Google would have a good reason to use it.
- octoberfranklin 6y ago> Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years It's not just robots.txt, it's also cloudflare and IP-based throttling. And it is very, very commonplace: http://gigablast.com/blog.html http://gigablast.com/blog.html
- ColinHayhurst 6y agoNot commonplace https://news.ycombinator.com/item?id=25373909 https://news.ycombinator.com/item?id=25373909
- eznzt 6y agooof https://commoncrawl.org/wp-content/uploads/2016/03/box-5.png https://commoncrawl.org/wp-content/uploads/2016/03/box-5.png