4 ms·
I wonder why the author doesn't mention robots.txt if you only give "permissions" to be crawled by Google and/or Bing you are part of the problem not leaving sp
by wslh 3y ago
I wonder why the author doesn't mention robots.txt if you only give "permissions" to be crawled by Google and/or Bing you are part of the problem not leaving space for new search innovations.
- _heimdall 3y agoIsn't that just an issue of individual choice though? What's wrong with a site admin deciding how they would like their content to be used?
- wslh 3y agoBecause search engines were built because they can crawl sites and if there is a monopoly, adhering to robots.txt just support them. It is important to note that I am not saying that there should be indiscriminate crawling that breaks your site, just saying that robots.txt is not the solution.
- _heimdall 3y agoWell that I agree with, though mainly because robots.txt is only an ask and can be ignored by scrapers. I do prefer site owners being able to limit how their content can be used though. That's individual choice much like choosing a software license for open source code, and totally unrelated to net neutrality as far as I understand it.
- jupp0r 3y agoAre there legal obligations to adhere to robots.txt?
- zlg_codes 3y agorobots.txt is not an official standard by any governmental standards bodies. This is because HTTP is a global protocol. Crawlers can and do ignore this file and its contents. It's much like a post-it that says 'no peeking in these specific places', giving away the location of sensitive information you don't want crawled. It's a well-intentioned initiative but cannot in any way prevent abuse.