3 ms·
Isn't that just an issue of individual choice though? What's wrong with a site admin deciding how they would like their content to be used?
by _heimdall 3y ago
Isn't that just an issue of individual choice though? What's wrong with a site admin deciding how they would like their content to be used?
- wslh 3y agoBecause search engines were built because they can crawl sites and if there is a monopoly, adhering to robots.txt just support them. It is important to note that I am not saying that there should be indiscriminate crawling that breaks your site, just saying that robots.txt is not the solution.
- _heimdall 3y agoWell that I agree with, though mainly because robots.txt is only an ask and can be ignored by scrapers. I do prefer site owners being able to limit how their content can be used though. That's individual choice much like choosing a software license for open source code, and totally unrelated to net neutrality as far as I understand it.