4 ms·
robots.txt was originally created to include such bots. That they think they don't need to respect it goes against the original intent. Eg: https://www.robotst
by Arnavion 4y ago
robots.txt was originally created to include such bots. That they think they don't need to respect it goes against the original intent.
Eg: https://www.robotstxt.org/faq/kinds.html https://www.robotstxt.org/faq/kinds.html >"What's New" monitoring
- counttheforks 4y agoI want my software to obey me, not someone else. If the software is discovering resources on its own, then obeying robots.txt is fair. But if the software is polling a resource I explicitly told it to, I would not expect it to make additional requests to fetch unrelated files such as a robots.txt
- chillfox 4y agoI can almost see both sides here... But ultimately when you are using someone else's resources, then not respecting their wishes (within reason) just makes you an asshole.