3 ms·
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey tha
by arjie 1mo ago
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
- xena 1mo agoOne of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
- arjie 1mo agoThat makes sense, Xe, but you don't enable it at: - https://techaro.lol/robots.txt https://techaro.lol/robots.txt - https://anubis.techaro.lol/robots.txt https://anubis.techaro.lol/robots.txt and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
- deleted 1mo ago[deleted]
- mitxela 1mo agoIn this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
- arjie 1mo agoThat is true, but if you have Anubis you presumably do intend to `Disallow` all bots.
- 9cb14c1ec0 1mo agoNot necessarily. You might want to rate limit bots instead of disallowing them.
- pinocchioruns 1mo ago[flagged]