2 ms·
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
by mitxela 2mo ago
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
- arjie 2mo agoThat is true, but if you have Anubis you presumably do intend to `Disallow` all bots.
- 9cb14c1ec0 2mo agoNot necessarily. You might want to rate limit bots instead of disallowing them.
- pinocchioruns 2mo ago[flagged]