2 ms·
Author here - yes, I know about robots.txt! This is one of those cases where, because I was already looking at data in one place, I implemented the fix I could
by gmemstr 2y ago
Author here - yes, I know about robots.txt! This is one of those cases where, because I was already looking at data in one place, I implemented the fix I could in the same place :p I do plan to add a robots.txt and contribute one upstream as well.
- deleted 2y ago[deleted]
- kazinator 2y agoDon't bother; robots.txt is utterly useless. It is widely ignored by bots. IDEA: What you could do is instantly ban anything with "bot" in its agent string, which accesses your site without having probed robots.txt first. That's how robots.txt could be useful, whether you have one or not. Anything which doesn't even bother fetching it is rude garbage.