3 ms·
One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
by xena 27d ago
One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
- arjie 27d agoThat makes sense, Xe, but you don't enable it at: - https://techaro.lol/robots.txt https://techaro.lol/robots.txt - https://anubis.techaro.lol/robots.txt https://anubis.techaro.lol/robots.txt and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.