3 ms·
It's incredible that the author of this is not aware that robots.txt is widely ignored. How more useful tool would be something that generates a piece of web s
by kazinator 2y ago
It's incredible that the author of this is not aware that robots.txt is widely ignored.
How more useful tool would be something that generates a piece of web server config to actually block bots.