3 ms·
Why? Isn’t it better to use a vps or two (or even just a vpn), and strong rate limiting? I.e. try not to be noticed regardless of the robots.txt.
by eccfcco15 9y ago
Why? Isn’t it better to use a vps or two (or even just a vpn), and strong rate limiting? I.e. try not to be noticed regardless of the robots.txt.
- JosephRedfern 9y agoBecause it's considered polite.
- slaymaker1907 9y agoA lot of robots.txt exist to help crawlers automatically know which URLs are safe and which ones are mutable (i.e. the sign up link). You should only ignore the robots.txt if you are very careful about what you are doing and have a very good reason for doing so.