5 ms·
robots.txt is not a law. It's what google pushed. No one has to obey robots.txt and will often use it to find things you forbid indexing
by ipaddr 4y ago
robots.txt is not a law. It's what google pushed. No one has to obey robots.txt and will often use it to find things you forbid indexing
- nl 4y agorobots.txt was around before Google