4 ms·
Note that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard. You can block crawlers if you can identify them, but reliably identif
by Xylakant 6y ago
Note that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard.
You can block crawlers if you can identify them, but reliably identifying them is hard.
- tleb_ 6y agoWe should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.
- ddorian43 6y agohttps://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
- Xylakant 6y agoThis helps to verify that a bot that announces itself as google bot is indeed a google bot. It doesn’t help identify a bot that pretends to be a user/browser.