4 ms·
Do you have an example? A crawler ignoring robots.txt sounds like a bug, and if you have details I can file a bug with the appropriate team.
by cbr 11y ago
Do you have an example? A crawler ignoring robots.txt sounds like a bug, and if you have details I can file a bug with the appropriate team.
- zippy786 11y agoFeedFetcher User Agent