2 ms·
Wouldn't it be impossible to know if it ignored robots.txt? Just because it crawled it doesn't mean it stored it.
by jaimex2 4y ago
Wouldn't it be impossible to know if it ignored robots.txt?
Just because it crawled it doesn't mean it stored it.
- hrbf 4y agoStorage or not is entirely irrelevant to robots.txt directives. It guides automated access. It must be parsed first and excluded URLs must not be accessed at all.