3 ms·
No `robots.txt` indeed means weapons free, but crawler gleans useful info from it, e.g. which areas of the site are dynamically generated. More likely the site
by rdancer 11y ago
No `robots.txt` indeed means weapons free, but crawler gleans useful info from it, e.g. which areas of the site are dynamically generated.
More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.
- Animats 11y agoThe strict interpretation is that if "robots.txt" returns 403 Forbidden, it's interpreted as "deny all". That's what the Python library does. We list those sites as "Blocked".