3 ms·
Their robots.txt is password protected now. (?) I'm not sure what a crawler is supposed to conclude in that case. I don't see a lot of cpedia content on Googl
by neilk 17y ago
Their robots.txt is password protected now. (?)
I'm not sure what a crawler is supposed to conclude in that case.
I don't see a lot of cpedia content on Google yet. The only links I can see are clearly due to the threads here and on sites like Reddit.
- gojomo 17y agoLast I checked, Google's crawler treated 4xx errors on a /robots.txt (which includes unauthorized/not-allowed) the same as 404-not-found. (In most cases, it's a configuration error by the site owner, and for those sites where it's intentional, the operator is savvy enough to block crawler access to other URLs by password or user-agent if desired.)
- blasdel 17y agoNow it's this: User-agent: * Disallow: /imgsrv Disallow: /search Disallow: /pref Disallow: /suggest
- chbarts 17y ago> Their robots.txt is password protected now. (?) It lets you in regardless of the username/password combination. I just logged in with this: username: you morons password: this defeats the whole damn purpose I wonder if any of this gets logged at their end.