4 ms·

Wrap this up in Adsense and you get Mahalo 2.0

• slig 17y ago
  You joke, but it's not that far: http://cpedia.com/robots.txt http://cpedia.com/robots.txt — No disallow on /wiki Except, maybe, that mahalo's "content" is far better scraped.
  • deleted 17y ago
    [deleted]
  • neilk 17y ago
    Their robots.txt is password protected now. (?) I'm not sure what a crawler is supposed to conclude in that case. I don't see a lot of cpedia content on Google yet. The only links I can see are clearly due to the threads here and on sites like Reddit.
    • gojomo 17y ago
      Last I checked, Google's crawler treated 4xx errors on a /robots.txt (which includes unauthorized/not-allowed) the same as 404-not-found. (In most cases, it's a configuration error by the site owner, and for those sites where it's intentional, the operator is savvy enough to block crawler access to other URLs by password or user-agent if desired.)
    • blasdel 17y ago
      Now it's this: User-agent: * Disallow: /imgsrv Disallow: /search Disallow: /pref Disallow: /suggest
    • chbarts 17y ago
      > Their robots.txt is password protected now. (?) It lets you in regardless of the username/password combination. I just logged in with this: username: you morons password: this defeats the whole damn purpose I wonder if any of this gets logged at their end.