4 ms·

I wonder if part of the reason is that the Arc web server doesn't send HTTP status codes properly. Every URL path, valid or not, returns a 200, even if the cont

I wonder if part of the reason is that the Arc web server doesn't send HTTP status codes properly. Every URL path, valid or not, returns a 200, even if the content is only "Unknown." For example, check the headers sent back for: http://news.ycombinator.com/no/way/this/is/possibly/valid http://news.ycombinator.com/no/way/this/is/possibly/valid HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Connection: close If a search engine sees that, it will periodically revisit it forever.

pierrefar 17y ago
The correct solution is to fix the Arc server to return correct HTTP headers, not block everything using robots.txt.
- Qz 17y ago
  Maybe they're blocking everything while they're in the process of fixing it.
- Xichekolas 17y ago
  That may be the correct solution, but I'm sure the current option was chosen more because it was the quick solution.
sjs 17y ago
Until someone creates an HTML link to such a document, how does this problem manifest itself?
- nkassis 17y ago
  typos?
- zachbeane 17y ago
  The site itself created zillions of them itself with its random, expiring fnids. Although the scheme has changed now, cralwers would still hit those obsolete URLs.