4 ms·
How often does it crawl? Just this week I tried to look up a reference on a page and found it dead, probably for quite some time (I forget the page I was on).
by eridius 14y ago
How often does it crawl? Just this week I tried to look up a reference on a page and found it dead, probably for quite some time (I forget the page I was on).
- Wingman4l7 14y agoWell it turns out I may have been mistaken; the bot that I thought did that is defunct, and I cannot easily find another that does that. After thinking about it, I can see some reasons why you might not want a bot automatically marking links as dead. There are these, which I think are a bit more manual in nature: http://en.wikipedia.org/wiki/Wikipedia:Link_rot#Bots http://en.wikipedia.org/wiki/Wikipedia:Link_rot#Bots
- emiliobumachar 14y agoPlease share those reasons. The only one I can think of is false positives of temporarily down servers, which could be mitigated by timed retries.
- Wingman4l7 14y agoThat is the main reason. Another would be some sort of "Content not here anymore" message that doesn't conform to HTTP status code standards, giving a false negative.