5 ms·
FWIW, there is at least one bot on Wikipedia that crawls for dead reference links and marks them as such.
by Wingman4l7 14y ago
FWIW, there is at least one bot on Wikipedia that crawls for dead reference links and marks them as such.
- eridius 14y agoHow often does it crawl? Just this week I tried to look up a reference on a page and found it dead, probably for quite some time (I forget the page I was on).
- Wingman4l7 14y agoWell it turns out I may have been mistaken; the bot that I thought did that is defunct, and I cannot easily find another that does that. After thinking about it, I can see some reasons why you might not want a bot automatically marking links as dead. There are these, which I think are a bit more manual in nature: http://en.wikipedia.org/wiki/Wikipedia:Link_rot#Bots http://en.wikipedia.org/wiki/Wikipedia:Link_rot#Bots
- emiliobumachar 14y agoPlease share those reasons. The only one I can think of is false positives of temporarily down servers, which could be mitigated by timed retries.
- Wingman4l7 14y agoThat is the main reason. Another would be some sort of "Content not here anymore" message that doesn't conform to HTTP status code standards, giving a false negative.