6 ms·
I have run a news website since 2019. Every hour, I have a crawler look for dead links. I replace about one link a day with a link to archive.org. The funniest
by earthboundkid 2y ago
I have run a news website since 2019. Every hour, I have a crawler look for dead links. I replace about one link a day with a link to archive.org. The funniest ones are the day after an election when all the candidate websites go blank. The saddest are the government websites that go offline from 3am to 5am every week.
- notRobot 2y agoInteresting, does your crawler check every link every hour, or does it go through them a batch at a time?
- earthboundkid 2y agoEvery link every hour. https://github.com/spotlightpa/linkrot https://github.com/spotlightpa/linkrot