3 ms·
Interesting, does your crawler check every link every hour, or does it go through them a batch at a time?
by notRobot 2y ago
Interesting, does your crawler check every link every hour, or does it go through them a batch at a time?
- earthboundkid 2y agoEvery link every hour. https://github.com/spotlightpa/linkrot https://github.com/spotlightpa/linkrot