Got it. Yeah, I can give this approach a shot. One of the things I've been doing with this scraping project is "health checking" job posting links. There's not

Got it. Yeah, I can give this approach a shot. One of the things I've been doing with this scraping project is "health checking" job posting links. There's nothing more annoying than clicking an interesting looking link on a job site, only to find it's been filled. (This is one of the lousiest parts of Indeed and similar, IMO.) I wrote some pretty simple routines Caching solves the problem of potentially missing data while the scraper is running, but it doesn't really alleviate the network strain of requesting pages to see that they are actually still posted, valid job links.