Write the pages you are scraping to a cache. The simplest way would be to write them each to folder. Check if the page you are going to scrape is cached before

Write the pages you are scraping to a cache. The simplest way would be to write them each to folder. Check if the page you are going to scrape is cached before trying to request it from the server.