Generally this means that you should be writing all the important information about the program to disk so in the event of a crash, you can read it back and con

Generally this means that you should be writing all the important information about the program to disk so in the event of a crash, you can read it back and continue where you left off. Alternatively, your state could BE on disk (or whatever persistent store, like a DB) and your program should process it in atomic chunks with zero difference between a "from scratch" and "resumed" run. I work on web scrapers at [day job] and can say the latter approach is far better, but the former is far more common. An implementation of the former could be as simple as "dump the current URL queue to a CSV file every minute". As for doing this "properly" in a way that works at scale, my preferred way is with a producer-consumer pipeline and out-of-process shared queues in between each pair. So, for example, you have 4 queues: URL queue, verify queue, reponse queue, item queue and 5 stages: fetch (reading from URLQ and writing to verifyQ), response check (reading from verifyQ, writing good responses to reponseQ and bad response URLs back to fetchQ), parse (reading from responseQ and writing to itemQ) and publish (reading from itemQ and writing to database or whatever). This is is be both horrendously overcomplicated and beautifully simple. I've implemented this both with straight up bash scripts and with dedicated queues and auto-scaling armies of docker containers.