3 ms·

Seems like it would be more robust to just persist the state of your scrape so you can resume it. Say more about this. I'm not a software engineer by training,

Seems like it would be more robust to just persist the state of your scrape so you can resume it. Say more about this. I'm not a software engineer by training, so I don't really know what "persist your state" would look like in this case.

derac 5y ago
Write the pages you are scraping to a cache. The simplest way would be to write them each to folder. Check if the page you are going to scrape is cached before trying to request it from the server.
throwaway894345 5y ago
It just means to write to disk where you left off, like saving your progress in a video game.
MaxDPS 5y ago
If you are scraping 1000 pages and your program crashes after the 800th page, you are going to be much happier if you have the data for the 800 pages saved somewhere vs having to start all over.
- cushychicken 5y ago
  Got it. Yeah, I can give this approach a shot. One of the things I've been doing with this scraping project is "health checking" job posting links. There's nothing more annoying than clicking an interesting looking link on a job site, only to find it's been filled. (This is one of the lousiest parts of Indeed and similar, IMO.) I wrote some pretty simple routines Caching solves the problem of potentially missing data while the scraper is running, but it doesn't really alleviate the network strain of requesting pages to see that they are actually still posted, valid job links.
franga2000 5y ago
Generally this means that you should be writing all the important information about the program to disk so in the event of a crash, you can read it back and continue where you left off. Alternatively, your state could BE on disk (or whatever persistent store, like a DB) and your program should process it in atomic chunks with zero difference between a "from scratch" and "resumed" run. I work on web scrapers at [day job] and can say the latter approach is far better, but the former is far more common. An implementation of the former could be as simple as "dump the current URL queue to a CSV file every minute". As for doing this "properly" in a way that works at scale, my preferred way is with a producer-consumer pipeline and out-of-process shared queues in between each pair. So, for example, you have 4 queues: URL queue, verify queue, reponse queue, item queue and 5 stages: fetch (reading from URLQ and writing to verifyQ), response check (reading from verifyQ, writing good responses to reponseQ and bad response URLs back to fetchQ), parse (reading from responseQ and writing to itemQ) and publish (reading from itemQ and writing to database or whatever). This is is be both horrendously overcomplicated and beautifully simple. I've implemented this both with straight up bash scripts and with dedicated queues and auto-scaling armies of docker containers.

derac 5y ago

Write the pages you are scraping to a cache. The simplest way would be to write them each to folder. Check if the page you are going to scrape is cached before trying to request it from the server.

throwaway894345 5y ago

It just means to write to disk where you left off, like saving your progress in a video game.

MaxDPS 5y ago

If you are scraping 1000 pages and your program crashes after the 800th page, you are going to be much happier if you have the data for the 800 pages saved somewhere vs having to start all over.

cushychicken 5y ago
Got it. Yeah, I can give this approach a shot. One of the things I've been doing with this scraping project is "health checking" job posting links. There's nothing more annoying than clicking an interesting looking link on a job site, only to find it's been filled. (This is one of the lousiest parts of Indeed and similar, IMO.) I wrote some pretty simple routines Caching solves the problem of potentially missing data while the scraper is running, but it doesn't really alleviate the network strain of requesting pages to see that they are actually still posted, valid job links.

franga2000 5y ago

Generally this means that you should be writing all the important information about the program to disk so in the event of a crash, you can read it back and continue where you left off. Alternatively, your state could BE on disk (or whatever persistent store, like a DB) and your program should process it in atomic chunks with zero difference between a "from scratch" and "resumed" run. I work on web scrapers at [day job] and can say the latter approach is far better, but the former is far more common. An implementation of the former could be as simple as "dump the current URL queue to a CSV file every minute". As for doing this "properly" in a way that works at scale, my preferred way is with a producer-consumer pipeline and out-of-process shared queues in between each pair. So, for example, you have 4 queues: URL queue, verify queue, reponse queue, item queue and 5 stages: fetch (reading from URLQ and writing to verifyQ), response check (reading from verifyQ, writing good responses to reponseQ and bad response URLs back to fetchQ), parse (reading from responseQ and writing to itemQ) and publish (reading from itemQ and writing to database or whatever). This is is be both horrendously overcomplicated and beautifully simple. I've implemented this both with straight up bash scripts and with dedicated queues and auto-scaling armies of docker containers.