6 ms·

Author heeyah. Would love your feedback, either here or at @cushychicken on the tweet street.

funnyflamigo 5y ago
Can you elaborate on what you mean by not interrupting the scrape and instead flagging those pages? Let's say you're scraping product info from a large list of products. I'm assuming you mean if it's strange one-off type errors to handle those, and you'd stop altogether if too many fail? Otherwise you'd just be DOS'ing the site.
- cushychicken 5y ago
  Can you elaborate on what you mean by not interrupting the scrape and instead flagging those pages? Sure! I can get a little more concrete about this project more easily than I can comment on your hypothetical about a large list of products, though, so forgive me in advance for pivoting on the scenario here. I'm scraping job pages. Typically, one job posting == one link. I can go through that link for the job posting and extract data from given HTML elements using CSS selectors or XPath statements. However, sometimes the data I'm looking for isn't structured in a way I expect. The major area I notice variations in job ad data is location data. There are a zillion little variations in how you can structure the location of a job ad. City+country, city+state+country, comma separated, space separated, localized states, no states or provinces, all the permutations thereof. I've written the extractor to expect a certain format of location data for a given job site - let's say "<city>, <country>", for example. If the scraper comes across an entry that happens to be "<city>, <state>, <country>", it's generally not smart enough to generalize its transform logic to deal with that. So, to handle it, I mark that particular job page link as needing human review, so it pops up as an ERROR in my logs, and as an entry in the database that has post_status == 5. After that, it gets inserted into the database, but not posted live onto the site. That way, I can go in and manually fix the posting, approve it to go on the site (if it's relevant), and, ideally, tweak the scraper logic so that it handles transforms of that style of data formatting as well as the "<city>, <country>" format I originally expected. Does that make sense? I suspect I'm just writing logic to deal with malformed/irregular entries that humans make into job sites XD
  - marginalia_nu 5y ago
    I've had a lot of success just saving the data into gzipped tarballs, like a few thousand documents per tarball. That way I can replay the data and tweak the algorithms without causing traffic.
    - muxator 5y ago
      First of all, thanks for marginalia.nu. Have you considered stored compressed blobs in a sqlite file? Works fine for me, you can do indexed searches on your "stored" data, and can extract single pages if you want.
      - marginalia_nu 5y ago
        The main reason I'm doing it this way is because I'm saving this stuff to a mechanical drive, and I want consistent write performance and low memory overhead. Since it's essentially just an archive copy, I don't mind if it takes half an hour to chew through looking for some particular set of files. Since this is a format deigned for tape drives, it causes very little random access. It's important that it's relatively consistent to write since my crawler does while it's crawling, and it can reach speeds of 50-100 documents per second, which would be extremely rough on any sort of database based on a single mechanical hard drive. These archives are just an intermediate stage that's used if I need to reconstruct the index to tweak say keyword extraction or something, so random access performance isn't something that is particularly useful.
    - cushychicken 5y ago
      Is that still practical even if you're storing the page text? The reason I don't do that is because I have a few functions that analyze the job descriptions for relevance, but don't store the post text. I mostly did that to save space - I'm just aggregating links to relevant roles, not hosting job posts. I figured saving ~1000 job descriptions would take up a needlessly large chunk of space, but truth be told I never did the math to check. Edit: I understand scrapy does something similar to what you're describing; have considered using that as my scraper frontend but haven't gotten around to doing the work for it yet.
gringo_flamingo 5y ago
I can truly relate to this article, especially where you mentioned trying to extract only the specific contents of elements that you need; without bloating your software. To me, that seemed intuitive with the minimal experience I have in web scraping. However, I ended up fighting the frameworks. Me being stubborn, I did not try your approach and kept trying to be a perfectionist about it LMAO. Thank you for this read, glad I am not the only one who has been through this. Haha...
- cushychicken 5y ago
  Yeah it's an easy thing to get into a perfectionist streak over. Thinking about separation of concerns helped me a lot in getting over the hump of perfectionism. Once I realized I was trying to make my software do too much, it was easier to see how it would be much less work to write as two separate programs bundled together. (Talking specifically about the extract/transform stages here.) Upon reflection, this project has been just as much self-study of good software engineering practices as it has been learning how to scrape job sites XD
  - gringo_flamingo 5y ago
    Thank you for your reply, and your feedback, man! I will be sure to take this knowledge with me on my next web scraping journey! I appreciate your time XD
cogburnd02 5y ago
There's a project called woob (https://woob.tech/ https://woob.tech/) that implements python scripts that are a little bit like scrapers but only 'scrape' on demand from requests from console & desktop programs. How much of this article do you think would apply to something like that? e.g. something like 'wait a second (or even two!) between successive hits' might not be necessary (one could perhaps shorten it to 1/4 second) if one is only doing a few requests followed by long periods of no requests.
- cushychicken 5y ago
  Interesting question. My first instinct is to say that woob seems closer in use case to a browser than a scraper, as it seems largely geared towards making rich websites more easily accessible. (If I'm reading the page right, anyway.) A scraper is basically just hitting a web page over and over again, as fast as you can manage. The trick, IMO, is to be closer to browser level loading on a server than scraper. Make sense?

Can you elaborate on what you mean by not interrupting the scrape and instead flagging those pages? Let's say you're scraping product info from a large list of products. I'm assuming you mean if it's strange one-off type errors to handle those, and you'd stop altogether if too many fail? Otherwise you'd just be DOS'ing the site.

cushychicken 5y ago
Can you elaborate on what you mean by not interrupting the scrape and instead flagging those pages? Sure! I can get a little more concrete about this project more easily than I can comment on your hypothetical about a large list of products, though, so forgive me in advance for pivoting on the scenario here. I'm scraping job pages. Typically, one job posting == one link. I can go through that link for the job posting and extract data from given HTML elements using CSS selectors or XPath statements. However, sometimes the data I'm looking for isn't structured in a way I expect. The major area I notice variations in job ad data is location data. There are a zillion little variations in how you can structure the location of a job ad. City+country, city+state+country, comma separated, space separated, localized states, no states or provinces, all the permutations thereof. I've written the extractor to expect a certain format of location data for a given job site - let's say "<city>, <country>", for example. If the scraper comes across an entry that happens to be "<city>, <state>, <country>", it's generally not smart enough to generalize its transform logic to deal with that. So, to handle it, I mark that particular job page link as needing human review, so it pops up as an ERROR in my logs, and as an entry in the database that has post_status == 5. After that, it gets inserted into the database, but not posted live onto the site. That way, I can go in and manually fix the posting, approve it to go on the site (if it's relevant), and, ideally, tweak the scraper logic so that it handles transforms of that style of data formatting as well as the "<city>, <country>" format I originally expected. Does that make sense? I suspect I'm just writing logic to deal with malformed/irregular entries that humans make into job sites XD
- marginalia_nu 5y ago
  I've had a lot of success just saving the data into gzipped tarballs, like a few thousand documents per tarball. That way I can replay the data and tweak the algorithms without causing traffic.
  - muxator 5y ago
    First of all, thanks for marginalia.nu. Have you considered stored compressed blobs in a sqlite file? Works fine for me, you can do indexed searches on your "stored" data, and can extract single pages if you want.
    - marginalia_nu 5y ago
      The main reason I'm doing it this way is because I'm saving this stuff to a mechanical drive, and I want consistent write performance and low memory overhead. Since it's essentially just an archive copy, I don't mind if it takes half an hour to chew through looking for some particular set of files. Since this is a format deigned for tape drives, it causes very little random access. It's important that it's relatively consistent to write since my crawler does while it's crawling, and it can reach speeds of 50-100 documents per second, which would be extremely rough on any sort of database based on a single mechanical hard drive. These archives are just an intermediate stage that's used if I need to reconstruct the index to tweak say keyword extraction or something, so random access performance isn't something that is particularly useful.
  - cushychicken 5y ago
    Is that still practical even if you're storing the page text? The reason I don't do that is because I have a few functions that analyze the job descriptions for relevance, but don't store the post text. I mostly did that to save space - I'm just aggregating links to relevant roles, not hosting job posts. I figured saving ~1000 job descriptions would take up a needlessly large chunk of space, but truth be told I never did the math to check. Edit: I understand scrapy does something similar to what you're describing; have considered using that as my scraper frontend but haven't gotten around to doing the work for it yet.

gringo_flamingo 5y ago

I can truly relate to this article, especially where you mentioned trying to extract only the specific contents of elements that you need; without bloating your software. To me, that seemed intuitive with the minimal experience I have in web scraping. However, I ended up fighting the frameworks. Me being stubborn, I did not try your approach and kept trying to be a perfectionist about it LMAO. Thank you for this read, glad I am not the only one who has been through this. Haha...

cushychicken 5y ago
Yeah it's an easy thing to get into a perfectionist streak over. Thinking about separation of concerns helped me a lot in getting over the hump of perfectionism. Once I realized I was trying to make my software do too much, it was easier to see how it would be much less work to write as two separate programs bundled together. (Talking specifically about the extract/transform stages here.) Upon reflection, this project has been just as much self-study of good software engineering practices as it has been learning how to scrape job sites XD
- gringo_flamingo 5y ago
  Thank you for your reply, and your feedback, man! I will be sure to take this knowledge with me on my next web scraping journey! I appreciate your time XD

cogburnd02 5y ago

There's a project called woob (https://woob.tech/ https://woob.tech/) that implements python scripts that are a little bit like scrapers but only 'scrape' on demand from requests from console & desktop programs. How much of this article do you think would apply to something like that? e.g. something like 'wait a second (or even two!) between successive hits' might not be necessary (one could perhaps shorten it to 1/4 second) if one is only doing a few requests followed by long periods of no requests.

cushychicken 5y ago
Interesting question. My first instinct is to say that woob seems closer in use case to a browser than a scraper, as it seems largely geared towards making rich websites more easily accessible. (If I'm reading the page right, anyway.) A scraper is basically just hitting a web page over and over again, as fast as you can manage. The trick, IMO, is to be closer to browser level loading on a server than scraper. Make sense?