3 ms·

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. Wh

by apeace 8y ago

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wanted to try the following solution to it but never had the chance. Assume you have a database of URLs and the fields you've scraped from them in the past (title, author, date, etc). If you ever fail to scrape one of those values from a new URL, here's what you do: - Go back to one of the old URLs where you already have the correct value (let's say it's the title). - Walk through the whole DOM until you find that known title. At each node you will have to remove child nodes except for text, to deal with titles like "Foo <span>Bar</span>" which you want to match against "Foo Bar". So this is going to be an expensive search. - Generate several possible selectors which match the node you walked to (maybe you have ".title", ".title h2", ".content .top h2", etc). - Test each new selector on several other already-crawled pages. If any of the selectors work 100% of the time, there's your new selector. Any thoughts on whether something like this would work?

moepstar 8y ago
I've had the same idea, it kind of sounds like the obvious thing to do... That said, i haven't found a need to do that yet to verify the idea itself. Maybe that is similar to what they do using their "ML approach" mentioned?
- RossM 8y ago
  Their ML approach is most likely based on their https://github.com/scrapy/scrapely https://github.com/scrapy/scrapely project, which uses instance-based learning (as I understand it, lightweight ML) to scrape other pages from a few examples.
AznHisoka 8y ago
That sounds like a very creative idea. What I do is call a regression test every x minutes. If it fails, set a flag to save/store the html everytime we crawl pages. Now we can go back and process these saved pages when we fix our crawler
- w0rd-driven 8y ago
  I crawl a specific site somewhere up to 50 unique URLs a day. I store both the unparsed full html as a file and the json I'm looking for as another separate file. The idea is if something breaks instead of taking a hit to make the call again, I have the data and I should just process that. It's come in extremely handy when a site redesign changed the DOM and broke the parser. I do the same at $dayJob where I'm parsing results of an internal API. Instead of making a call later that may not have the same data, I store the json and just process that. I feel like treating network requests as an expensive operation, even though they're not really, helped me come up with some clever ideas I've never had before. It's a premature optimization considering I've had like 0.000001% of failure but being able to replay that one breakage made debugging an esoteric problem waaaaaay simpler than it would've been otherwise.
  - pdimitar 8y ago
    Off-topic: I so wish I worked for a company where my work involves scraping and storing and analyzing data. :(
    - hoju 8y ago
      Now is a good time to work in this field since data science is hot and companies need web scrapers to provide the data for these models. Atleast that has been my experience in finance. Try applying!
      - pdimitar 8y ago
        I have zero experience in data science though. I am a pretty solid and experienced programmer and can learn it all but... don't know. Maybe I should just try indeed. Do you have any recommendations for places and/or interview practices?
nkozyra 8y ago
Without going too detailed into this ... Creating a domain-specific model can be done a million ways, obviously, but the nice thing about HTML is the markup gives hints all along the way. The tree, the class names, the tags, etc. Coupled with the content of each tag, it's absolutely possible to determine collections of items with metadata using ML. But like most ML problems, the underlying data that feeds the model is the time-consuming part. If you are working in a single or few domains, I 100% recommend this approach. If you're scraping something far more generalized, first you need to have models that you care about and then you need to create models to determine the content type of what you're scraper is looking at. 1) What kind of content do I have? 2) Does this match a known domain with a model? 3) Apply appropriate model to domain, hopefully extract correct data Another huge issue is, of course, validation, because you're going to be dealing with an inordinate amount of unknown and unpredictable data depending on what you're looking at.
Alex3917 8y ago
> Any thoughts on whether something like this would work? If it really works 100% of the time then probably. A lot sites though use multiple markup styles for seemingly no reason though. E.g. if you created an account before a certain date then your profile keeps the old HTML, even though the old pages look identical to the new pages.
- jd20 8y ago
  I encountered this on one website, that recently rewrote their website to be mobile-responsive. Depending on the width of the browser window, the right selectors would be totally different, talk about confusing!
tomjen3 8y ago
Lets say you are scraping for "Lawn Chair" and I change it to specify "A chair for your lawn" in an attempt to improve SEO. How would your system cope with that? Suppose alternatively that I have previously shown the status as "available" or "out of stock" and now I change it for some products to "no longer available". Can your system handle those edge cases?
- blacksmith_tb 8y ago
  Some of that will depend on how much you structure the page for bots (not just scrapers) - if the availability info is exposed via json-ld, I would expect everyone to consume changes just fine. Note that changing from "lawn chair" (phrase humans actually use and search for) to "chair for your lawn" (phrase which is unlikely to ever be used by anyone) is not an SEO boost...
arciini 8y ago
At https://WrapAPI.com https://WrapAPI.com, we have a tool that lets you get a selector by clicking on a specific element and an interface that lets you test that selector on different sample pages. We'll make a note of this! It does seem like a cool idea to put these 2 features together and automate the updating of API endpoints
hoju 8y ago
This approach is known as "wrapper induction"

by apeace 8y ago

moepstar 8y ago

I've had the same idea, it kind of sounds like the obvious thing to do... That said, i haven't found a need to do that yet to verify the idea itself. Maybe that is similar to what they do using their "ML approach" mentioned?

RossM 8y ago
Their ML approach is most likely based on their https://github.com/scrapy/scrapely https://github.com/scrapy/scrapely project, which uses instance-based learning (as I understand it, lightweight ML) to scrape other pages from a few examples.

AznHisoka 8y ago

That sounds like a very creative idea. What I do is call a regression test every x minutes. If it fails, set a flag to save/store the html everytime we crawl pages. Now we can go back and process these saved pages when we fix our crawler

w0rd-driven 8y ago
I crawl a specific site somewhere up to 50 unique URLs a day. I store both the unparsed full html as a file and the json I'm looking for as another separate file. The idea is if something breaks instead of taking a hit to make the call again, I have the data and I should just process that. It's come in extremely handy when a site redesign changed the DOM and broke the parser. I do the same at $dayJob where I'm parsing results of an internal API. Instead of making a call later that may not have the same data, I store the json and just process that. I feel like treating network requests as an expensive operation, even though they're not really, helped me come up with some clever ideas I've never had before. It's a premature optimization considering I've had like 0.000001% of failure but being able to replay that one breakage made debugging an esoteric problem waaaaaay simpler than it would've been otherwise.
- pdimitar 8y ago
  Off-topic: I so wish I worked for a company where my work involves scraping and storing and analyzing data. :(
  - hoju 8y ago
    Now is a good time to work in this field since data science is hot and companies need web scrapers to provide the data for these models. Atleast that has been my experience in finance. Try applying!
    - pdimitar 8y ago
      I have zero experience in data science though. I am a pretty solid and experienced programmer and can learn it all but... don't know. Maybe I should just try indeed. Do you have any recommendations for places and/or interview practices?

nkozyra 8y ago

Without going too detailed into this ... Creating a domain-specific model can be done a million ways, obviously, but the nice thing about HTML is the markup gives hints all along the way. The tree, the class names, the tags, etc. Coupled with the content of each tag, it's absolutely possible to determine collections of items with metadata using ML. But like most ML problems, the underlying data that feeds the model is the time-consuming part. If you are working in a single or few domains, I 100% recommend this approach. If you're scraping something far more generalized, first you need to have models that you care about and then you need to create models to determine the content type of what you're scraper is looking at. 1) What kind of content do I have? 2) Does this match a known domain with a model? 3) Apply appropriate model to domain, hopefully extract correct data Another huge issue is, of course, validation, because you're going to be dealing with an inordinate amount of unknown and unpredictable data depending on what you're looking at.

Alex3917 8y ago

> Any thoughts on whether something like this would work? If it really works 100% of the time then probably. A lot sites though use multiple markup styles for seemingly no reason though. E.g. if you created an account before a certain date then your profile keeps the old HTML, even though the old pages look identical to the new pages.

jd20 8y ago
I encountered this on one website, that recently rewrote their website to be mobile-responsive. Depending on the width of the browser window, the right selectors would be totally different, talk about confusing!

tomjen3 8y ago

Lets say you are scraping for "Lawn Chair" and I change it to specify "A chair for your lawn" in an attempt to improve SEO. How would your system cope with that? Suppose alternatively that I have previously shown the status as "available" or "out of stock" and now I change it for some products to "no longer available". Can your system handle those edge cases?

blacksmith_tb 8y ago
Some of that will depend on how much you structure the page for bots (not just scrapers) - if the availability info is exposed via json-ld, I would expect everyone to consume changes just fine. Note that changing from "lawn chair" (phrase humans actually use and search for) to "chair for your lawn" (phrase which is unlikely to ever be used by anyone) is not an SEO boost...

arciini 8y ago

At https://WrapAPI.com https://WrapAPI.com, we have a tool that lets you get a selector by clicking on a specific element and an interface that lets you test that selector on different sample pages. We'll make a note of this! It does seem like a cool idea to put these 2 features together and automate the updating of API endpoints

hoju 8y ago

This approach is known as "wrapper induction"