5 ms·

Why are they prone to breaking?

• d2viant 16y ago
  Becuase they'll break whenever the source tweaks their formatting in the slightest.
  • chuhnk 16y ago
    Yea see thats a pain but the concept is great. To turn everything into a json feed. There has to be a better way of doing though that js scrapers.
    • mistermann 16y ago
      I think any other solution (for example, auto reconfiguring smart scripts based on html changes), would quickly become so complex that it would become far more time consuming to maintain than simply going and tweaking the individual javascript when an html change actually does occur.
• chime 16y ago
  Web-scrapers in general are prone to breaking because the HTML design can change. A div could become a span class='blocky'. Now your scrape doesn't work anymore.
  • weego 16y ago
    Also you get a poor user experience when the service fails because their scraper domain gets explicitly blocked. Scraping in general is not system you can base any service anyone might ever want to "rely" on.
  • xtacy 16y ago
    One way to handle such breakage would be to define an extraction template on the fly, based on some content that doesn't change on the page. If, for e.g., you know that the stock quote for Google has GOOG and some number somewhere inside the body, you could search for that pattern and save the XPath or something and reuse it on the other scraping. scRUBYt (http://scrubyt.org/ http://scrubyt.org/) has a nice way of specifying such templates.