5 ms·
Web-scrapers in general are prone to breaking because the HTML design can change. A div could become a span class='blocky'. Now your scrape doesn't work anymore
by chime 16y ago
Web-scrapers in general are prone to breaking because the HTML design can change. A div could become a span class='blocky'. Now your scrape doesn't work anymore.
- weego 16y agoAlso you get a poor user experience when the service fails because their scraper domain gets explicitly blocked. Scraping in general is not system you can base any service anyone might ever want to "rely" on.
- xtacy 16y agoOne way to handle such breakage would be to define an extraction template on the fly, based on some content that doesn't change on the page. If, for e.g., you know that the stock quote for Google has GOOG and some number somewhere inside the body, you could search for that pattern and save the XPath or something and reuse it on the other scraping. scRUBYt (http://scrubyt.org/ http://scrubyt.org/) has a nice way of specifying such templates.