3 ms·
This is one of the best arguments for scraping data, if you can manage it. Generally businesses invest much more time and resources into maintaining an up-to-da
by hartleybrody 15y ago
This is one of the best arguments for scraping data, if you can manage it. Generally businesses invest much more time and resources into maintaining an up-to-date, usable website than they would on an API, especially if they don't use a service-oriented architecture internally (ie, consume their own API)
- rollypolly 15y agoWhat type of business can afford being down every time the site they're scrapping is tweaked?
- tantalor 15y ago1. Any business that scrapes many sites. 2. Who says it goes down? Might just have stale or incorrect data before the tweak is patched.
- hartleybrody 15y agoHow often do websites really change the layout of their DOM? Sure, some sites that are actively developed (ie social networks) but for a lot of business websites where information doesn't change that often, like restaurants, reviews or clothing, websites are pretty consistent. Plus, building a parser isn't always that much more work than trying to build a client for a convoluted API. It's obviously case-by-case, but I think a lot of people dismiss scraping outright when there are a lot of useful applications.
- sad_panda 15y agoOMG, what? Scraping data for offline processing, but certainly should not be for user-facing traffic. Cosmetic tweaks can blow up the scraper's parser, UI technologies are built on a taller stack and are prone to higher latencies and points of failure, yada yada.
- puxedo 15y agoIndeed. Very risky. Scraping is not really a sustainable model when you need predictability. Too many variables.
- tantalor 15y agoAn example comes to mind: The Yelp API doesn't expose a given user's public bookmarks, reviews, etc., but they can be easily scraped from the user-facing website. http://www.yelp.com/developers/documentation/v2/overview http://www.yelp.com/developers/documentation/v2/overview