6 ms·
It's more complicated than that, almost all spidering that I do lately for clients is not about scraping the content in a classical sense, but some form of "spy
by ivanhoe 9y ago
It's more complicated than that, almost all spidering that I do lately for clients is not about scraping the content in a classical sense, but some form of "spying" of one's competition, mainly in e-commerce. Tracking inventories, fine-tuning prices, tracking how they promote certain products online, etc. Not to mention really blackhat stuff that many people do like attacking competition with fraudulent ad clicks and similar. Many times "the content providers" have every right to wish to protect themselves.
- jon-wood 9y agoI'll add my piece of anecdata to this. I've done a few projects that involved scraping over my career. 1. Scraping results from a property listing website, specifically to pull the properties an agent had listed and put them on their website. The agent didn't want to pay the fee for API access (they probably ended up paying my employer more to scrape it, and keep that updated, but hey). 2. Scraping an e-commerce website of a company my employer were working with to keep our product catalogues in sync - the partner had their own platform, but no API for it. 3. Automating requests to a price comparison website to find out what prices competitors are offering for particular types of customer.
- jandrese 9y agoAPI keys can be revoked at any time or the API service can be suddenly terminated. You can't trust them. Scraping is more maintenance, but it is more reliable.
- emodendroket 9y agoBut whoever you're scraping can suddenly break your scraping tool too.
- jandrese 9y agoTemporarily until you work around what they're doing. API key revocation leaves you dead in the water until you rewrite your app to do scraping.
- emodendroket 9y agoYeah, but imagine they totally redesign their Web site. The "workaround" is a complete rewrite.
- jandrese 9y agoUltimately it has to show the same data unless the site is completely discontinuing the service (in which case you are up a creek in either scenario). Even in the face of a total redesign you're usually just adjusting a few signatures and maybe tweaking the traversal tree. You can run into problems when they remove things like chronological listing view in favor of some "algorithm" that you can't control, but even that just adds a little work on the back end to detect duplicates and an asterisk that some of the content will be missed. There can also be an arms race where the site developers will start trying all kinds of crazy techniques to make scraping harder. Ironically this often happens after they revoke API access, meaning users don't have an alternative except to participate in the arms race.
- emodendroket 9y agoI'm not anti-scraping or anything but I'd pretty much never choose it over an API for stability.
- ubernostrum 9y agoHaving once worked with the "machine-readable" formats the realty industry produces, I'd rather scrape.
- RandomInteger4 9y agoIt's probably simple enough to implement both methods if available.
- RhodesianHunter 9y ago> The "workaround" is a complete rewrite. No it's not, it's just swapping out a few selectors or regular expressions...
- RandomInteger4 9y agoReorganizing the scraping queries is a lot easier than redesigning the website's layout.
- metalliqaz 9y agoThen they publish a new site format update with no notice. Doesn't sound very reliable to me.
- elorant 9y agoSure, but how often do websites change their layout? Once every five years at most. And it will take you no more than an hour to rewrite the signature. Ain't that hard. I've been doing that for a living and it's manageable, even if you monitor thousands of sites.
- TeMPOraL 9y agoIndeed. A web site is an API, a crappy one, yes, but also one without unnecessary auth requirements and one they can't remove without removing their business.
- kuschku 9y ago> "spying" of one's competition, mainly in e-commerce. Tracking inventories, fine-tuning prices, tracking how they promote certain products online, etc. That’s not spying, that’s a completely legitimate and normal use case of the web. Everything on the web is public, and the web was also created to allow that.
- klodolph 9y agoThat’s why the word “spying” is in quotes.
- dx034 9y agoThat has been done for decades. Most manufacturers employ scores of people going into shops to check how products are presented. Supermarkets employ people to check traffic and sales at competing stores. Doing this online can be easier (if you know how to do it) but it's hardly new. In the offline world companies also try to protect themselves (you can get banned from stores when they find out that you work for the competition) but it's harder to do.