3 ms·
I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not do
by minimaxir 11y ago
I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do.
I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.
(For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business.)
- kh_hk 11y agoDisclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it is a public service. On the same line, a private company providing a public service should already do so. See motives [1]. > I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem. Same as CityBikes is doing. If we receive a cease and desist, we remove their service from our API. As for Foursquare, I do not see Foursquare as a public service. Your taxdollars at work, and all that. I tried to keep the article balanced but maybe it wasn't clear. There are many transportation companies willing and happy to be scraped, or looking forward to provide their information for people to reuse [2]. [1]: https://blog.scrapinghub.com/2016/03/30/web-scraping-to-create-open-data/#benefits https://blog.scrapinghub.com/2016/03/30/web-scraping-to-crea... [2]: http://nabsa.net/current-members/ http://nabsa.net/current-members/
- seanp2k2 11y agoWhy does your blog intentionally crash browsers that it thinks are Safari?
- fucking_tragedy 11y agoWhat do you mean by intentionally?
- deleted 11y ago[deleted]
- mryan 11y agoYou appear to have replied to the wrong comment. ScrapingHub is not the site that attempts to crash Safari - that is weboob.
- kh_hk 11y ago> For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business. I do not understand that implication. How is providing bike share information creating a competitor? I can't run a bike sharing service.
- minimaxir 11y agoLess a competitor, more a non-canonical source of information that they cannot manage. If your offshoot were to misrepresent data, for example, then you would become a liability, even if you weren't making money.
- kh_hk 11y agoThat's a fair point. Easily enforceable by a proper license. One example is ETALAB open data license.
- chrisweekly 11y agoRandom tangent: your comment about unmanaged non-canonical misrepresented data reminds me of Zillow, who publish "facts" about real estate transactions, with no mechanism for error correction. Afrw years ago they posted an erroneous sale of my house -- which transaction never took place -- listing a sale price 20% lower than we'd paid for it. It directly harmed me, when we later tried to sell the house, when potential buyers cited zillow's "estimates" which of course were artificially, drastically lower because of the phantom transaction. There was no avenue for recourse; angry tweets got a half-baked response from a junior social media person, but it was never resolved. I wonder how many others zillow must have messed up.
- unsettledtck 11y agoOut of curiosity, where are the boundaries of your gray area when it comes to scraping?
- minimaxir 11y agoIf the service has an API with a fair rate limit (Foursquare does at 5000 requests/hour), I believe that is ok, since that implies their architecture is built for massive data requests. On the other hand, bypassing those rate limits with proxies is definitely bad. If a website does not have an API (BuzzFeed), I take care to only collect data that I need. Not anything that would damage the business. (E.g entire articles). Consequently, I sanitize the data of such things if I decide to release the dataset.
- jdc 11y agoWhy do you think scraping needs justifying in the first place?