3 ms·
whether it’s legally OK to scrape websites without using an API I'm not a lawyer either, but making such a frivolous distinction has always bothered me --- HTT
by userbinator 7y ago
whether it’s legally OK to scrape websites without using an API
I'm not a lawyer either, but making such a frivolous distinction has always bothered me --- HTTP(S) and HTML is an API, and it's the one the web browser uses. Maybe the "official" API offers some better formatting and such, but ultimately you're just getting the same information from the same source. As long as you don't hammer the server to the point that it becomes disruptive to other users, as far as they're concerned you're just another user visiting the site.
IMHO making such a distinction is harmful because it places an artificial barrier to understanding how things actually work. I've had a coworker think that it was impossible to automate retrieving information from a (company internal) site "because it doesn't have an API". It usually takes asking them "then how did you get that information?" and a bit more discussion before they finally realise.
"If you asked a hundred people to go to different pages on a site and tell you what they found, is that legal?"
- nostrademons 7y agoThe distinction is usually based on implied consent. The general legal principle is that if you own property and you grant consent for people to use it for one purpose, they are free to use it for that purpose, but you haven't necessarily granted consent for other purposes. Offering an API is a strong indication that you actually intend to allow people to consume the data with software, because otherwise you wouldn't have bothered. Offering an HTML interface is usually an indication that you intend for people to consume the data with a web browser. Offering an HTML interface may be an indication that you also consent to allowing machines to read the data through the HTML - that's the idea behind search engines. But that's where it gets complicated, and that's why there's all sorts of other considerations to the legal question. Things like did you include the pages in question in robots.txt, did you say anything explicitly about scrapers in the ToS, does the scraper offer a way to contact its owner about abuse, has the website actually contacted them, has an IP ban been issued, is the scraping for commercial purposes, does it compete directly with the site, does it interfere with legitimate human use, etc.
- closeparen 7y agoWhen I buy a book, the publisher has no say in whether I use it for personal enjoyment, or for class discussion, or to write a (potentially negative) review, or to feed my fireplace. They have some control over wholesale reproduction via copyright law, not arbitrary power to decide what I do with it like, say, a restaurant that says I can only use my seat at the table to eat their food for a reasonable amount of time. Why would bytes on the wire be any different from printed words on the page here?
- nostrademons 7y agoThere's a distinction between the physical pages of the book and the text contained within the book. You own the physical pages of the book; you can use it as a paperweight, coaster, toilet paper, fireplace fuel, whatever. You do not own the copyright on the words of the book, and in many of the cases you list, the publisher does have a say in that. If you want to put on a school play based on the book, you need to get permission from the author. (My high school put on an in-house adaptation of Out of the Dust, and we had to write Karen Hesse and get her okay to do so.) If you put the entirety of the book on your website so that readers of your negative review can refer back to it, the publisher can come after you with a cease & desist or, if you ignore it, a lawsuit. If you write fanfiction based on the characters in the book, the publisher can come after you with a C&D. If you want to make a movie based on that book, you need to buy the film rights. (There's currently an interesting situation with Game of Thrones where HBO owns the film rights to the world of Westeros, but the film rights to the characters & story of Dunk & Egg are still owned by GRRM, so if the film rights to the earlier Dunk & Egg stories were ever bought by a studio other than HBO, they would have to be scrubbed of mentions of Targaryens, the Iron Throne, King's Landing, etc.) In the pre-Internet days, the chance of enforcement was next to nil for many of these cases, because the big studios and publishers all got licenses for any IP, while class discussions, high school plays, and hobbyists never got a wide audience for their work and so the original publisher would probably never know (unless you did something really stupid like send it to them). The Internet's blurred a lot of these boundaries.
- 7y ago
- SilasX 7y ago>IMHO making such a distinction is harmful because it places an artificial barrier to understanding how things actually work. I've had a coworker think that it was impossible to automate retrieving information from a (company internal) site "because it doesn't have an API". It usually takes asking them "then how did you get that information?" and a bit more discussion before they finally realise. Yes! Similar pet peeve about e.g. "You can't use encryption in Gmail." No, nothing stops you from encrypting the message outside of Gmail and pasting the ciphertext in your message's body. It's that e.g. there might not be native support in the web client.
- pvg 7y agobut making such a frivolous distinction has always bothered me Dismissing various legal and social conventions as 'frivolous distinctions' is, in the end, probably a more harmful viewpoint than the inconveniences the 'distinctions' introduce. It's also too easy to apply it in arbitrary and self-serving ways. Scraping data off some website? Frivolous distinction. Someone hoards your personal data? Venal violation of your privacy rights.
- malshe 7y agoI agree. This is exactly why I made that distinction.