4 ms·
Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have y
by mindslight 10y ago
Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it.
The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be out of jobs.
As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database.
- cookiecaper 10y ago>As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database. This would be copyright infringement, since the content of the page is a substantive unique work that is automatically copyrighted by its author. A site that doesn't want you scraping its content is not going to want you posting dumps of its pages. Much like BitTorrent, they'd get into the protocol and send subpoenas to the ISPs behind the IPs that serve their pages, and use that info to sue the customer. When my company was shut down by a legal threat related to scraping, I did suggest to my lawyer that we create something like a browser extension that would grab the data we needed out of normal client-side browsing sessions. This wouldn't be as nice as controlling the flow of information ourselves but it would've worked OK. My lawyer strongly suggested avoiding that as it could've been construed as conspiratorial conduct that would've made criminal prosecution more likely.
- niftich 10y agoNot the discount the validity of your experience, but the usual counterpoint to this is Google, who (like mentioned elsewhere in the thread) has been continuously scraping since the very beginning and in fact built their entire business model on doing so. They are also responsible for advancing the state-of-the-art of scraping (albeit mostly internally), through the development of V8 and headless Chromium so that they can inspect dynamic pages too. Perhaps this illustrates the fungibility of the legal system: it's an inherently human construct that pits a plaintiff against a defendant, and given a big enough warchest and persuasive-enough arguments, catastrophe can be avoided -- by Google; perhaps not by you, me, or someone else.
- cookiecaper 10y agoYeah, Google violates the CFAA and infringes on copyright as a matter of course. Their service would be impossible if they weren't doing so. The main difference when Google was small was that Google was not dependent on any data source in particular, so even if someone denied their robot or sued them, they could cease and desist without affecting the overall value of their offering. This is different if you are getting data that is only available from one or two sources. Now, the main difference is that Google is one of the biggest companies in the world, and they'll sick an army of $1,000/hr lawyers on you if you even think about taking legal action against them. The only people who can afford to fight are other big companies, but that's not going to happen because they all depend on breaking the CFAA for their own purposes and then using their position as a huge company to bully small innovators.
- wtracy 10y ago> they'll sick an army of $1,000/hr lawyers on you They don't even need to do that. They just cheerfully agree to not scrape you, and wait for you to come back and beg to be re-instated when your search traffic plummets.
- mthoms 10y agoGoogle's crawling and caching has been largely found to be fair use and thus is not considered to be infringing copyrights. https://en.wikipedia.org/wiki/Field_v._Google,_Inc https://en.wikipedia.org/wiki/Field_v._Google,_Inc. There are similar rulings for thumbnail images: https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com,_Inc https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com.... And of course books: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,_Inc https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,....
- cookiecaper 10y agoIncidentally, this only further proves my point. If you're a big company that's retained massive law firms, you can successfully raise a fair use and implied license defense. If you're not, you can neither mount a strong offense against that defense nor mount a strong defense against Google's hypocritical offense if you find yourself on the other side. Google's primary out here is its reputation (not guarantee) for obeying robots.txt. If Google indexed a page that disallowed it in robots.txt, the case would be much stronger. There's also the unofficial out, which is that judges think Google is a cool large company, so they rule in their favor based on their personal biases. Fair use is a case-by-case basis, so you can't say that Google's infringing conduct is generally accepted to be fair use. The EFF had to take on Universal in Lenz v. Universal Music Group, and that went up to the Supreme Court. That's how individuals are left to assert their fair use rights.
- mindslight 10y agoOh for sure. But BitTorrent is still around and works great!
- SilasX 10y agoIsn't that what webarchive/wayback machine do? I think they use a "Fair Use" defense.
- Asooka 10y agoWell, it's just a technical response code. 200 OK - everything went as normal, here's your data. By the same margin, the door on a shop doesn't stop you walking out without paying and the road markings don't stop you from driving in the wrong lane. I think imbuing technical protocols with legal implications would be even worse than the current situation since then changing anything on a protocol would require changing the law and getting a protocol implementation slightly wrong would carry real-world legal repercussions on the order of licensing your work in the public domain rather than retaining copyright. Let the lawyers make the law and check the human terms of service before using the data. Trying to out-lawyer the lawyers is like challenging a hedgehog to a butt-kicking brawl.
- Doctor_Fegg 10y agoThe protocol is also that you send a valid, non-faked User-Agent: "The User-Agent request-header field contains information about the user agent originating the request. This is for [...] the tracing of protocol violations [...]. User agents SHOULD include this field with requests" Many scrapers disregard this part of the protocol. Of course, whether a headless browser should send a different UA is an interesting question. https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
- TheCoelacanth 10y agoUser-Agent is a SHOULD, not a MUST. There are also practically no browsers that send a non-fake User-Agent, since they almost all claim to be Mozilla/5.0.