12 ms·
Web Scraping and Crawling Are Perfectly Legal, Right? (2017)
- nkozyra 7y agoThe short version is you can be sued for pretty much any reason. And as an individual you're unlikely to have the resources to defend yourself in such a suit. In most cases described as "gray area" it comes down to cases where the stronger party made a compelling argument against what might have been a very valid reason to scrape. The threat is often wielded as intimidation and intimidation alone.
- wills_forward 7y agoSo true. And the United States has 2x the number of civil cases per capita than the next most litigious country, The United Kingdom. Tort reform just isn’t a sexy campaign promise either.
- Causality1 7y agoI've always wondered why there's not a federal law requiring that the losing party of a suit is required to pay the winner's legal fees, up to an amount equal to what the losing party itself spent.
- dooglius 7y ago"The rationale for the American rule is that people should not be discouraged from seeking redress for perceived wrongs in court or from trying to extend coverage of the law. The rationale continues that society would suffer if a person was unwilling to pursue a meritorious claim merely because that person would have to pay the defendant's expenses if they lost." via https://en.wikipedia.org/wiki/American_rule_(attorney%27s_fees) https://en.wikipedia.org/wiki/American_rule_(attorney%27s_fe...
- Causality1 7y agoSeems like it would have the opposite effect, discouraging everyone but the wealthy from filing claims. 60% of Americans don't have an extra thousand dollars, let alone the means to support real legal representation.
- deleted 7y ago[deleted]
- TrackerFF 7y agoWhere I'm from (Scandinavia), we have laws like that. You pay for your and the other sides legal fees. While you rarely, if ever, see cases where losing parties have to pay millions - I personally haven't seen any like that - you can see people having to pay tens to lower hundred thousand $ in legal fees. That can be a substantial sum for regular people, and those circumstances happen when the other party has hired in high-$$$ lawyers, and the case goes on for a while. Especially if there's multiple appeals. So there's always the risk of that.
- jjeaff 7y agoSeems like that would have the undesirable effect of making large corporations untouchable by anyone except the government and other large corporations. If you want to sue a big company for treating you poorly, you risk a very high price of you lose.
- yub 7y agoThat’s the point of the American Rule - every side pays their own way.
- RichEO 7y agoThat only holds if the laws allow the large corporation to treat people unfairly.
- icebraining 7y agoEven if the law is fair in 99.9% of the cases, if that 0.1% means you're on the hook for more than you can afford, it's rational not to risk it.
- liability 7y agoIf "lawyer" were a political party, they would control the Senate. Perhaps the American legislature goes too easy on the lawsuit industry out of some form of professional courtesy.
- elliekelly 7y agoThe US is unusual in that if you bring a civil suit and lose you don’t have to compensate the other party for their legal fees. Unusually high medical bills in the US are also probably a contributing factor. I’d be curious to see how often a civil suit is brought after a car accident in EU vs US.
- PeterisP 7y ago> I’d be curious to see how often a civil suit is brought after a car accident in EU vs US. Depends on the country of course (UK is probably closer to USA instead of the rest of EU with civil law), but in general that would very rarely result in a dispute that needs to be resolved in court. The mandatory liability insurance would cover the usual claims without a civil suit, and unusual claims are unlikely to succeed and expensive to pursue, so they aren't.
- fourthark 7y agoIt's pretty common after an auto accident in the US (at least in no-fault states) to sue the insurance company for pain and suffering. They will only cover medical bills and damages otherwise. Vis the derogatory term "ambulance chasers" for personal injury lawyers.
- tomhschmidt 7y agoThis article is a bit dated. AFIK the latest in web scraping legality is LinkedIn vs. HiQ, where HiQ was scraping public LinkedIn profiles. LinkedIn issued a C&D under CFAA, but HiQ received an injunction that allowed it to continue scraping. This was supposed to be tried in the Ninth Circuit court over a year ago, but not sure what happened https://www.eff.org/cases/hiq-v-linkedin https://www.eff.org/cases/hiq-v-linkedin
- staticautomatic 7y agoPretty sure the ninth published what's now the controlling opinion in the circuit. YMMV in other circuits. EDIT: Yep. hiQ Labs, Inc. v. LinkedIn Corp., 273 F. Supp. 3d 1099 (N.D. Cal. 2017) "In summary, the balance of hardships tips sharply in hiQ’s favor. hiQ has demonstrated there are serious questions on the merits. In particular, the Court is doubtful that the Computer Fraud and Abuse Act may be invoked by LinkedIn to punish hiQ for accessing publicly available data; the broad interpretation of the CFAA advocated by LinkedIn, if adopted, could profoundly impact open access to the Internet, a result that Congress could not have intended when it enacted the CFAA over three decades ago. Furthermore, hiQ has raised serious questions as to whether LinkedIn, in blocking hiQ’s access to public data, possibly as a means of limiting competition, violates state law."
- snazz 7y agoThat’s an extremely well-thought-out court finding, especially since they thought about how the CFAA was intended to be applied. Far too often we see a more liberal interpretation of the CFAA used to the detriment of a small player.
- brighter2morrow 7y ago>In particular, the Court is doubtful that the Computer Fraud and Abuse Act may be invoked by LinkedIn to punish hiQ for accessing publicly available data; the broad interpretation of the CFAA advocated by LinkedIn, if adopted, could profoundly impact open access to the Internet, a result that Congress could not have intended when it enacted the CFAA over three decades ago. I didn't realize original intent could be used in courts. How the heck did "original intent" lead to federal abortion and federal gay marriage when all law in letter and in practice had delegated these questions to the states?
- modeless 7y agoIf search engines hadn't started scraping the web back in the 90s, imagine the outcry if you tried to start a search engine today. You would instantly be sued into oblivion. Copyright, trademark, TOS violations, etc. Nothing remotely similar to a modern search engine would ever be allowed to start if it wasn't grandfathered in by the time people started getting serious about law enforcement on the internet. The law is not as clear cut as people imagine it to be. In many cases the text of the law is less important than precedent and established practice.
- deleted 7y ago[deleted]
- beamatronic 7y agoIf you tried it today, you would need a paid subscription to get access to a search engine
- quickthrower2 7y agoThere were paid search engines in the 90's. I don't recall the names. Maybe the fact I don't remember the name says it all!
- ReinholdNiebuhr 7y agoI'd be okay with a paid subscription service if I got privacy and some other things as well. There's room for both free and pay. Duckduckgo is ok, but not google. Would be nice to have google, without the bias and without the privacy issues.
- treerock 7y agoExcept perhaps Google is actually better at search because it knows so much about you. In that case a Google without the privacy issues is basically the same as duckduckgo.
- mosselman 7y agoI always hear people complain about DuckDuckGo and it gives me the impression that they haven't actually used it for a while. I only use DuckDuckGo and I find it to be a great search engine.
- tingletech 7y agoneeds a (2017)
- deleted 7y ago[deleted]
- nshepperd 7y agoThe idea that you could be sued[1] with breach of contract for violating a ToS in a case like this is goddamn insane. A contract can only be formed if both parties agree to it; and you can't be forced to agree to a contract against your will. Coercing someone into entering a contract is literally illegal. So just don't accept the ToS. They can't make you accept it, no matter what "by [breathing] you accept our ToS" verbiage they put in it. Sure, maybe that would make your access of the website "hacking" ("accessing a computer without permission", since you rejected the contract which would grant permission)... but the court rejected that argument in HiQ vs LinkedIn. [I get that this probably isn't how a typical court would see it, but to me that reflects pro-corporate bias and corruption more than any kind of sane application of common law.] [1]. Edit: originally this read "charged", but "sued" is more accurate. But I guess what I really mean is "successfully sued". Anyhow.
- elchief 7y agoIt's funny how a 3-line bash script is considered an automated agent, but the 100's of millions of lines of code that Chrome uses to download, parse, and display HTML is somehow not "automated"
- danso 7y agoWhy is the number of lines of code particularly relevant?
- elchief 7y agoChrome uses a LOT of automation to do what it does The actual number of lines is not important
- gitgud 7y agoChrome (to my knowledge) doesn't automatically download webpages, it strictly downloads what you type into the url. This begs the questions; - At what point does web-scraping become research? - Is it considered scraping if a human is actioning it (no automation)? - Could I manually click links and copy the data from the browser, is this scraping? There's no strict definition, so it's up to a judges and lawyers to decide what the definitions entail...
- mantap 7y agoChrome does automatically download webpages: it supports prefetching and iframes. Not only is it up to judges and lawyers but there's >190 juresdictions in which you could be sued. There are more important things to worry about, such as heart disease.
- gitgud 7y ago> There are more important things to worry about, such as heart disease. Exactly, so much of humanity is wasting time needlessly arguing in courts... we need better systems of laws to reduce these redundant debates...
- 7y ago
- calebclark 7y agoWhat happened to the web's early vision of an information superhighway, an open repository of data that could be linked, shared and remixed? The efforts (and lawsuits) to lock down and control data on the public web threatens to stifle innovation. Websites are only valuable because they're connected to the internet, which for all intents and purposes, is a general utility. It was funded by the U.S. government, and is now managed by a consortium of NGOs such as ICANN, W3C, etc. When you publish a website, you're distributing it to the world. It's similar to when you publish a book. It has always been the right of those who buy and read a book to be able to use the facts contained therein. Although readers are not allowed to steal the creative elements, they have a basic human right to use the facts however they desire. And the medium doesn't change this truth -- whether you read the book yourself, ask your friend to read it out loud, or employ a bot to read it. So it should be with websites. Data placed on the public Web is accessible by everyone and should be usable by everyone. It shouldn't matter who parses the page's HTML, whether it's a person, a web browser, a bot or a robot. We're starting a movement to ensure that the facts of the world are always available for innovators to build on top of: https://dataliberationfoundation.org/ https://dataliberationfoundation.org/
- masonic 7y agoIt's similar to when you publish a book. When you publish a book, copyright and Fair Use still apply.
- calebclark 7y ago100% agree.
- ospider 7y agoThings are a little different here in China. The government just published a law that if you scraping traffic is more the one third of the total traffic of a website, it is considered illegal.
- meritt 7y agoWhoa. Do you have a link to the legislation or news?
- severine 7y agoIt doesn't reference that measure explicitely, but you might find this article interesting. China’s Draft Data Security Measures and How They Compare to the GDPR: https://www.natlawreview.com/article/china-s-draft-data-security-measures-and-how-they-compare-to-gdpr https://www.natlawreview.com/article/china-s-draft-data-secu...
- Mirioron 7y agoI hope there's some kind of minimum there or some warning system, because you might not know what a third of a specific website is. On the other hand, this kind of a rule seems somewhat sensible, because it prevents web scraping from ballooning the costs of the service.
- CaliforniaKarl 7y agoHere's a hypothetical scenario: You've just made a container image that does something cool, and have published it on your site, which is running on the cheapest Amazon Lightsail plan (in Ohio). The final container image is 50 MB in size. (Yes, it's weird that you wouldn't publish it to Docker Hub or the like. Suspension of disbelief, please…) A grad student somewhere decides they like your container. They write a script that downloads your container to $TMP, and runs it with their input. Over a week, the student runs their job 50,000 times. (Running a compute job 50,000 times is not unusual in HPC. Each run has a different input, and since your cluster would have a job scheduler, breaking up the work helps make better use of the cluster.) Because the student downloaded the container to $TMP, a new download has to happen each time the job is run. 50,000 downloads of a 50 MB container is 2.5 TB. (The docs say TB and GB, so I'm assuming base-10 instead of base-2 here.) Your lightsail plan is $3.50 per month, and includes 1 TB of transfer. You will have at least 1.5 TB of extra transfer this month. At 9¢ per GB, 1500 GB of extra transfer is $135. Please do not dismiss this out-of-hand. The scenario may seem a bit contrived, but unexpected bandwidth charges do hit people, and can make you really question your dedication, when having to pay for the access by people you don't know.
- codingslave 7y agoIf you don't want to be charged for the compute power, don't make the service.
- CaliforniaKarl 7y agoIn this example, the person who created the container image isn’t paying for the compute power, they’re paying for the bandwidth cost of those users downloading the thing.
- syntheticcdo 7y agoI would be responsible. It is up to me - the server operator - the fulfill my contract with Amazon. I COULD have set up billing alerts, rate limiting, WAF, etc to avoid being charged or to shut down the unexpected usage, but I didn't. I put something on the internet so I am responsible for it.
- hardwaresofton 7y agoIt seems like the biggest step forward that scraping/crawling could take is to get some relevant legal precedent -- is it possible to manufacture/make a case that could finally set the legal precedence for this? For example on the issue of insta-agree TOS's that you are in no way forced to read ("by being on this website you agree to our TOS" BS), is it possible to make cases to establish precedence from that? Let's say I set up a site where the TOS says everyone that views the site agrees to give me $5 and I sue someone -- clearly that wouldn't hold up in court (gulp, hopefully) -- and having it struck down would help clear up this gray area...
- mgamache 7y agoTry and scrape scores or sports results or stock prices (and re-publishing). Pretty sure you will get a "cease and desist" if you get any traffic at all. That data is owned by corporations even if it is publicly displayed.
- teddyh 7y agoThat data isn’t “owned”. Mere data such as you describe (sports results or stock prices) cannot be owned. There is no legal mechanism for it. Copyright can’t cover mere facts such as those, and patents or trademarks don’t apply, nor does trade secrets, obviously. However, the corporations involved sure would like you to think otherwise, and they will fight tooth and nail to keep it that way.
- mgamache 7y agoIt's complicated... https://pando.com/2014/02/06/who-owns-real-time-sports-data/ https://pando.com/2014/02/06/who-owns-real-time-sports-data/
- icebraining 7y agoWell, not in the US. In the EU, on the other hand: https://en.wikipedia.org/wiki/Database_Directive https://en.wikipedia.org/wiki/Database_Directive
- hyfgfh 7y ago"But law has apparently nothing to do with fairness. It's based on rules, interpreted by people." That`s why I`m chaotic!
- jarfil 7y ago1. Create a random web page with some random words. 2. Add a ToS notice stating that anyone downloading it now owes you $1000. 3. List every client IP in the logs and sue everyone even if you don't know who they belong to. 4. Profit?
- hartator 7y agoDisclaimer: I run SerpApi.com and IANAl. In the US, the law has kind of settle in favor of scrapers. Thanks to the first amendment and the fair use exemption. It the data doesn’t require to sign up and your are not DDOS the website. Europe is another story.
- rayascott 7y agoThe author attempts to contrast web scraping and web crawling and gets it very wrong. This is what it actually is: web scraping - I download the content of a webpage or select webpages, web crawling - I download the content of the entire internet (or part thereof). How he can say that someone who's crawling the internet and building a search index with the contents of the downloaded data isn't also scraping the web, makes no sense.
- zaro 7y agoI am confused. Where does robots.txt come into play here? I always thought that if url is allowed in robots.txt it's fair game to scrape.
- patagonia 7y agoJust a thought. The entire user data / data broker business is just businesses “scraping” my digital presence. No one is respecting my terms and conditions. They’re making trillions off that.
- patagonia 7y agoComments accompanying downvotes always appreciated.
- maitredusoi 7y agoIs life perfectly legal ?
- clashmeifyoucan 7y agoNow, what I'm not sure about and don't have the means to consult a lawyer about is 1) How liable is the author of a scraper who put it on GitHub but never did use it themselves. Because I remember seeing a lot of implementations and tutorials of tools like BeautifulSoup that don't necessarily comply with the ToS. 2) What if you edit the User-Agent to something that resembles a browser provided the scraper is just substituting browser interaction, e.g. those search from your terminal applications. Could/would it be construed as impersonation or something?
- kamfc 7y agoIt made billions for companies. The rule of thumb is to start off "legal", build a big audience with huge benefits to the companies you're burrowing the data from, then quickly shift to legal. If you're not fast enough, you're dead. Web scraping and crawling has become less acceptable because the market leaders carved out the niches before you in a very "legal" way, and DO NOT want you as a competitor.