17 ms·
Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informa
by minimaxir 10y ago
Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues-after-scraping-of-user-data/d/d-id/1113362 http://www.informationweek.com/software/social/linkedin-sues...
OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupid-profiles-taken-down-dmca-claim/ https://www.engadget.com/2016/05/17/publicly-released-okcupi...
Since both of these incidents, I now only scrape if it's a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration.
(Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.)
- yeowMeng 10y agoI am no expert, but I always thought you could scrape without consequence provided you never distribute your scrapings?
- kuschku 10y agoOr you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.
- madamelic 10y agoScraping being illegal is as dumb as saying it's illegal to take photos in public. You aren't affecting anyone if you do it respectfully.
- elmigranto 10y agoBeing dumb doesn't prevent laws from existing (we all know those "funny US law stuff" in the realms of "no kissing toy camels on the cheek (but the mouths are okay)". Also, since this is somewhat untouched territory, don't be so sure that you'll get a judge who is as well-versed in web scarping and infrastructure as you, or shares your opinions on the subject. (And given that precedents are so important in US laws, you better hope someone else before you didn't get such a judge.)
- cookiecaper 10y agoI was actually searching for such a jurisdiction as my startup was shut down by a company that invoked the CFAA late last year. What do you suggest? The EU is even worse than the US when it comes to data freedom and tech access. The law on the books in many former British colonies provides marginally more protection (the "Telecommunications Act"), but it'd probably still be disputable, and you'd be shut down anyway unless you had millions sitting around with your lawyers' name on it.
- detaro 10y agoIt depends on what you are doing. The CFAA is very far-reaching, but of course many aspects were the US answer is "CFAA" are covered by other laws. [EDIT: removed outdated information superseded by european decisions, which make the situation a lot less clear] anti-scraping: If somebody were to offer a telephone book database online and you created a copy of that to sell on your own, you'd almost certainly loose in the EU (since unlike in the US, databases as pure collections of facts have their own copyright protections) The legally safest locations probably are outside the western world if you are targeting western sites.
- cookiecaper 10y ago>Pro-scraping: Last big case I remember here was a flight-search site that did flight search and booking(!) via a scraper and Ryanair lost when they tried to sue them for that, since they couldn't argue convincingly how that was damaging them. Every case I've seen wrt Ryanair (they sue a lot of people) has resulted in a win for Ryanair. Do you have details on the case you're describing? >anti-scraping: [...] Scraping purely factual data is one of my points of defense in the US. I don't want to give it away. >It's still risky though, the safest locations probably are outside the western world if you are targeting western sites. Yeah, this was ultimately the conclusion I had to come to. However, outside the West, the Western companies will just send someone with a briefcase full of $100 bills and pay them off. Corrupt government officials in these locations want the goodwill of a big American company a lot more than they care about any particular random guy. There is only one workable solution: run the service totally anonymously and maintain good opsec so that your cover isn't blown. All under the table. This has its own issues, like making it difficult to receive payment and putting one at much greater legal risk than a mere CFAA dispute, but it's the only option if you don't plan to get shut down.
- hobs 10y agoThis post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
- madamelic 10y ago>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them. In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine. tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.
- muglug 10y ago> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
- cookiecaper 10y agoEh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too. Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser. Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect. Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser. It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.
- cookiecaper 10y agoYes, it's important to understand that in the United States, web scraping is usually an illegal activity under the CFAA. If you draw enough attention, your scrape target will notice and threaten you, and probably follow through with the suit. Since the CFAA prescribes both civil and criminal penalties, you may even find yourself in jail for accessing data without the company's approval. Aaron Swartz was being prosecuted under these provisions for scraping public domain data. The CFAA is a really bad law and creates the network effect lock-in that we all considered a natural part of the web. It doesn't have to be that way -- users should be free to use any browsing appliance they want, including so-called "scrapers". Big companies like Google not only got their start by flagrantly violating the CFAA, copyright, and privacy laws, but they continue to do so. The moral of the story is hurry up and get big before you get sued or arrested. There's a long history of ridiculous web scraping rulings based on technical misunderstandings by neophyte judges, including Ticketmaster v. RMG, where infringement was found because the company scraped data out of a page with the Ticketmaster logo on it. Facebook sued a company called Power Ventures which read out only the user's own data. The founder was found personally liable for $3 million in damages. Facebook did this because they don't want it to be easy for their users to move between social media services. If it's easy, Facebook has to compete on merit instead of just keeping switching costs high. Facebook doesn't like that, so they sue people who make it possible -- and the law says they should win. We badly need a revised law, but the powers-that-be will strongly oppose it because it would threaten their monopoly over web properties. They continue to flaunt their strategic ignorance of these laws and then take shelter behind them to stop risk from small innovators (i.e., having to compete fair and square). In the real world, we have a lot of laws that mostly prevent this kind of bad behavior. In cyberspace, the structure is such that most of those laws are not applicable. We need to update and port the pro-small-business logic we have for meatspace companies so that it counts online too. The state of affairs online is really bad. I want to get a law called the "Consumer Data Freedom Act" passed, which would allow users to access any web property with any non-disruptive browsing device, including custom scrapers that don't impose much more load than a typical user browsing session would.
- 0xdeadbeefbabe 10y agoYes laws are neat and a reason for attending law school I suppose. I'm of the simpleton opinion that TCP/IP and the other protocols are the law of the net, and you ought to start with those.
- twa927 10y agoHow can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
- minimaxir 10y agoThat analogy is not equitable. If you take photographs of a building while on the building's property, they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.
- mindslight 10y agoSure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it. The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be out of jobs. As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database.
- cookiecaper 10y ago>As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database. This would be copyright infringement, since the content of the page is a substantive unique work that is automatically copyrighted by its author. A site that doesn't want you scraping its content is not going to want you posting dumps of its pages. Much like BitTorrent, they'd get into the protocol and send subpoenas to the ISPs behind the IPs that serve their pages, and use that info to sue the customer. When my company was shut down by a legal threat related to scraping, I did suggest to my lawyer that we create something like a browser extension that would grab the data we needed out of normal client-side browsing sessions. This wouldn't be as nice as controlling the flow of information ourselves but it would've worked OK. My lawyer strongly suggested avoiding that as it could've been construed as conspiratorial conduct that would've made criminal prosecution more likely.
- jurgenwerk 10y agoThere are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?
- cookiecaper 10y agoThey aren't, or at least, they won't be if Google decides it doesn't like them anymore and decides to bring the matter to court. The CFAA says it's a crime to exceed "authorized access". Authorized access is whatever the server's owner says it is. If they change their mind, you must cease and desist or risk both civil and criminal penalties. A contract defining the length and nature of your authorization from the server's owner would go a long way to establishing your rights to access, but no one is going to give that to a small player.
- lossolo 10y agoYou forgot to add: In USA.
- cookiecaper 10y agoUnfortunately this is true in almost all of the developed world. While developing countries may not have specific legal prohibitions, we know that that doesn't stop big companies from having their way. Heck, even a first-world country like Sweden couldn't resist the pressure from Hollywood to prosecute and jail the operators of the Pirate Bay, which had long been recognized as totally legal in Sweden. Another issue is that on the internet, jurisdiction is a very messy affair. An American judge will likely determine that California and/or the federal government has jurisdiction over such a case because Google is based in California. Most developed countries have treaties with one another that allow them to enforce foreign civil judgments on behalf of the jurisdiction that entered them. Most developed countries also have mutual extradition treaties. The countries that don't can easily be paid off by an interested party.
- lossolo 10y agoThey probably doing it from country where it's legal. In most countries there is no law that would be applicable in this case.
- franciskim 10y agoOn the ethics side, I don't scrape large amounts of data - eg. giving clients lead gen (x leads for y dollars) - in fact, I have never done a scraping job and don't intend to do those jobs for profit. For me it's purely for personal use and my little side projects. I don't even like the word scraping because it comes loaded with so many negative connotations (which sparked this whole comment thread) - and for a good reason - it's reflective of how the the demand in the market. People want cheap leads to spam, and that's bad use of technology. Generally I tend to focus more on words and phases like 'automation' and 'scripting a bot'. I'm just automating my life, I'm writing a bot to replace what I would have to do on a daily basis - like looking on Facebook for some gifs and videos then manually posting them to my site. Would I spend an hour each and every day doing this? No, I'm much more lazier than that. Who is anyone to tell me what I can and can't automate in my life?
- ccvannorman 10y agoThis is exactly my response to "you can't legally scrape my site because of TOS." I don't think anyone has a legal right to tell me HOW I use their service. Making "browsing my website using a script you wrote yourself" illegal is akin to "You cannot use the tab key to tab between fields on my website, you must only use the touchpad to move the cursor over each field individually." It's baloney.
- cookiecaper 10y agoUnder the CFAA, they do have the right to determine what constitutes authorized access. If they say you're unauthorized for using the wrong buttons on your mouse, then you're unauthorized. It's treated very similarly to trespass on private land. You can try telling the judge it's baloney, but if he's going by current precedent, he probably won't agree with you.
- ccvannorman 10y agoIt is not the same flavor of violation as trespassing on land, more like riding a bicycle on a sidewalk. Fortunately, I'm not in the business of scraping sites, but I still find this legal precedent abhorable, and I hope it gets struck down in court when push comes to shove. I would certainly vote that way if given the chance.
- sandGorgon 10y agoThe code is pretty cool. Thanks for releasing that! May I ask why you built your own scraper infrastructure and not build it on top of a known framework like scrapy (which is in python as well).
- siegecraft 10y agoI think the lessons learned from those lawsuits was to always have some sort of 3rd-party intermediary scraping consultancy firm you engage that is totally not just your business under another name.
- headmelted 10y agoOne thing I'm not quite clear on here. I understand the use of ToS clauses to prevent scraping but I do kind of wonder to what extent they have authority here. IANAL, but surely this would fall under copyright law? While re-publishing copyright-protected data without consent is probably unlawful in your region (like scraping an art site and re-posting the images), I wouldn't think just scraping data points for a different purpose (like scraping amazon for the purposes of price comparison) is nearly so clear cut (or enforceable), but maybe I'm just naive.
- cookiecaper 10y agoThe content falls under copyright law. The problem is that you have to enter the company's servers to obtain this data, and the CFAA says that the company can treat their public-facing web servers like private property, and if you're caught "trespassing", you can be sued and jailed. Scraping plaintiffs are usually granted an injunction based on "trespass to chattels" (among other rationales), i.e., trespass to an individual's property (as opposed to land). Companies like PriceZombie are forced to stop because the CFAA says that Amazon can prevent them from accessing their servers by decree alone. A ToS isn't even really necessary for this, but it helps them pin down their argument. PriceZombie could try to get the data from third-party caches, but it only solves part of the problem, because copyright and trademarks come back into the picture once you have a replica of the target page. In Ticketmaster v. RMG Technologies, the judge found RMG infringing on Ticketmaster's trademarks and copyrights because the page they were scraping included Ticketmaster's logo. The judge said the copy of the full page that existed momentarily in RAM while the scraper extracted the non-copyrightable data constituted a copy that infringed on Ticketmaster's rights, even though the logo was never used by the application in any way, it just happened to be on the page.
- downandout 10y agoObviously it's a good idea to follow TOS. But as a practical matter, they have to know that you're doing it before they can take action. You wouldn't want to put up a site announcing that you're selling scraped LinkedIn data, for example. But if that data is valuable to your business - collecting names of people that work in certain positions at certain companies so that you can do targeted snail mail campaigns for example - you could quietly scrape and use the data without issue. Use proxies and prosper.
- cookiecaper 10y agoThis only goes so far, and if you get found out, you're looking at willful infringement (which usually triples damages) and probably criminal charges under the CFAA. However, it should be acknowledged that there are many people making quiet livings off scrapes that are not legal. There are even a few companies making loud livings off such scrapes, like Google. If you're not going to run it totally anonymously, you should be prepared to jettison and repackage it when you get found it (so that you appear to be complying with the C&D). Scraping is a huge part of the web, and everyone does it. It sucks that it has to live underground because only big companies can duke it out in court.
- downandout 10y agoTOS violations have been found to not be subject to the criminal provisions of the CFAA. The only circumstance under which it would become a criminal issue is if they successfully sued you for it, and obtained a judgment that included a provision ordering you to cease scraping. Ignoring such a court order would then potentially expose you to a criminal contempt of court action.
- tangue 10y agoIt would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.
- sjwright 10y agoAs the owner of a large website, I don't care what you think. I block by default and whitelist when I decide it's in my interest. If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic. Unless you have a way for me to automatically identify "honourable" scrapers such as yourself as distinct from the thousands upon thousands of extremely dodgy scrapers from across the world, my policy shall remain.
- bigtunacan 10y agoAs the user of large websites I don't care. I'm not going to read the TOS and I will continue to scrape what I like since it makes my life more convenient. Like OP when blocked I'll just drive my scraping through a web browser which is the same as I've done for years on various sites that never provided APIs.
- louprado 10y ago"As the user of large websites I don't care". Are you sure ? Do you want your OK Cupid or LinkedIn profile to be crossposted on another website without your knowledge.
- ChuckMcM 10y agoI was going to post something similar. When you go to all that trouble that the web site owner is pretty clearly trying to prevent, that is convincing evidence that you are breaking the terms of service. And breaking the terms of service for a web site has been held to be a civil violation (a number of times on Ebay and Amazon) and potentially a CFAA violation by the Justice department.
- downandout 10y agoActually it's been held that TOS violations are NOT subject to the criminal provisions of the CFAA.
- ChuckMcM 10y agoAre you referring to the MySpace case? Or the July 2016 decision by the Ninth Circuit (https://cdn.ca9.uscourts.gov/datastore/opinions/2016/07/05/14-10037.pdf https://cdn.ca9.uscourts.gov/datastore/opinions/2016/07/05/1...). In US v Nosal it seems like they come down in favor of a CFAA violation if the user acts in an unauthorized way. The author of the piece talks about bypassing captcha's which are, in one interpretation, a demand for authorization (by proving that you are a human and not a program) and by circumventing that authorization they have stepped quite clearly into CFAA territory. If you were referring to a different decision I'd love to read it. I follow this stuff (and at one time explored what legal action our startup could take against scrapers). In our case we also offered a paid API so it was fairly easy to establish damages.
- downandout 10y agoThe case you're referring to is an entirely different set of circumstances. From the text: "The panel held that the defendant, a former employee whose computer access credentials were revoked, acted “without authorization” in violation of the CFAA when he or his former employee co-conspirators used the login credentials of a current employee to gain access to computer data owned by the former employer and to circumvent the revocation of access. " I think that case is unambiguous - this guy was using someone else's credentials to access secured systems after having been explicitly told that he could not. I was referring to the MySpace case. I don't think these two cases are in conflict; IMO they are very different. Additionally, for our purposes in this comment thread, we're talking about scraping of publicly available websites by outside parties, not by former employees whose access has been explicitly revoked. That is different than either of these cases.
- jbmorgado 10y agoI actually wonder if that constraining by the ToS has any legal validity in EU. Since in here, typical that kind of stuff cannot be enforced legally.
- kbenson 10y ago> (Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.) I played with the idea of creating some social aggregation type service with some friends (as a business). The more I read about FB's past behavior with regard to this, and how essential they are to any sort of service like, that, I canned the project. Regardless of what their TOS say, if you get on their radar and they send you a cease-and-desist, it's game over. Facebook is not in the business of subverting their revenue stream, so if you are making money off them and it's preventing them from capitalizing on their users, don't expect to last long if you exist by their grace. Really, there's an interesting space between so small nobody cares and large enough that getting shut down is a real problem. A lot of projects start small and end up (relatively) large, but without a good way to pay for the service itself. While not every service needs to be a business and make money, once you reach the level where you risk either being shut out of your data source or you need to somehow work out an understanding with that source, how do you approach that when being able to pay is off the table? Not to mention the problem approaching before you have to and forcing the situation, or waiting too long and risking the wrath of the source because you've abused their service as long as you have. Has anyone else been in this situation and found an approach that works?