10 ms·

Web scraping for me, but not for thee

582 points by mhb 3y ago

rvnx 3y ago
Sweet memories of Facebook, which was spamming all the contacts to invite them to join Facebook.
- ilrwbwrkhv 3y ago
  Same as LinkedIn. In fact that was LinkedIn's actual growth hack. Now they sell books talking about other things.
  - rendall 3y ago
    I remember when LinkedIn had an anti-pattern of displaying a check list of your gmail contacts and asking which of them you want to connect with. These were all unchecked by default. But under the fold - that is, you needed to scroll down to see them - were all the rest of your contacts, only these were all checked. If you just went with what was displayed without scrolling down and unchecking the rest, you would unintentionally blast your entire contact list with contact requests. That was everyone in your contact list, whether appropriate or not: your dog groomer, people you were in a lawsuit with, whoever. I never did figure out how they were able to get our email contact lists. I vaguely remember reading about web-scraping shenanigans. I for one never knowingly gave them access.
SenAnder 3y ago
> Mark Lemley observed this happening nearly 20 years ago, in his prescient, seminal article, “Terms of Use.”: The problem is that the shift from property law to contract law takes the job of defining the Web site owner’s rights out of the hands of the law and into the hands of the site owner. With "contracts" of adhesion proliferating, and how impossible it has become to exist in the modern world without acceding to them (something as simple as buying a new SSD involves agreeing to one), this problem is getting worse by the day. The law is becoming increasingly irrelevant, and more and more we are ruled by one-sided "contracts" from giant companies that are in a position to push them on us.
- nre 3y ago
  > With "contracts" of adhesion proliferating, and how impossible it has become to exist in the modern world without acceding to them (something as simple as buying a new SSD involves agreeing to one), this problem is getting worse by the day. The craziest example of this is how all these contracts are appearing in the physical world as well. There are stores that actually have a sign indicating that entering the store constitutes acceptance of contract terms (with a QR code that you presumably can scan with your phone to read the contract). I've also seen public parks with the same thing basically indicating that entry binds you to a legal agreement to not sue the park/follow posted rules/etc.
  - golemiprague 3y ago
    It is partly due to the crazy litigation in the US and the rewards people get for some coffee they spill on themselves just because the coffee was "too hot". There are two sides to this story and it is not only the business' owners who are the culprits here.
    - scolby33 3y ago
      Look up that case. The victim suffered third-degree burns and required skin grafts. The myth that it’s a ridiculous, over-litigious case is just that: a myth. The coffee was absolutely too hot.
      - genewitch 3y ago
        And it was caused by a profit-seeking behavior - an assumption that most, if not all people, would wait to drink the coffee until after they were out of the vehicle, which could be up to X minutes; as such they wouldn't want cold or lukewarm coffee. If most people equated mcdonalds coffee with "bad lukewarm coffee", less people would go to mcdonalds for anything, including coffee. I forget how hot they served the coffee, but it was up around 95C/200F. I know that you have to specifically request "extra hot" at places like SB and CB&TL, otherwise you get coffee hot enough to still be "hot coffee" after adding creamer, i venture 130F or so. AFAIK McDs doesn't offer "extra hot" coffee. Once bitten, and all. I don't know about the changes made due to this lawsuit, but i reckon they changed their cups to keep ~120F coffee at that temperature longer. edit: aside: I just realized why i prefer Fahrenheit even though i understand Centigrade. Stuff over 100F is "hotter than i am" It's intuitive in a way that merely remembering 37 is body temp (my dad was born in '37 so that's why i remember) and "anything over 40 is hot" doesn't really roll off the tongue, even though it's roughly as accurate as my ">100F is hot"
karaterobot 3y ago
The perceived hypocrisy sort of goes away when you stop thinking about it as a collaboration or a community of equals, and instead think of it as a competition, which is what it is. You would not say of a football team "oh, it's okay for you to try to score a goal on me, but if I try to score a goal on you, suddenly you're blocking the ball?!" Naturally, they're going to say "web scraping uses resources, stop it!" but then keep web scraping in the background. To be clear, it's bad behavior, it's just not hypocritical behavior, as it's completely in keeping with what amoral corporations locked in constant battle would be expected to do: maximize benefits to themselves while minimizing benefits to others.
- Kareem71 3y ago
  The problem is as this article points out is democratically elected courts should not be choosing winners in a capitalistic competition
  - karaterobot 3y ago
    Only commenting on how we should expect corporations to act, or more accurately why we should not be surprised at their behavior.
- backtoyoujim 3y ago
  There are clearly more than two teams in these issues. It is not a game, and it is not football. It is a public policy issue also outpaces "competition" which is merely a subject change.
  - karaterobot 3y ago
    Football is metaphorical in this case.
- philipov 3y ago
  Hypocrisy doesn't require one to believe what they say and utter it in good faith but fall short of those ideals in practice. Equivocating about football teams doesn't change that one is trying to impose standards on another without holding oneself equally to them. It is still hypocrisy, even if they do it amorally in bad faith. Especially then. What matters is what policy you espouse; you don't get a pass for not really believing what you say. The implication is that hypocrites are acting in bad faith.
fasterik 3y ago
>Let’s look at what Microsoft is doing right now, as an example. In the last couple of weeks, Microsoft updated its general terms of use to prohibit scraping, harvesting, or similar extraction methods of its AI services. Also in the couple of weeks, Microsoft affiliate OpenAI released a product called GPTbot, which is designed to scrape the entire internet. And while they don’t admit this publicly, OpenAI has almost certainly already scraped the entire non-authwalled-Internet and used it is training data for GPT-3, ChatGPT, and GPT-4. Nonetheless, without any obvious hints of irony, OpenAI’s own terms of use prohibits scraping. I don't understand why this demonstrates hypocrisy. There is a big difference between crawling the publicly accessible web (which legitimate search engines do all the time) and scraping an authenticated web application or API.
- Atotalnoob 3y ago
  Or that while Microsoft is an investor in openai, they do not control openai
  - dvhh 3y ago
    If SDGs learned us anything is that companies are responsible for to scrutinize the company they are investing in. Of course this is not on the level of child labor, nor environmental pollution.
- johneth 3y ago
  The hypocrisy stems from: 1. OpenAI (etc.) scrape the public web to train and build their models. 2. They use these models to sell subscriptions (make money). None of this goes to the creators of the data used to train the models. 3. They deny others to do what they have done themselves. If you compare to, say, search engines scraping the public web: 1. Search engines scrape the public web to create their search indexes. 2. They use these indexes to provide search results and sell ads around them. Importantly, these search results direct people to the websites that they have scraped (much of the time), offering opportunity for them to make money.
  - fasterik 3y ago
msie 3y ago
The first company that came to mind from reading the title was Google.
version_five 3y ago
Good example from the Allen Institute discussed last week https://news.ycombinator.com/item?id=37181415 https://news.ycombinator.com/item?id=37181415 They "released" a dataset scraped from public domain stuff under a license that restricts how people can use it
waydegg 3y ago
It's interesting seeing the reactions from other websites/orgs after OpenAI publicly announced GPTBot. Tons of people blocking GPTBot outright (made a small page that track this: https://wayde.gg/websites-blocking-openai https://wayde.gg/websites-blocking-openai)
- version_five 3y ago
  I wonder if blocking gptbot is a good signal that a website has non LLM generated content on it, and is therefore good training data...
  - deleted 3y ago
    [deleted]
  - Springtime 3y ago
    An illustrator specifically wrote[1] that this is why they won't be tagging their social media posts with #HumanMade, #NoAI and similar, as it's a signal there's unadulterated training data. [1] https://www.davidrevoy.com/article977/artificial-inteligence-why-i-ll-not-hashtag-my-art-humanart-humanmade-or-noai https://www.davidrevoy.com/article977/artificial-inteligence...
- dontupvoteme 3y ago
  Is there any legal issue with a spider trap designed to poison LLMs?
einpoklum 3y ago
Perhaps it's legal to scrape in some world states outside the USA?
- dontupvoteme 3y ago
  A good question. Japan came out and declared that copyright does not apply to training AI. There must be a good chunk of the world that doesn't have any laws forbidding it. This isn't under the jurisdiction of WIPO or anything like that, it's just a completely insane evolution of anglo common law
deepsun 3y ago
> But the content that they’re trying to protect isn’t theirs -- it belongs to their users. Kinda. Yes, Facebook says that content belongs to users (otherwise they'd have harder time explaining they are not liable when it's illegal), but users also agree to give Facebook “non-exclusive, transferable, sub-licensable, royalty-free, worldwide license to use any IP content that you post on or in connection with Facebook.” For example, if a user deleted their* content, Facebook can still use it and show to their friends. That's why it's "kinda".
- waynesonfire 3y ago
  Did you read the posted sign? “no walking on the road outside my property”
- sib 3y ago
  That doesn't change who the content belongs to. It just gives some rights to FB. Any, in fact, without something like "perpetual" and/or "irrevocable" in there, it doesn't imply that they could keep using it after you deleted it (or that you couldn't revoke a grant of rights.)
- antonf 3y ago
  > For example, if a user deleted their* content, Facebook can still use it and show to their friends. That's why it's "kinda". I don't think it is correct. If you asked Facebook to remove your data from platform, it will be a GDPR (and probably CCPA, etc...) violation for Facebook to not delete your data within 1 month.
- jeremyjh 3y ago
  A license is not ownership. Anyway that part of the article is just context - none of what you describe constitutes the legal basis for the suits or rulings discussed in it - it’s just explaining why property law isn’t being used.
Klonoar 3y ago
Hmmm, I'm a bit confused on something. The HiQ vs LinkedIn case, to my knowledge, went through the following stages: - LinkedIn sues HiQ, Ninth Circuit sides with HiQ - LinkedIn pushes to Supreme Court, Supreme Court vacates citing Van Buren - Ninth Circuit re-reviews and affirms their decision - LinkedIn moves to get the injunction preventing them from blocking HiQ dissolved, which is granted - A mixed judgement is finally issued in Nov 2022 ultimately resulting in a private settlement Where exactly does this leave things at? I feel like everyone loves to cite this case but never goes into the finer details. Reading a summary of the mixed judgement from Nov 2022, it looks like maybe the issue came from HiQ using people to log in and thus the ToS came into play...? If I'm reading correctly, it looks like the court did eventually side with LinkedIn in stating that HiQ violated the ToS. https://www.natlawreview.com/article/court-finds-hiq-breached-linkedin-s-terms-prohibiting-scraping-mixed-ruling-declines https://www.natlawreview.com/article/court-finds-hiq-breache... Edit: Formatting.
- dontupvoteme 3y ago
  What is the legal precedent of a mixed ruling? I was unaware such a thing was even possible.
- KieranMac 3y ago
  Not a mixed judgment in Nov. 22. It was a massive defeat for hiQ Labs. Read the permanent injunction issued by the court.
  - dontupvoteme 3y ago
    >Read the permanent injunction issued by the court. Happen to have a link? The question that matters is if this establishes any precedence.
    - KieranMac 3y ago
      I don't. I just have a .pdf. Email me at Kieran(at)McCarthyLG(dot)com if you want a copy.
    - 3y ago
dclowd9901 3y ago
The primary grounds on which these cases rest is some nebulous understanding of contractual agreement. I have two thoughts: - EULAs aren’t written for companies to sign. - I think EULAs are garbage anyway. They’re completely one sided and in most cases probably illegal or wouldn’t hold up in court if anyone actually had the resources to fight one. Imo, the burden of ensuring someone has read and understands a EULA should be on the company who creates it and they should not be enforceable unless they can prove the person understood the EULA entirely before accessing the site. EULAs are not a business agreement. They’re some kind of corporate pseudo-law companies try to attach to the usage of a product. But what other product in the world has a big list of rules that come with it that way how you can use it (or be sued)? So how does this all come back to this “company vs company scraping”? If you put it on the web, and you don’t have REAL copyright on the content (that is, you didn’t make it yourself), you have no right to protect it from “theft.” PS yes, I know John Deere doesn’t let its customers work on its tractors but that’s some bullshit too.
- KieranMac 3y ago
  These online agreements are often enforceable, even when companies have lots of resources to defend themselves.
deleted 3y ago
[deleted]
sneak 3y ago
If you think about it, if free lending libraries and web search indices did not exist, and you tried to create them today, you would get sued into oblivion.
zarazas 3y ago
How is the situatuon legally and ethically for you to use scraped data as text embeddings for a commercial product?
MBCook 3y ago
I see two issues. Web scraping is clearly a business model problem, and that’s partly due to scale. If you give away your content for free and expect ads to sustain you, that will start failing once others get the value out of your content without seeing the ads. Examples are ad blockers, answers embedded in Google results, Stack Overflow clones, and things like ChatGPT. If ads weren’t your business model you wouldn’t be using revenue from it. The other issue is scale, and I don’t know how to address it. It’s easy for someone (say the government) to have a friendly policy and say “you can use dig in a park” thinking it’s useful to campers and such. But when someone shows up with a professional strip mining crew, things are different. If you run a site providing quality information for free, making money off book sales or professional services or such can be a good living. Even if answers end up in the Google answer box, more complicated stuff or analysis still requires a visit to read and people can start following you from there. But if ChatGPT or whatever can “read” your stuff and give out 80% of the value without anyone even knowing it came from you, you’re screwed. Your business model no longer works. Any kind of “give away good information” business model fails. Same issue artists are now seeing. And I don’t know how you fix that without some kind of ban. But unless every country everywhere enforces one… you have to work with the lowest common denominator and lock all your content up. No web search. No Google answers. No chat GPT. “Please don’t scrape me” in robots.txt won’t work.
- drunkencoder 3y ago
  You’re right . That’s why scraping must be unlimited and legal for all. Any information accessible from internet should be legal to refine. Thus also us using GPT services to train our own models, scraping anything that’s publicly accessible. Our only defense is competing services that refines the data even more than any general llm. The solution is almost never regulation but competition. Fair competition
  - antonf 3y ago
    > You’re right . That’s why scraping must be unlimited and legal for all. Unlimited scraping makes some of privacy regulations moot. Such as right to erasure (ability to delete personal data from a platform).
kazinator 3y ago
> Some of the biggest companies on earth—including Meta and Microsoft—take aggressive, litigious approaches to prohibiting web scraping on their own properties, while taking liberal approaches to scraping data on other companies’ properties. Umm, no; author needs to study the word "hypocrisy" more deeply than a cursory glance in the dictionary. Doing something to others, while defending against the same thing, is not hypocrisy. For instance, soccer player isn't a hypocrite for defending against the ball going into his net, while trying to put it into the other team's net. A soldier on the war front isn't a hypocrite for shooting, while also taking cover and dodging bullets. These subjects are not hypocrites because they are not acting in one way, while preaching that they, or others, ought to be acting in a different way. Microsoft would be hypocrites if they published an official statement such that nobody who engages in web scraping has the right to defend their own site against web scraping, because that would not resemble their actual behavior and position which could be inferred from their behavior. (Is there such a statement somewhere?) For hypocrisy to take place, you have to actually preach that you and others should behave in a certain way, and then not actually behave in exactly that way. If you only act, and don't preach, you cannot be a hypocrite. Moreover, your team's net is not the same object as another team's net. If a soccer player loudly professes "it is morally wrong for anyone to kick the ball into our net", but then kicks the ball into the other team's net, that is not hypocrisy. His statement references only his own net; he didn't proclaim that it's wrong to kick any ball into any net whatsoever.
- KieranMac 3y ago
  Scraping other sites while prohibiting it on your own is "do what I say, not what I do" behavior, which I think is a fair, consensus understanding of what it means to be hypocritical.
  - kazinator 3y ago
    To convict Microsoft or Meta of hypocrisy, you need to find some official statement from the company which says that web scraping is wrong. Hypocrisy is the professing of a moral position to which one does not conform. If you don't preach that web scraping is wrong, and engage in it, that doesn't mean you can't defend your site against it. Everyone shits; it is not ipso facto hypocrisy not to want it on your lawn. Hypocrisy could be involved in the way you articulate your wish not to have it on your lawn.
chias 3y ago
I am, in general, strongly of the opinion that legal blocks to web scraping are ridiculous and a Bad Thing. That said, ever since I added some invisible unicode characters into my name on LinkedIn and can see the fraction of spam that I receive that addresses me with a bunch of `?`s where I put the invisible characters is rather astounding. Probably half the garbage email I get has clearly scraped my name from LinkedIn. All that to say: if you're someone who feels affronted that you aren't allowed to scrape LinkedIn, and you try to garner my sympathy about the situation, please accept my invitation to get stuffed.
- x0x0 3y ago
  I couldn't disagree more re: legal blocks are ridiculous, and this is a very pro-privacy decision. I shared info on LinkedIn for use by myself and other linkedin members, and for linkedin to use in their various products. You can't be pro-privacy and simultaneously believe that my putting info on LI has granted the entire world license to do with it what they will. Including running research programs or ML stuff or whatever hiq's business was, selling it to spammers, and using it in products by other companies that I've never heard of.
  - Genghis_Khan 3y ago
    > You can't be pro-privacy and simultaneously believe ... You gave information to LI, trusting that they wouldn't disclose it to people you wouldn't want to have it. Meanwhile, untrustworthy third parties were free to view that very same information. Blame LI for being casual with data about you. Blame yourself for trusting LI. Don't blame others for reading what they were free to read.
    - x0x0 3y ago
      That's simply wrong -- this went well beyond HiQ could read. HiQ's claim is that LI shouldn't be allowed to prevent HiQ from reading data that I had no intention or interest in sharing with HiQ, simply because LI makes it available to other LI users. It's wildly anti-privacy to claim that because I shared data with LI, and LI put some piece of it on the website, that HiQ has a right to read it and use it for arbitrary things.
1vuio0pswjnm7 3y ago
According to the District Court's October 27, 2022 Order, hiQ's litigation was being funded by a third party. Who was this "litigation funder".^1 Perhap the funder wanted to settle, even though hiQ could have prevailed at trial on the CFAA claims. Arguably, hiQ's goal was not to establish new CFAA precedent, it was to compete with LinkedIn. It wanted a Court to order LinkedIn to stop blocking its access to public data. This took too long. hiQ ran out of money. (Or its funding source(s) ran out of patience.) If hiQ had the funding to go the distance, then it's impossible to predict what would have happened. It's sad that the top comment in this thread believes LinkedIn sued hiQ to stop it from scraping. That's not what happened. hiQ's "business" was going to fail because LinkedIn was effectively blocking its access, even when hiQ was using proxies and mechanical turks. hiQ filed for a declaratory judgment against LinkedIn: hiQ sued LinkedIn. Here is the Order for anyone who cares to read it. https://ia600100.us.archive.org/29/items/gov.uscourts.cand.312704/gov.uscourts.cand.312704.404.0.pdf https://ia600100.us.archive.org/29/items/gov.uscourts.cand.3... 1. According to the Order, the funder is identified in some correspondence filed as an exhibit. It is not clear from the Order whether names in the correspondence were redacted, i.e., it's not clear whether it's public or private information. I'm assuming the former.
- deleted 3y ago
  [deleted]
- Klonoar 3y ago
  > it's interesting that the top comment in the thread believes LinkedIn sued hiQ to stop it from scraping. At time of writing, I believe that top comment is mine - and I'll be clear that I wasn't saying that I think that's why LinkedIn sued HiQ. In fact I don't particularly care why they did it, I'm just interested in what precedents the case has set and whether the typical comments thrown around regarding the case have any actual merit. There's a comment replying to mine that has a better breakdown of it all that I'd trust over my own anyway. ;P
  - 1vuio0pswjnm7 3y ago
    "In fact I don't particularly care why they did it, I'm just interested in what precedents the case has set and whether the typical comments thrown around regarding the case have any merit." What are the "typical comments thrown around". Perhaps could provide some references.
klabb3 3y ago
It seems like the reason the shit is hitting the fan at this particular point in time is because of the recent AI and LLM race. If I were to guess, it probably originated in an edgy, hyperbolic statement like “the new AI empires will be built on the free data from the old ones”, uttered by someone scary and important, perhaps Altman. Through the rumor mill all the feudal lords in the data-kingdom, like Steve Huffman, Musk etc are freaking out and are fortifying their defenses against this new perceived existential threat, sacrificing whatever minuscule openness remained in the process. To me, it’s obvious that a new copyright interpretation is needed for commercial AI applications, which would alleviate the gold rush panic. However, the regulatory bodies (ironically, eroded by the same corporations), don’t have any teeth left to do anything about it, so it’s a Wild West, and they all know it. The legal defense against scraping has nothing to do with principles, even less so these days, it’s all just about abusing the legal system for dominance games. Personally, I think the threat is overblown. Yes, AI will force eg Google to improve their shit ranking of recipe sites, but AI can’t provide up-to-date stuff people care about, so Google, Facebook, Twitter etc will keep providing services just like today.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- ReactiveJelly 3y ago
  I think it was already the case that a number of social media sites were cycling from the "Offer free bait" phase to the "Harvest the walled garden" phase. Twitter barely works on Tor. Reddit barely works if you're logged out. Reddit is pushing their app really hard and obviously going to drop old.reddit one day when their internal API finally breaks it off. If you make revenue from advertisements, having a web page that can be interpreted by any machine besides your own code is wasting money on freeloaders. Be they 3rd-party clients, Tor lurkers, or scrapers for search engines and AI.
  - noduerme 3y ago
jboydyhacker 3y ago
Fair use is how cos like Google got started in the first place. AI is similarly dependent on sucking up vast quantities of information, with a light touch of course, and use that to create the NEXT Google. For that matter, who will build the next LLM if anyone like Elon Musk can slap terms of use change on an account because he wants to stop fair use. This is anti innovation and means challenging all these monopolies- Google in search and twitter in Social media will be all the harder. This needs to get to SCOTUS fast for clarification as it's evil, wrong and will kill innovation.
nullifidian 3y ago
The Web Environment Integrity (WEI) API also plays into FAANGX's capability to monopolize scraping and therefore create competing AI systems.
WesolyKubeczek 3y ago
> Courts need to realize that if you allow private companies to invent intellectual property rights through online contracts of adhesion, courts will be at the mercy of private decision-makers on questions that should be questions of public interest. Courts feel very fine doing the bidding of private decision-makers on what should be questions of public interest, thank you very much. Reminds me of this joke: “Wake up, sir, wake up, you have shat yourself!” — “I’m not sleeping.”
xg15 3y ago
I found the linked Register.com vs Verio case [1] very interesting, because the courts actually made a more nuanced decision on contracts of adhesion than typically understood, I believe: In the case, Verio was calling Register's API for a purpose that Register disallowed. However, they only supplied the "contract" text that declared the restrictions after the call was made (as part of the API response I think). The court did in fact agree that this was too late: If you can only find out the terms and conditions for an API call by making that API call, then this is a "shrink-wrap agreement" and the terms are void. The restriction the court made on this was that it only applied to the first time they called the API: Verio has employees who can be expected to have common sense. So after having called the API for the first time, they had an opportunity to read the text and become aware of the restrictions. This meant that for all the subsequent API calls they made, Verio's staff was aware that they were doing something that Register explicitly forbid, but did so anyway - hence the court ruled that as "breach of contract". The important point here was that the courts never abandoned the principle that an individual must be aware of a contract's conditions before it can enter a contract - the case was just shooting down a situation where a party was pretending to be ignorant of the terms when they really weren't. [1] https://en.m.wikipedia.org/wiki/Register.com_v._Verio https://en.m.wikipedia.org/wiki/Register.com_v._Verio
1B05H1N 3y ago
Legal discussion aside, as a person working in security, it's pretty annoying to have to block certain web scrapers / automation while allowing a certain few.
FrustratedMonky 3y ago
Can I just Vent, That this stuff burns me up. If we can't be clear on Web Scraping, do we really have any hope for more clear rules on more complicated things like AI or privacy or copywrite?
two_in_one 3y ago
Just my opinion. Web scrapping isn't important for AI or AGI anymore. It's important only for the facts, like news. But not for the logic. With current datasets it should be possible to move forward, beyond GPT-4. Blocking access now is like investing in aluminum was, when it was more expensive than gold. And suddenly it's not. When we get facts separated from the model it will be enough to have just one article about, say, landing on the Mars for the model to be able to actively use this new knowledge. In this case 'AI providers' will need to cooperate with one news agency, buy right for only for a few classic books. Judging by how fast everything is moving this will happen within a couple of years if not a few months.
mcculley 3y ago
FTFA: “ All the world’s knowledge is available for the taking on the Internet” No. This is not true. Quit saying this. A small fraction of the world’s knowledge is on the Internet, much less scrapable.
offmychest 3y ago
With regards to the hiQ case (IANAL, but I followed this case closely): the 9th circuit asserted that the CFAA (Computer Fraud & Abuse Act), which is a _criminal_ statute, does not apply to scraping websites. On the other hand, if you create a user account and scrape while logged in, that is a matter of contract/civil law, and the website can sue you to stop breaching the terms of service (plus maybe damages). The takeaway for those of us in the trenches who feel we have as much a right as Google or Bing to crawl websites, is that we are not subject to arrest and imprisonment, just tort law. The ruling has apparently caused the major social networks to switch from criminalizing web scraping to putting most content behind a login and threatening to sue small companies out of existence.

582 points by mhb 3y ago

rvnx 3y ago

Sweet memories of Facebook, which was spamming all the contacts to invite them to join Facebook.

ilrwbwrkhv 3y ago
Same as LinkedIn. In fact that was LinkedIn's actual growth hack. Now they sell books talking about other things.
- rendall 3y ago
  I remember when LinkedIn had an anti-pattern of displaying a check list of your gmail contacts and asking which of them you want to connect with. These were all unchecked by default. But under the fold - that is, you needed to scroll down to see them - were all the rest of your contacts, only these were all checked. If you just went with what was displayed without scrolling down and unchecking the rest, you would unintentionally blast your entire contact list with contact requests. That was everyone in your contact list, whether appropriate or not: your dog groomer, people you were in a lawsuit with, whoever. I never did figure out how they were able to get our email contact lists. I vaguely remember reading about web-scraping shenanigans. I for one never knowingly gave them access.

SenAnder 3y ago

> Mark Lemley observed this happening nearly 20 years ago, in his prescient, seminal article, “Terms of Use.”: The problem is that the shift from property law to contract law takes the job of defining the Web site owner’s rights out of the hands of the law and into the hands of the site owner. With "contracts" of adhesion proliferating, and how impossible it has become to exist in the modern world without acceding to them (something as simple as buying a new SSD involves agreeing to one), this problem is getting worse by the day. The law is becoming increasingly irrelevant, and more and more we are ruled by one-sided "contracts" from giant companies that are in a position to push them on us.

nre 3y ago
> With "contracts" of adhesion proliferating, and how impossible it has become to exist in the modern world without acceding to them (something as simple as buying a new SSD involves agreeing to one), this problem is getting worse by the day. The craziest example of this is how all these contracts are appearing in the physical world as well. There are stores that actually have a sign indicating that entering the store constitutes acceptance of contract terms (with a QR code that you presumably can scan with your phone to read the contract). I've also seen public parks with the same thing basically indicating that entry binds you to a legal agreement to not sue the park/follow posted rules/etc.
- golemiprague 3y ago
  It is partly due to the crazy litigation in the US and the rewards people get for some coffee they spill on themselves just because the coffee was "too hot". There are two sides to this story and it is not only the business' owners who are the culprits here.
  - scolby33 3y ago
    Look up that case. The victim suffered third-degree burns and required skin grafts. The myth that it’s a ridiculous, over-litigious case is just that: a myth. The coffee was absolutely too hot.
    - genewitch 3y ago
      And it was caused by a profit-seeking behavior - an assumption that most, if not all people, would wait to drink the coffee until after they were out of the vehicle, which could be up to X minutes; as such they wouldn't want cold or lukewarm coffee. If most people equated mcdonalds coffee with "bad lukewarm coffee", less people would go to mcdonalds for anything, including coffee. I forget how hot they served the coffee, but it was up around 95C/200F. I know that you have to specifically request "extra hot" at places like SB and CB&TL, otherwise you get coffee hot enough to still be "hot coffee" after adding creamer, i venture 130F or so. AFAIK McDs doesn't offer "extra hot" coffee. Once bitten, and all. I don't know about the changes made due to this lawsuit, but i reckon they changed their cups to keep ~120F coffee at that temperature longer. edit: aside: I just realized why i prefer Fahrenheit even though i understand Centigrade. Stuff over 100F is "hotter than i am" It's intuitive in a way that merely remembering 37 is body temp (my dad was born in '37 so that's why i remember) and "anything over 40 is hot" doesn't really roll off the tongue, even though it's roughly as accurate as my ">100F is hot"

karaterobot 3y ago

The perceived hypocrisy sort of goes away when you stop thinking about it as a collaboration or a community of equals, and instead think of it as a competition, which is what it is. You would not say of a football team "oh, it's okay for you to try to score a goal on me, but if I try to score a goal on you, suddenly you're blocking the ball?!" Naturally, they're going to say "web scraping uses resources, stop it!" but then keep web scraping in the background. To be clear, it's bad behavior, it's just not hypocritical behavior, as it's completely in keeping with what amoral corporations locked in constant battle would be expected to do: maximize benefits to themselves while minimizing benefits to others.

Kareem71 3y ago
The problem is as this article points out is democratically elected courts should not be choosing winners in a capitalistic competition
- karaterobot 3y ago
  Only commenting on how we should expect corporations to act, or more accurately why we should not be surprised at their behavior.
backtoyoujim 3y ago
There are clearly more than two teams in these issues. It is not a game, and it is not football. It is a public policy issue also outpaces "competition" which is merely a subject change.
- karaterobot 3y ago
  Football is metaphorical in this case.
philipov 3y ago
Hypocrisy doesn't require one to believe what they say and utter it in good faith but fall short of those ideals in practice. Equivocating about football teams doesn't change that one is trying to impose standards on another without holding oneself equally to them. It is still hypocrisy, even if they do it amorally in bad faith. Especially then. What matters is what policy you espouse; you don't get a pass for not really believing what you say. The implication is that hypocrites are acting in bad faith.

fasterik 3y ago

>Let’s look at what Microsoft is doing right now, as an example. In the last couple of weeks, Microsoft updated its general terms of use to prohibit scraping, harvesting, or similar extraction methods of its AI services. Also in the couple of weeks, Microsoft affiliate OpenAI released a product called GPTbot, which is designed to scrape the entire internet. And while they don’t admit this publicly, OpenAI has almost certainly already scraped the entire non-authwalled-Internet and used it is training data for GPT-3, ChatGPT, and GPT-4. Nonetheless, without any obvious hints of irony, OpenAI’s own terms of use prohibits scraping. I don't understand why this demonstrates hypocrisy. There is a big difference between crawling the publicly accessible web (which legitimate search engines do all the time) and scraping an authenticated web application or API.

Atotalnoob 3y ago
Or that while Microsoft is an investor in openai, they do not control openai
- dvhh 3y ago
  If SDGs learned us anything is that companies are responsible for to scrutinize the company they are investing in. Of course this is not on the level of child labor, nor environmental pollution.
johneth 3y ago
The hypocrisy stems from: 1. OpenAI (etc.) scrape the public web to train and build their models. 2. They use these models to sell subscriptions (make money). None of this goes to the creators of the data used to train the models. 3. They deny others to do what they have done themselves. If you compare to, say, search engines scraping the public web: 1. Search engines scrape the public web to create their search indexes. 2. They use these indexes to provide search results and sell ads around them. Importantly, these search results direct people to the websites that they have scraped (much of the time), offering opportunity for them to make money.
- fasterik 3y ago

msie 3y ago

The first company that came to mind from reading the title was Google.

version_five 3y ago

Good example from the Allen Institute discussed last week https://news.ycombinator.com/item?id=37181415 https://news.ycombinator.com/item?id=37181415 They "released" a dataset scraped from public domain stuff under a license that restricts how people can use it

waydegg 3y ago

It's interesting seeing the reactions from other websites/orgs after OpenAI publicly announced GPTBot. Tons of people blocking GPTBot outright (made a small page that track this: https://wayde.gg/websites-blocking-openai https://wayde.gg/websites-blocking-openai)

version_five 3y ago
I wonder if blocking gptbot is a good signal that a website has non LLM generated content on it, and is therefore good training data...
- deleted 3y ago
  [deleted]
- Springtime 3y ago
  An illustrator specifically wrote[1] that this is why they won't be tagging their social media posts with #HumanMade, #NoAI and similar, as it's a signal there's unadulterated training data. [1] https://www.davidrevoy.com/article977/artificial-inteligence-why-i-ll-not-hashtag-my-art-humanart-humanmade-or-noai https://www.davidrevoy.com/article977/artificial-inteligence...
dontupvoteme 3y ago
Is there any legal issue with a spider trap designed to poison LLMs?

einpoklum 3y ago

Perhaps it's legal to scrape in some world states outside the USA?

dontupvoteme 3y ago
A good question. Japan came out and declared that copyright does not apply to training AI. There must be a good chunk of the world that doesn't have any laws forbidding it. This isn't under the jurisdiction of WIPO or anything like that, it's just a completely insane evolution of anglo common law

deepsun 3y ago

> But the content that they’re trying to protect isn’t theirs -- it belongs to their users. Kinda. Yes, Facebook says that content belongs to users (otherwise they'd have harder time explaining they are not liable when it's illegal), but users also agree to give Facebook “non-exclusive, transferable, sub-licensable, royalty-free, worldwide license to use any IP content that you post on or in connection with Facebook.” For example, if a user deleted their* content, Facebook can still use it and show to their friends. That's why it's "kinda".

waynesonfire 3y ago
Did you read the posted sign? “no walking on the road outside my property”
sib 3y ago
That doesn't change who the content belongs to. It just gives some rights to FB. Any, in fact, without something like "perpetual" and/or "irrevocable" in there, it doesn't imply that they could keep using it after you deleted it (or that you couldn't revoke a grant of rights.)
antonf 3y ago
> For example, if a user deleted their* content, Facebook can still use it and show to their friends. That's why it's "kinda". I don't think it is correct. If you asked Facebook to remove your data from platform, it will be a GDPR (and probably CCPA, etc...) violation for Facebook to not delete your data within 1 month.
jeremyjh 3y ago
A license is not ownership. Anyway that part of the article is just context - none of what you describe constitutes the legal basis for the suits or rulings discussed in it - it’s just explaining why property law isn’t being used.

Klonoar 3y ago

Hmmm, I'm a bit confused on something. The HiQ vs LinkedIn case, to my knowledge, went through the following stages: - LinkedIn sues HiQ, Ninth Circuit sides with HiQ - LinkedIn pushes to Supreme Court, Supreme Court vacates citing Van Buren - Ninth Circuit re-reviews and affirms their decision - LinkedIn moves to get the injunction preventing them from blocking HiQ dissolved, which is granted - A mixed judgement is finally issued in Nov 2022 ultimately resulting in a private settlement Where exactly does this leave things at? I feel like everyone loves to cite this case but never goes into the finer details. Reading a summary of the mixed judgement from Nov 2022, it looks like maybe the issue came from HiQ using people to log in and thus the ToS came into play...? If I'm reading correctly, it looks like the court did eventually side with LinkedIn in stating that HiQ violated the ToS. https://www.natlawreview.com/article/court-finds-hiq-breached-linkedin-s-terms-prohibiting-scraping-mixed-ruling-declines https://www.natlawreview.com/article/court-finds-hiq-breache... Edit: Formatting.

dontupvoteme 3y ago
What is the legal precedent of a mixed ruling? I was unaware such a thing was even possible.
KieranMac 3y ago
Not a mixed judgment in Nov. 22. It was a massive defeat for hiQ Labs. Read the permanent injunction issued by the court.
- dontupvoteme 3y ago
  >Read the permanent injunction issued by the court. Happen to have a link? The question that matters is if this establishes any precedence.
  - KieranMac 3y ago
    I don't. I just have a .pdf. Email me at Kieran(at)McCarthyLG(dot)com if you want a copy.
  - 3y ago

dclowd9901 3y ago

The primary grounds on which these cases rest is some nebulous understanding of contractual agreement. I have two thoughts: - EULAs aren’t written for companies to sign. - I think EULAs are garbage anyway. They’re completely one sided and in most cases probably illegal or wouldn’t hold up in court if anyone actually had the resources to fight one. Imo, the burden of ensuring someone has read and understands a EULA should be on the company who creates it and they should not be enforceable unless they can prove the person understood the EULA entirely before accessing the site. EULAs are not a business agreement. They’re some kind of corporate pseudo-law companies try to attach to the usage of a product. But what other product in the world has a big list of rules that come with it that way how you can use it (or be sued)? So how does this all come back to this “company vs company scraping”? If you put it on the web, and you don’t have REAL copyright on the content (that is, you didn’t make it yourself), you have no right to protect it from “theft.” PS yes, I know John Deere doesn’t let its customers work on its tractors but that’s some bullshit too.

KieranMac 3y ago
These online agreements are often enforceable, even when companies have lots of resources to defend themselves.

deleted 3y ago

[deleted]

sneak 3y ago

If you think about it, if free lending libraries and web search indices did not exist, and you tried to create them today, you would get sued into oblivion.

zarazas 3y ago

How is the situatuon legally and ethically for you to use scraped data as text embeddings for a commercial product?

MBCook 3y ago

I see two issues. Web scraping is clearly a business model problem, and that’s partly due to scale. If you give away your content for free and expect ads to sustain you, that will start failing once others get the value out of your content without seeing the ads. Examples are ad blockers, answers embedded in Google results, Stack Overflow clones, and things like ChatGPT. If ads weren’t your business model you wouldn’t be using revenue from it. The other issue is scale, and I don’t know how to address it. It’s easy for someone (say the government) to have a friendly policy and say “you can use dig in a park” thinking it’s useful to campers and such. But when someone shows up with a professional strip mining crew, things are different. If you run a site providing quality information for free, making money off book sales or professional services or such can be a good living. Even if answers end up in the Google answer box, more complicated stuff or analysis still requires a visit to read and people can start following you from there. But if ChatGPT or whatever can “read” your stuff and give out 80% of the value without anyone even knowing it came from you, you’re screwed. Your business model no longer works. Any kind of “give away good information” business model fails. Same issue artists are now seeing. And I don’t know how you fix that without some kind of ban. But unless every country everywhere enforces one… you have to work with the lowest common denominator and lock all your content up. No web search. No Google answers. No chat GPT. “Please don’t scrape me” in robots.txt won’t work.

drunkencoder 3y ago
You’re right . That’s why scraping must be unlimited and legal for all. Any information accessible from internet should be legal to refine. Thus also us using GPT services to train our own models, scraping anything that’s publicly accessible. Our only defense is competing services that refines the data even more than any general llm. The solution is almost never regulation but competition. Fair competition
- antonf 3y ago
  > You’re right . That’s why scraping must be unlimited and legal for all. Unlimited scraping makes some of privacy regulations moot. Such as right to erasure (ability to delete personal data from a platform).

kazinator 3y ago

> Some of the biggest companies on earth—including Meta and Microsoft—take aggressive, litigious approaches to prohibiting web scraping on their own properties, while taking liberal approaches to scraping data on other companies’ properties. Umm, no; author needs to study the word "hypocrisy" more deeply than a cursory glance in the dictionary. Doing something to others, while defending against the same thing, is not hypocrisy. For instance, soccer player isn't a hypocrite for defending against the ball going into his net, while trying to put it into the other team's net. A soldier on the war front isn't a hypocrite for shooting, while also taking cover and dodging bullets. These subjects are not hypocrites because they are not acting in one way, while preaching that they, or others, ought to be acting in a different way. Microsoft would be hypocrites if they published an official statement such that nobody who engages in web scraping has the right to defend their own site against web scraping, because that would not resemble their actual behavior and position which could be inferred from their behavior. (Is there such a statement somewhere?) For hypocrisy to take place, you have to actually preach that you and others should behave in a certain way, and then not actually behave in exactly that way. If you only act, and don't preach, you cannot be a hypocrite. Moreover, your team's net is not the same object as another team's net. If a soccer player loudly professes "it is morally wrong for anyone to kick the ball into our net", but then kicks the ball into the other team's net, that is not hypocrisy. His statement references only his own net; he didn't proclaim that it's wrong to kick any ball into any net whatsoever.

KieranMac 3y ago
Scraping other sites while prohibiting it on your own is "do what I say, not what I do" behavior, which I think is a fair, consensus understanding of what it means to be hypocritical.
- kazinator 3y ago
  To convict Microsoft or Meta of hypocrisy, you need to find some official statement from the company which says that web scraping is wrong. Hypocrisy is the professing of a moral position to which one does not conform. If you don't preach that web scraping is wrong, and engage in it, that doesn't mean you can't defend your site against it. Everyone shits; it is not ipso facto hypocrisy not to want it on your lawn. Hypocrisy could be involved in the way you articulate your wish not to have it on your lawn.

chias 3y ago

I am, in general, strongly of the opinion that legal blocks to web scraping are ridiculous and a Bad Thing. That said, ever since I added some invisible unicode characters into my name on LinkedIn and can see the fraction of spam that I receive that addresses me with a bunch of `?`s where I put the invisible characters is rather astounding. Probably half the garbage email I get has clearly scraped my name from LinkedIn. All that to say: if you're someone who feels affronted that you aren't allowed to scrape LinkedIn, and you try to garner my sympathy about the situation, please accept my invitation to get stuffed.

x0x0 3y ago
I couldn't disagree more re: legal blocks are ridiculous, and this is a very pro-privacy decision. I shared info on LinkedIn for use by myself and other linkedin members, and for linkedin to use in their various products. You can't be pro-privacy and simultaneously believe that my putting info on LI has granted the entire world license to do with it what they will. Including running research programs or ML stuff or whatever hiq's business was, selling it to spammers, and using it in products by other companies that I've never heard of.
- Genghis_Khan 3y ago
  > You can't be pro-privacy and simultaneously believe ... You gave information to LI, trusting that they wouldn't disclose it to people you wouldn't want to have it. Meanwhile, untrustworthy third parties were free to view that very same information. Blame LI for being casual with data about you. Blame yourself for trusting LI. Don't blame others for reading what they were free to read.
  - x0x0 3y ago
    That's simply wrong -- this went well beyond HiQ could read. HiQ's claim is that LI shouldn't be allowed to prevent HiQ from reading data that I had no intention or interest in sharing with HiQ, simply because LI makes it available to other LI users. It's wildly anti-privacy to claim that because I shared data with LI, and LI put some piece of it on the website, that HiQ has a right to read it and use it for arbitrary things.

1vuio0pswjnm7 3y ago

According to the District Court's October 27, 2022 Order, hiQ's litigation was being funded by a third party. Who was this "litigation funder".^1 Perhap the funder wanted to settle, even though hiQ could have prevailed at trial on the CFAA claims. Arguably, hiQ's goal was not to establish new CFAA precedent, it was to compete with LinkedIn. It wanted a Court to order LinkedIn to stop blocking its access to public data. This took too long. hiQ ran out of money. (Or its funding source(s) ran out of patience.) If hiQ had the funding to go the distance, then it's impossible to predict what would have happened. It's sad that the top comment in this thread believes LinkedIn sued hiQ to stop it from scraping. That's not what happened. hiQ's "business" was going to fail because LinkedIn was effectively blocking its access, even when hiQ was using proxies and mechanical turks. hiQ filed for a declaratory judgment against LinkedIn: hiQ sued LinkedIn. Here is the Order for anyone who cares to read it. https://ia600100.us.archive.org/29/items/gov.uscourts.cand.312704/gov.uscourts.cand.312704.404.0.pdf https://ia600100.us.archive.org/29/items/gov.uscourts.cand.3... 1. According to the Order, the funder is identified in some correspondence filed as an exhibit. It is not clear from the Order whether names in the correspondence were redacted, i.e., it's not clear whether it's public or private information. I'm assuming the former.

deleted 3y ago
[deleted]
Klonoar 3y ago
> it's interesting that the top comment in the thread believes LinkedIn sued hiQ to stop it from scraping. At time of writing, I believe that top comment is mine - and I'll be clear that I wasn't saying that I think that's why LinkedIn sued HiQ. In fact I don't particularly care why they did it, I'm just interested in what precedents the case has set and whether the typical comments thrown around regarding the case have any actual merit. There's a comment replying to mine that has a better breakdown of it all that I'd trust over my own anyway. ;P
- 1vuio0pswjnm7 3y ago
  "In fact I don't particularly care why they did it, I'm just interested in what precedents the case has set and whether the typical comments thrown around regarding the case have any merit." What are the "typical comments thrown around". Perhaps could provide some references.

klabb3 3y ago

It seems like the reason the shit is hitting the fan at this particular point in time is because of the recent AI and LLM race. If I were to guess, it probably originated in an edgy, hyperbolic statement like “the new AI empires will be built on the free data from the old ones”, uttered by someone scary and important, perhaps Altman. Through the rumor mill all the feudal lords in the data-kingdom, like Steve Huffman, Musk etc are freaking out and are fortifying their defenses against this new perceived existential threat, sacrificing whatever minuscule openness remained in the process. To me, it’s obvious that a new copyright interpretation is needed for commercial AI applications, which would alleviate the gold rush panic. However, the regulatory bodies (ironically, eroded by the same corporations), don’t have any teeth left to do anything about it, so it’s a Wild West, and they all know it. The legal defense against scraping has nothing to do with principles, even less so these days, it’s all just about abusing the legal system for dominance games. Personally, I think the threat is overblown. Yes, AI will force eg Google to improve their shit ranking of recipe sites, but AI can’t provide up-to-date stuff people care about, so Google, Facebook, Twitter etc will keep providing services just like today.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
ReactiveJelly 3y ago
I think it was already the case that a number of social media sites were cycling from the "Offer free bait" phase to the "Harvest the walled garden" phase. Twitter barely works on Tor. Reddit barely works if you're logged out. Reddit is pushing their app really hard and obviously going to drop old.reddit one day when their internal API finally breaks it off. If you make revenue from advertisements, having a web page that can be interpreted by any machine besides your own code is wasting money on freeloaders. Be they 3rd-party clients, Tor lurkers, or scrapers for search engines and AI.
- noduerme 3y ago

jboydyhacker 3y ago

Fair use is how cos like Google got started in the first place. AI is similarly dependent on sucking up vast quantities of information, with a light touch of course, and use that to create the NEXT Google. For that matter, who will build the next LLM if anyone like Elon Musk can slap terms of use change on an account because he wants to stop fair use. This is anti innovation and means challenging all these monopolies- Google in search and twitter in Social media will be all the harder. This needs to get to SCOTUS fast for clarification as it's evil, wrong and will kill innovation.

nullifidian 3y ago

The Web Environment Integrity (WEI) API also plays into FAANGX's capability to monopolize scraping and therefore create competing AI systems.

WesolyKubeczek 3y ago

> Courts need to realize that if you allow private companies to invent intellectual property rights through online contracts of adhesion, courts will be at the mercy of private decision-makers on questions that should be questions of public interest. Courts feel very fine doing the bidding of private decision-makers on what should be questions of public interest, thank you very much. Reminds me of this joke: “Wake up, sir, wake up, you have shat yourself!” — “I’m not sleeping.”

xg15 3y ago

I found the linked Register.com vs Verio case [1] very interesting, because the courts actually made a more nuanced decision on contracts of adhesion than typically understood, I believe: In the case, Verio was calling Register's API for a purpose that Register disallowed. However, they only supplied the "contract" text that declared the restrictions after the call was made (as part of the API response I think). The court did in fact agree that this was too late: If you can only find out the terms and conditions for an API call by making that API call, then this is a "shrink-wrap agreement" and the terms are void. The restriction the court made on this was that it only applied to the first time they called the API: Verio has employees who can be expected to have common sense. So after having called the API for the first time, they had an opportunity to read the text and become aware of the restrictions. This meant that for all the subsequent API calls they made, Verio's staff was aware that they were doing something that Register explicitly forbid, but did so anyway - hence the court ruled that as "breach of contract". The important point here was that the courts never abandoned the principle that an individual must be aware of a contract's conditions before it can enter a contract - the case was just shooting down a situation where a party was pretending to be ignorant of the terms when they really weren't. [1] https://en.m.wikipedia.org/wiki/Register.com_v._Verio https://en.m.wikipedia.org/wiki/Register.com_v._Verio

1B05H1N 3y ago

Legal discussion aside, as a person working in security, it's pretty annoying to have to block certain web scrapers / automation while allowing a certain few.

FrustratedMonky 3y ago

Can I just Vent, That this stuff burns me up. If we can't be clear on Web Scraping, do we really have any hope for more clear rules on more complicated things like AI or privacy or copywrite?

two_in_one 3y ago

Just my opinion. Web scrapping isn't important for AI or AGI anymore. It's important only for the facts, like news. But not for the logic. With current datasets it should be possible to move forward, beyond GPT-4. Blocking access now is like investing in aluminum was, when it was more expensive than gold. And suddenly it's not. When we get facts separated from the model it will be enough to have just one article about, say, landing on the Mars for the model to be able to actively use this new knowledge. In this case 'AI providers' will need to cooperate with one news agency, buy right for only for a few classic books. Judging by how fast everything is moving this will happen within a couple of years if not a few months.

mcculley 3y ago

FTFA: “ All the world’s knowledge is available for the taking on the Internet” No. This is not true. Quit saying this. A small fraction of the world’s knowledge is on the Internet, much less scrapable.

offmychest 3y ago

With regards to the hiQ case (IANAL, but I followed this case closely): the 9th circuit asserted that the CFAA (Computer Fraud & Abuse Act), which is a _criminal_ statute, does not apply to scraping websites. On the other hand, if you create a user account and scrape while logged in, that is a matter of contract/civil law, and the website can sue you to stop breaching the terms of service (plus maybe damages). The takeaway for those of us in the trenches who feel we have as much a right as Google or Bing to crawl websites, is that we are not subject to arrest and imprisonment, just tort law. The ruling has apparently caused the major social networks to switch from criminalizing web scraping to putting most content behind a login and threatening to sue small companies out of existence.