18 ms·

Web scraping is legal, US appeals court reaffirms

1040 points by spenvo 5y ago

altdataseller 5y ago
Does the ruling make it illegal to block scrapers?
- postalrat 5y ago
  Doubt it. And it should never be illegal to attempt to block scrapers. Many sites do try.
ricardo81 5y ago
Interesting as I've seen a few search engine start ups that seem to scrape other search engine results, depending on your definition of scraping. My definition would be a user agent that doesn't uniquely identify itself that isn't using an authorised API.
CWuestefeld 5y ago
While I have sympathy for what the scrapers are trying to do in many cases, it bothers me that this doesn't seem to address what happens when badly-behaved scrapers cause, in effect, a DOS on the site. For the family of sites I'm responsible for, bot traffic comprises a majority of traffic - that is, to a first approximation, the lion's share of our operational costs are from needing to scale to handle the huge amount of bot traffic. Even when it's not as big as a DOS, it doesn't seem right to me that I can't tell people they're not welcome to cause this additional system load. Or even if there was some standardized way that we could provide a dumb API, just giving them raw data so we don't need to incur the additional expense of the processing for creature comforts on the page designed to make our users happier but the bots won't notice.
- svnpenn 5y ago
  Implement TLS fingerprinting on your server. People can still fake that if they are determined, but it should cut the abuse way down.
  - CWuestefeld 5y ago
    No, nor can we just do it by IP. The bots are MUCH more sophisticated than that. More often than not, it's a cooperating distributed net of hundreds of bots, coming from multiple AWS, Azure, and GCP addresses. So they can pop up anywhere, and that IP could wind up being a real customer next week. And they're only recognizable as a botnet with sophisticated logic looking at the gestalt of web logs. We do use a 3rd party service to help with this - but that on its own is imposing a 5- to 6-digit annual expense on our business.
    - z3c0 5y ago
      Have you considered setting up an API to allow the bots to get what they want without hammering your front-end servers?
      - kayodelycaon 5y ago
        What makes you think they would use it?
        z3c0 5y ago
        It's in their interest. I've scraped a lot, and it's not easy to build a reliable process on. Why parse a human interface when there's an application interface available?
      - CWuestefeld 5y ago
        Yes. And if I could get the perpetrators to raise their hands so I could work out an API for them, it would be the path of least resistance. But they take great pains to be anonymous, although I know from circumstantial evidence that at least a good chunk of it is various competitors (or services acting on behalf of competitors) scraping price data. IANAL, but I also wonder if, given that I'd be designing something specifically for competitors to query our prices in order to adjust their own prices, this would constitute some form of illegal collusion.
        5y ago
- borski 5y ago
  You could ban their IPs?
  - KMnO4 5y ago
    IP bans are equivalent to residential door locks. They’re only deterring the most trivial attacks. In school I needed to scrape a few hundred thousand pages of a proteomics database website. For some reason you had to view each entry one at a time. There was IP throttling which banned you if you made requests too quickly. But slowing the script to 1 request per second would have taken days to scrape the site. So I paid <$5 for a list of 500 proxy servers and distributed it, completing the task in under half an hour.
    - l33t2328 5y ago
      Using proxies to hide your identity to get around a denial of access seems to get awfully close to violating the Computer Fraud and Abuse Act(in USA, at least). I’m surprised your school was okay with it.
      - KMnO4 5y ago
        Don't worry, I don't live in the USA. Thanks for your concern though.
    - borski 5y ago
      I agree it’s not perfect. It’s also significantly better than nothing.
      - Gigachad 5y ago
        It's also completely insufficient despite being better. There are so many scraper services that its just a matter of paying a small amount of money to make use of tens of thousands of IPs spread across ISPs and countries.
    - inshadows 5y ago
      Can you share where did you get such a nice proxy for 500 proxies? TIA
- forgotmypw17 5y ago
  Yes, I think working to accommodate the non-humans along with the humans is the right approach here. Scrapers have a limited range of IPs, so rate-limiting them and stalling (or dropping) request responses is one way to deal with the DoS scenario. For my sites, I have placed the majority behind HTTP Basic Auth...
  - KptMarchewa 5y ago
    You realistically can't. There are services like [0][1] that mean any IP could be a scraper node. [0] https://brightdata.com/proxy-types/residential-proxies https://brightdata.com/proxy-types/residential-proxies [1] https://oxylabs.io/products/residential-proxy-pool https://oxylabs.io/products/residential-proxy-pool
    - orlp 5y ago
      > How does Bright Data acquire its residential IPs? > Bright Data has built a unique consumer IP model by which all involved parties are fairly compensated for their voluntary participation. App owners install a unique Software Development Kit (SDK) to their applications and receive monthly remuneration based on the number of users who opt-in. App users can voluntarily opt-in and are compensated through an ad-free user experience or enjoy an upgraded version of the app they are using for free. These consumers or ‘peers’ serve as the basis of our network and can opt-out at any time. This model has brought into existence an unrivaled, first of its kind, ethically sound, and compliant network of real consumers. I don't know how they can say with a straight face that this is 'ethically sound'. They have, essentially, created a botnet, but apparently because it's "AdTech" and the user "opts-in" (read: they click on random buttons until they hit one that makes the banner/ad go away) it's suddenly not malware.
      - TedDoesntTalk 5y ago
        NordVPN (Tesonet) has another business doing the same thing. They sell the IP addresses/bandwidth of their NordVPN customers to anyone who needs bulk mobile or residential IP addresses. That's right, installing their VPN software adds your IP address to a pool that NordVPN then resells. Xfinity/Comcast sort of pioneered this with their wifi routers that automatically expose an isolated wifi network called 'xfinity' (IIRC) whether you agree or not.
- Nextgrid 5y ago
  But don't you already have countermeasures to deter DoS attacks or malicious human users (what if someone pays or convinces people to open your site and press F5 repeatedly)? If not, you should, and the badly-behaved scrapers are actually a good wake-up call.
- rmbyrro 5y ago
  I have sympathy for your operational issues and costs, but isn't this kind of complaint the same as a shopping mall/center complaining of people who go in, check some info and go out without buying? I understand that bots have leverage and automation, but so does you to reach a larger audience. Should we continue to benefit from one side of the leverage, while complaining about the other side?
  - wlesieutre 5y ago
    It's more like a mall complaining that while they're trying to serve 1000 customers, someone has gone and dumped 10000000 roombas throughout the stores which are going around scanning all the price tags.
  - CWuestefeld 5y ago
    No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side. My complaint is more like, somebody wants to know the prices of all our products, and that we have roughly X products (where X is a very large number). They get X friends to all go into the store almost simultaneously, each writing down the price of the particular product they've been assigned to research. When they do this, there's scant space left in the store for even the browsing kind of customers to walk in. (of course I exaggerate a bit, but that's the idea)
    - mrobins 5y ago
      I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.
      - jensensbutton 5y ago
        You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.
- linuxdude314 5y ago
  If most of your traffic is bots, is the site even worth running? This really is akin to the question, “Should others be allowed to take my photo or try to talk to me in public?” Of course the answer should be yes, the internet is the digital equivalent of a public space. If you make it accessible, anyone should be able to consume. If you don’t want it scraped add auth!
  - simiones 5y ago
    Do you also believe someone running a drone to follow and photograph you, personally, wherever you go in public would be fair and legal?
- kstrauser 5y ago
  I've told this story before, but it was fun, so I'm sharing it again: I'll skip the details, but a previous employer dealt with a large, then-new .mil website. Our customers would log into the site to check on the status of their invoices, and each page load would take approximately 1 minute. Seriously. It took about 10 minutes to log in and get to the list of invoices available to be checked, then another minute to look at one of them, then another minute to get out of it and back into the list, and so on. My job was to write a scraper for that website. It ran all night to fetch data into our DB, and then our website could show the same information to our customers in a matter of milliseconds (or all at once if they wanted one big aggregate report). Our customers loved this. The .mil website's developer hated it, and blamed all sorts of their tech problems on us, although: - While optimizing, I figured out how to skip lots of intermediate page loads and go directly to the invoices we wanted to see. - We ran our scraper at night so that it wouldn't interfere with their site during the day. - Because each of our customers had to check each one of their invoices every day if they wanted to get paid, and we were doing it more efficiently, our total load on their site was lower than the total load of our customers would be. Their site kept crashing, and we were there scapegoat. It was great fun when they blamed us in a public meeting, and we responded that we'd actually disabled our crawler for the past week, so the problem was still on their end. Eventually, they threatened to cut off all our access to the site. We helpfully pointed out that their brand new site wasn't ADA compliant, and we had vision-impaired customers who weren't able to use it. We offered to allow our customers to run the same reports from our website, for free, at no cost to the .mil agency, so that they wouldn't have to rebuild their website from the ground up. They saw it our way and begrudgingly allowed us to keep scraping.
  - dwater 5y ago
    I have worked with .mil customers who paid us to scrape and index their website because they didn't have a better way to access their official, public documents.
    - jll29 5y ago
      This is not .mil specific: I've been told of a case where an airline first legally attacked a flight search engine (Skyscanner) for scraping, and then told them to continue when they realized that their own search engine couldn't handle all the traffic, and even if it could, it was more expensive per query than routing via Skyscanner.
- bequanna 5y ago
  As someone that has been on the other end, I can tell you devs don’t want to use selenium or inspect requests to reverse engineer your UI and wish there were more clean APIs. Have you tried making your UI more challenging to scrape and adding a simple API that requires free registration?
  - JimBlackwood 5y ago
    So much this! I work in E-Commerce and (needless to say) we scrape a lot of websites. Due to our growth and the increase in scrapers we require, I’ve been writing a proposal to a higher up to talk to our biggest competitors to all set up a public API that batches the data for a smaller amount of requests. It would save everyone quite some traffic and effort.
- userbinator 5y ago
  That's what rate-limiting is for. Don't be so aggressive with it that you start hitting the faster visitors, however, or they may soon go somewhere else (has happened to me a few times).
  - loceng 5y ago
    Do you know if there's a way to rate limit logged-in users differently than visitors of a site?
    - minusf 5y ago
      nginx can be set up to do that using the session cookie.
    - rolph 5y ago
      rate limiting can be a double edged sword, you can be better off giving a scraper highest bandwidth so they are gone sooner, otherwise somthing like making a zip or other sort of compilation of the site available may be an option. just what kind of scraper you have is a concern. does scraper just want a bunch of stock images; or does scraper have FOMO on web trinkets; or does scraper want to mirror/impersonate your site. the last option is the most concerning because then; scraper is mirroring bcz your site is cool and local UI/UX is wanted; or is scraper phishing smishing or otherwise duping your users.
      - loceng 5y ago
        Yeah, good points to consider. I think the sites that would be scrapped the most would be where the data is regularly and reliably up-to-date, and a large volume of it at that - so not just one scraper but many different parties may on a daily or weekly basis try to scrap every page. I feel that ruling should have the caveat that if a fair cost paid API version for getting publicly listed data then the scrapers must legally use that (say no more than 5% more than cost of CPU/bandwidth/etc of the scraping behaviour); ideally a rule too that at minimum there be a delay if they are republishing that data without your permission, so at least you as the platform/source/reason for the data being up-to-date aren't harmed too - which may then kill the source platform over time if regular visitors somehow start going to the competitor publishing the data.
- ryan_j_naughton 5y ago
  As other have said (A) there are plenty of countermeasures you can take, but also (B) you are frustrated that you are providing something free to the public and then annoyed at the "wrong" customers are using your product and costing you money. I'm sorry, but this is a failure of your business model. If we were to analogize this to a non-internet example: (1) A company throws a free concert/event and believes they will make money by alcohol sales. (2) A bunch of sober/non-drinking folks attend the concert but only drink water (3) Company blames the concert attendees for "taking advantage" of them when they really just had poor company policies and a bad business model. Put things behind authentication and authorization. Add a paywall. Implement DDOS and detection and banning approaches for scrapers. Etc etc. But don't make something public and then get mad at THE PUBLIC for using it. Behind that machine is a person, who happens to be a member of the public.
  - noisenotsignal 5y ago
    There are certain classes of websites where the proposed solutions aren’t a great fit. For example, a shopping site hiding their catalog behind paywalls or authentication would raise barriers to entry such that a lot of genuine customers would be lost. I don’t think the business model is in general to be blamed here and it’s ok to acknowledge the unfortunate overhead and costs added by site usage patterns (e.g. scraping) that are counter to the expectation.
  - inlined 5y ago
    Alternatively it could be seen that your juice company offers free samples. Then somebody abuses free and takes gallons home with them to bottle and sell as their own. That’s what it feels like when someone is scraping your network to bootstrap a competitor.
    - ryan_j_naughton 5y ago
      Again, what you call abuse of free samples, someone else calls a savvy strategy tailored to you're poorly crafted business plan. Have ways to limit the free samples or else it's your fault...
- deleted 5y ago
  [deleted]
- gdulli 5y ago
  When the original ruling in favor of HiQ came out, it still allowed for LinkedIn to block certain kinds of malicious scraping. LinkedIn had been specifically blocking HiQ, and was ordered to stop doing that.
- brailsafe 5y ago
  You probably can. On the protocol level with JSON-LD or other rich data packages that generate xml or standardized json endpoints. I did this for an open data portal, and this is something most G7 governments do with their federal open data portals using off the shelf packages (that are worth researching a bit obviously first), particularly in the python and flask world. We were still getting hammered by China at our Taiwanese language subdomain, but that was a different concern
- YPCrumble 5y ago
  Reading your comment my impression is that this is either an exaggeration or a very unique type of site if bots make up the majority of traffic to the point that scrapers are anywhere near the primary load factor. Would someone let me know if I’m just plain wrong in this assumption? I’ve run many types of sites and scrapers have never been anywhere close to the main source of traffic or even particularly noticeable compared to regular users. Even considering a very commonly scraped site like LinkedIn or Craigslist - for any site of any magnitude like this public pages are going to be cached so additional scrapers are going to have negligible impact. And a rate limit is probably one line of config. I’m not saying you are necessarily wrong, but I can’t imagine a scenario that you’re describing and would love to hear of one.
  - nojito 5y ago
    Bandwidth isn't free.
    - YPCrumble 5y ago
      I didn’t say it is free, I said that the bandwidth for bots is negligible compared to that of regular users.
      - nojito 5y ago
        Negligible isn’t free either.
  - VWWHFSfQ 5y ago
    > a very unique type of site if bots make up the majority of traffic Pretty much Twitter and the majority of such websites.
    - YPCrumble 5y ago
      Do you really believe bots make up a significant amount of Twitter’s operating cost? Like I said they’re just accessing cached tweets and are rate limited. How can the bot usage possibly be more than a small part of twitter’s operating cost?
- throw10920 5y ago
  Have you considered serving a proof-of-work challenge to clients accessing your website? Minimal cost on legit users, but large costs on large-scale web-scraping operations, and it doesn't matter if they split up their efforts across a bunch of IP addresses - they're still going to have to do those computations. https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash
  - nyuszika7h 5y ago
    No thanks, as a user I would stay far away from such websites. This is akin to crypto miners. I don't need them to drive up my electricity costs and also contribute to global warming in the process. It's not worth the cost.
    - fouric 5y ago
      This is completely absurd - anti-spam PoW is not remotely comparable to crypto miners, and the electricity cost will be so far below the noise floor of owning a computer in the first place that you will literally not notice (and neither will the environment), unless website owners are completely insane and set up multi-second challenges (which, they won't). And, it's absolutely worth the cost - as a website owner, you get to impose costs on botting operations with minimal penalties for normal users and minimal environmental impact. Bots work because the costs of renting an AWS server and scraping websites (or sending spam, whatever) are extremely tiny - adding PoW challenges to everything that could be spammed suddenly massively changes the cost of running those spam operations, and would result in noticeably less spam if deployed widely. In fact, the net "environmental impact" would be negative, as botters start to shut down operations due to greatly increased operational costs.
    - danamit 5y ago
      You do PoW every time you send an email.
- colinmhayes 5y ago
  I'm sympathetic to this. I built a search engine for my senior project and my half baked scraper ended up taking down duke law's site during their registration period. Ended up getting a not so kindly worded email from them, but honestly this wasn't an especially hard problem to solve. All of my traffic was coming from the cluster that was on my university's subnet, it wouldn't have been that hard to for them to IP address timeouts when my crawler started scraping thousands of pages a second on their site. Not to victim blame, this was totally my fault, but I was a bit surprised that they hadn't experienced this before with how much automated scraping goes on.
  - brightball 5y ago
    I’m honestly more interested in bot detection than anything else at this point. It seems like it should be perfectly legal to detect and then hold the connection open for a long period of time without giving a useful response. Or even send highly compressed gzip responses designed to fill their drives. Legal or not, I can’t see any good reason that we can’t make it painful.
    - fjabre 5y ago
      Make it painful if they abuse the site. We all benefit from open data. Polite scrapers are just fine and a natural part of the web ecosystem. Google has been scraping the web all day every day for decades now.
- deleted 5y ago
  [deleted]
- hardtke 5y ago
  The problem with many sites (and LinkedIn in particular) is that they whitelist a bunch of specific websites, presumably based on the business interests, but disallow everyone else in their robots.txt. You should either allow all scrapers that respect certain load requirements or allow none. Anything that Google is allowed to see and include in their search results should be fair game. Here's the end of LinkedIn's robots.txt: User-agent: * Disallow: / # Notice: If you would like to crawl LinkedIn, # please email whitelist-crawl@linkedin.com to apply # for white listing.
  - diamondage 5y ago
    And this is what the HiQ case hinged on. LinkedIn were essentially selectively applying the computer fraud and abuse act based on their business interests - that was never going to sit well with judges.
  - JimBlackwood 5y ago
    Btw, LinkedIn does have an API for things like Sales Navigator. It used some weird partnership program (SNAP) to get into it and it starts at (I think) 1500$/year per user. Still pretty cheap though, I think you’d get the value out of that quite quickly for a >300 people company.
- car_analogy 5y ago
  > Even when it's not as big as a DOS, it doesn't seem right to me that I can't tell people they're not welcome to cause this additional system load. You can tell them. You just can't prosecute them if they don't obey.
- voxic11 5y ago
  The court just ruled that scraping on its own isn't a violation of the CFAA. Meaning it doesn't count as the crime of "accessing a protected computer without authorization or exceeding authorized access and obtaining information". However presumably all the other provisions of the CFAA still apply, so if your scraping damages the functioning of a internet service then you still would have committed the crime of "Damaging a protected computer by intentional access". Negligently damaging a protected computer is punishable by 1 year in prison on the first offence. Recklessly damaging a protected computer is punishable by 1-5 years on the first offense. And intentionally damaging a protected computer is punishable by 1-10 years for the first offense. These penalties can go up to 20 years for repeated offenses.
- davidhyde 5y ago
  I don't know what kind of data you serve up but perhaps you could serve low quality or inaccurate content from addresses that are guessed from your api. I.e. endpoints not normally reachable in the normal functioning of your web app should return reasonable junk. A mixture of accurate and inaccurate data becomes worthless for bots and worthless data is not worth scraping. Just an idea!
- bastardoperator 5y ago
  Have you considered using a cache service like cloudflare?
- KennyBlanken 5y ago
  > While I have sympathy for what the scrapers are trying to do in many cases, it bothers me that this doesn't seem to address what happens when badly-behaved scrapers cause, in effect, a DOS on the site. Like when Aaron Swartz spent months hammering JSTOR causing it to become so slow it was almost unusuable, and despite knowing that he was causing widespread problems (including the eventual banning of MIT's entire IP range) actually worked to add additional laptops and improve his scraping speed...all the while going out of his way to subvert MIT's netops group trying to figure out where he was on the network. JSTOR, by the way, is a non-profit that provides aggregate access to their cataloged archive of journals, for schools and libraries to access journals they would otherwise never be able to afford. In many cases, free access.
  - uncletammy 5y ago
    The effect on Jstore's revenue would have been negligible. I'm surprised to see someone so cold and unfeeling about Aaron Swartz. Especially considering the massive injustice with regards to application of the law and sentencing. > Federal prosecutors, led by Carmen Ortiz, later charged him with two counts of wire fraud and eleven violations of the Computer Fraud and Abuse Act, carrying a cumulative maximum penalty of $1 million in fines, 35 years in prison, asset forfeiture, restitution, and supervised release.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- asdfasgasdgasdg 5y ago
  Seems like you could sue the scraper for that, then? If they cause you damages by their unapproved actions, you have a tort claim.
- FrenchDevRemote 5y ago
  What have you done to protect the site? Most automation libraries are detectable(puppeteer, even with extra-stealth, selenium, playwright...) The only library that I know that is more or less undetectable is used by a just a few hundred people...
  - JimBlackwood 5y ago
    Colour me interested in this library! :)
  - collegeburner 5y ago
    Curious, what library is that?
lazyjeff 5y ago
Now I wonder whether “retrieving” your own OAuth token from an app to make REST calls that extract your own data from cloud services is legal. It seems to fall under the same guideline, that exceeding authorization is not unauthorized access, so even though it's usually against the terms of service it doesn't violate CFAA?
TedDoesntTalk 5y ago
This is great news. A win for the Internet Archive and other archivists.
- ghaff 5y ago
  IANAL but it's not immediately obvious to me that this ruling covers bulk scraping and republishing untransformed. I'm genuinely curious about this personally. I presumably can't just grab anything I feel like off the web, curate it, and sell it.
  - cphoover 5y ago
    Why not isn't that how google has become one of the largest companies in the world?
    - simiones 5y ago
      Google became huge with the public by providing a way to search for data; and they became huge as a business by using this mediation to sell targeted ads. It's only after they had become truly huge that they started actually serving data directly (news snippets, song lyrics, factual answers, maps, reviews etc).
      - cphoover 5y ago
        .... I dunno what the distinction is...They clearly were scraping all data, in terms of display they extracted pieces of publicly available websites to use as descriptions and titles, and images. They displayed cached versions of websites. So the amount of content you take of the larger body makes the distinction? But as you get larger those rules no longer apply? It doesn't make sense to me.
        ghaff 5y ago
        >So the amount of content you take of the larger body makes the distinction? Of course. The amount of the original work is one of the key criteria on whether something is fair use or not. And there have been debates over whether Google crosses that line or not. But no one except the most ardent anti-copyright maximalists would argue that so long as something is available on a public web page, anyone can just reproduce it verbatim and distribute it on their own site. Scrapers do a lot of this anyway but doing anything about it is a wack-a-mole game of publications often don't bother.
bobajeff 5y ago
I had no idea this was even being discussed. I'm glad they are reasonable on this. Wish they had been as reasonable on breaking encryption/DRM schemes.
jakelazaroff 5y ago
If I have to stake out a binary position here, I’m pro scraping. But I really wish we could find a way to be more nuanced here. The scraper in question is looking at public LinkedIn profiles so that it can snitch to employers about which employees might be looking for new jobs. That’s not at all the same as archival; it’s using my data to harm me.
- brians 5y ago
  Yes, but it’s specifically using data you published to harm you. Compare Blind, which is engineered to not be attributable in this way.
  - jakelazaroff 5y ago
    I understand that I published it. That doesn’t mean I should accept that hostile parties will use it against me. This is kinda like telling someone who is being harassed on social media that they’re consenting to it by having a public account. We should strive to make our digital personae safe from bad actors, not throw our hands up and say “if you put yourself out there, you have no recourse”.
- xboxnolifes 5y ago
  It's a public page. Your employer could just as well check your page theirself. It may be a tragedy of efficiency, but it's not like the scraper is grabbing hidden data. The issue is in something else. Maybe it's the fact that your current employer would punish you for looking for a new job. Or maybe LinkedIn's public "looking for job" status is not sustainable in it's current form.
  - notch656a 5y ago
    Weev was charged, and eventually convicted based merely on scraping from AT&T [0]. When the charge was vacated, it was only on venue/jurisdiction, not on the basis of the scraping being legal. Seems there's precedent merely scraping this information is felonious behavior. https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-beginning-end https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-begi...
ketzu 5y ago
I am not versed in law, especially not in US law, but this case seems to be very specific that scraping is no violation of the CFAA. I do support this interpretation. However, the case of scraping I personally find more problematic is the use of personal data I provide to one side, then used by scrapers without my knowledge or permission. I truly wonder which way we are better off on that issue as a society. Independent of the current law, should anything that is accessible be essentially free-for-all or should there be limitations on what you are allowed to do. Cases highlighted in the article: Facial recognition by third parties on social media profiles, facebook scraping for personal data, search engines, journalists or archives. (Not all need to have the same answer to the question "do we want this") Besides that, the point I care slightly less about is the idea that allowing scaping with very leisure limits leads to even more closed up systems.
- henryfjordan 5y ago
  "I gave my data to Linkedin and now scrapers are reading it from the public web". Be mad at Linkedin before you are mad at the scraper.
  - ketzu 5y ago
    This seems to have various angles to it. First, one question is if the intent of the original owner of the data important? When I put data on linkedin (or facebook, my private website, hackenews or my employers website) I might have an opinion on who gets to do what with my data (see also GDPR discussions). Should I blame linkedin (or meta/myself/my employer) to do what I expected them to do, or should I blame those that do what I don't want them to do? Should I just be blamed directly because I even want to make a distinction between those? If I didn't want my data I could just not provide it (or participate in/surf the web at all if we extend the idea to more general data collection). Secondly, it touches on the idea that linkedin should not make the data publicly available (i.e., without authentication), and we end up with a less open system. Is that better? Is it what we want? Maybe there are also other ways that I am not aware just now. (Competing purely on value added is probably futile for data aggregators.)
cloudyporpoise 5y ago
These anti-scraping corporate activists need to get with the times and allow access to their data, legitimately, through an API. Third parties will scrape and sell the data regardless, so why not just cut them out and even charge for individuals to legitimately use the API. API keys could be tied to an individual and at least LinkedIn would know who was conducting what action. Make it easier to get the data through an API than having to scrape it.
8bitsrule 5y ago
Wondered how this works WRT copyright (since the article did not contain the word). Here's Kent State's (short) IPR advice [https://libguides.library.kent.edu/data-management/copyright https://libguides.library.kent.edu/data-management/copyright] It says "Data are considered 'facts' under U.S. law. They are not copyrightable.... Creative arrangement, annotation, or selection of data can be protected by copyright."
1vuio0pswjnm7 5y ago
"On LinkedIn, our members trust us with their information, which is why we prohibit unauthorized scraping on our platform." This is an unpersuasive argument because it ignores all the computer users who are not "members". Whether or not "members" trust LinkedIn should have no bearning on whether other computer users who may or may not be "members" can retrieve others' public information. Even more, this statement does not decry so-called scraping only "unauthorised" scraping. Who provides "authorisation". Surely not the LinkedIn members. It is presumptuous if not ridiculous for "tech" companies to claim computer users "trust" them. Most of these companies recieve no feedback from the majority of their "members". Tech companies generally have no "customer service" for the members they target with data collection. Further, there is an absence of meaningful choice. It is like saying people "trust" credit bureaus with their information. History shows these data collection intermediaries could not be trusted and that is why Americans have the Fair Credit Reporting Act.
- MWil 5y ago
  Great point about non-members/members
- maximilianroos 5y ago
  Relatedly — the court found that LinkedIn was selling a very similar dataset to the one they were attempting to cut off — so the claim that they were protecting customers' privacy was stretched
  - deleted 5y ago
    [deleted]
  - unityByFreedom 5y ago
    Haha what a joke. Do you have a link for that?
    - FrenchDevRemote 5y ago
      I think he's talking about the fact that with a premium membership you can have access to turnover rates stats etc...
MWil 5y ago
Next up, we just need all public court records to be freely available to BE scraped and not $3 per page https://patentlyo.com/patent/2020/08/court-pacer-should.html https://patentlyo.com/patent/2020/08/court-pacer-should.html
- KennyBlanken 5y ago
  Really the problem is that PACER has been turned into a cash cow for the federal court system, with fees and profits growing despite costs being virtually nill. But yeah, the irony of the federal court system legalizing screen scraping, something PACER contractually prohibits. Court records should be public record, not encumbered in terms of use/access/reproduction/copyright.
infiniteL0Op 5y ago
nomoreusernames 5y ago
so i dont have the right to not be scraped? thats like sending radiowaves and making me pay a license for a radio i dont use. same with spam. i should give you a token to send me emails i maybe want to look at your stuff.
notch656a 5y ago
An interesting departure, considering weev was convicted on merely scraping [0] AT&T. Although his charge was vacated, it was on the venue/jurisdiction, not that scraping was found to be legal. [0] https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-beginning-end https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-begi...
whatever1 5y ago
Is it legal though to login to a website and then scrape data (that would not be accessible if I was just browsing as a guest)?
- cphoover 5y ago
  Plaid seems to think so
amelius 5y ago
Where is the lobbying? Something seems wrong ...
ricardo81 5y ago
For what it's worth, Linkedin was incredibly easy to scrape back in the day, wrt profile/email correlation. Can't buy any aggressive stance they may have against 'scrapers'. 2 options. Their Linkedin ID's are base 12, and would redirect you if you simply wanted to enumerate them. You could also upload your 'contacts', 200-300 at a time and it'd leak profile IDs (Twitter and Facebook mitigated this ~5 years ago). I still have a @pornhub or some such "contact" that I can't delete from testing this.
hendersoon 5y ago
Simple answer, require users to login to view information then apply captchas and a series of bans with increasing durations when they act like a scraper rather than a human. LinkedIn already does the behavior detection, captchas, and bans.
fatih-erikli 5y ago
Technically doing a copy-paste and crawling is the same thing. If copy/pasting something is legal, then crawling it should be legal too. If not, vice versa. In any case, it ends up with copyright related laws, I think.
_eigenfoo 5y ago
"Web scraping is legal" seems to be an overbroad interpretation - the Ninth Circuit is merely reaffirming a preliminary injunction in light of a recent and separate ruling by the Supreme Court. The opinion [1] only weighs the merits hiQ's request for injunctive relief, and doesn't say anything about the meatier topic of web scraping legality. I think the real action will be the ruling of the district court. [1] https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/1...
- kemitchell 5y ago
  First Thought: That headline seems wropng. People are probably gonna get burned just reading that. Second Thought: I should really read the opinion before I make any comments. Clicked through, skimmed around, looked for a link to the slip looking for the opinion link ... couldn't find it. Thank you, sir.
- _eigenfoo 5y ago
  Upon more digging [1], the district court trial is currently scheduled for Feb 27 2023. It'll take a while! [1] https://storage.courtlistener.com/recap/gov.uscourts.cand.312704/gov.uscourts.cand.312704.236.0.pdf https://storage.courtlistener.com/recap/gov.uscourts.cand.31...
tastysandwich 5y ago
I'm (obviously) not a lawyer. Does "publicly accessible" also apply to websites that are totally free, but require registration (which is open to anyone)? And even though this is in the US, do cases like these set precedents for other western nations? asking for a friend ...
- wraptile 5y ago
  AFAIK anything that binds you a ToS (e.g. registration) is offlimit for legal scraping in USA as USA can literally put you in prison for violating _terms of service_, which is pretty crazy. Other countries are much better about this, though you should always consult a lawyer if something you want to scrape is behind a login page.
xtat 5y ago
Ok now do private data
pkdpic 5y ago
So glad I caught this post. I need to write more scrapers.
KieranMac 5y ago
This headline is horribly misleading. The Ninth Circuit said that web scraping is not a violation of the CFAA, but that other causes of action, such as trespass to chattels, breach of contract, misappropriation, may still apply. Just super irresponsible writing. Still, a very good day for scraping.
neycoda 5y ago
I mean, it's public data... you scrape it with your eyes whenever you view a web page, and your computer scrapes it in the browser. Saying web scraping should be illegal is like saying taking photos of buildings should be illegal.
- IAmEveryone 5y ago
  Taking a photo of a copyrighted photo (and distributing it) is illegal. You're just saying that anything that's technically possible should be legal, since there is no need to prohibit the technically impossible.
xrd 5y ago
Can anyone tell me what this means for sites that proxy and display content from other sites? For example, if this were applied to a RSS reader (something like Google Reader), how would this ruling affect that service?
nurettin 5y ago
This makes me feel less like an information pirate. Not cool.
WYepQ4dNnG 5y ago
If the data is publicly available then it should be legal to scrape it and use it. LinkedIn can decide, if they want, to put all the data behind login. Problem solved.
- cphoover 5y ago
  I agree. to me this is like people complaining someone is taking a photograph of their front yard... If you don't want something visible, don't display it to the world.
1vuio0pswjnm7 5y ago
Here is the opinion for anyone who is interested https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/1...
IAmEveryone 5y ago
Lots of people here seem to believe that scraping should be legal because it is possible. The flaw in that argument is that a law is never needed, by definition, to prevent things that can easily be avoided by technical means. The laws against burglary protect your property and health but, more importantly, they allow us all to live in peace without investing vast amounts of resources into physical protection.
Drakula2k 5y ago
Wasn't it already the case (https://en.wikipedia.org/wiki/Web_scraping#United_States https://en.wikipedia.org/wiki/Web_scraping#United_States)? That's why I usually recommend using US-based web scraping tools and services (such as https://webscraping.ai https://webscraping.ai). In other countries, it still may be a grey area.
etewiah 5y ago
Great news for the real estate sector. I actually built an open source project for scraping real estate websites several years ago: https://github.com/RealEstateWebTools/property_web_scraper https://github.com/RealEstateWebTools/property_web_scraper Might be time to go back and update it ;)
alkonaut 5y ago
Excellent. And the only ruling that makes any sense. What's important here though is to confirm that this also can't somehow be waivered by "Terms Of Service". The law needs to clearly say that web scraping is legal and that sites can't just circumvent the law by agreement with users. Even better if it could also clearly state that it includes any and all digital content and is not limited to (say) hypertext. I.e. if I as a user can reach content in any way, then I can also scrape/save/archive/whatever using the tech I want. E.g. I can save my youtube videos, spotify songs and so on.
hamasho 5y ago
Every time I hear news like this, I remember this local news in Japan [0]. An old bureaucratic company developed a system for a local library. It was very slow, so a user started scraping it and developed his own library system. A few weeks later, he was arrested. The system was so shit that it became unusable by only hundreds of requests per hour. The local government filed a claim, and the officer arrested him for the "dangerous cyber attack." [0] In Japanese: https://www.wikiwand.com/ja/%E5%B2%A1%E5%B4%8E%E5%B8%82%E7%AB%8B%E4%B8%AD%E5%A4%AE%E5%9B%B3%E6%9B%B8%E9%A4%A8%E4%BA%8B%E4%BB%B6 https://www.wikiwand.com/ja/%E5%B2%A1%E5%B4%8E%E5%B8%82%E7%A...
- dahfizz 5y ago
  Isn't that literally a DOS attack?
  - grardb 5y ago
    I'm not sure that "hundreds of requests per hour" meets the criteria for a DDOS. Assuming 999 requests per hour, that's about one request every four seconds, which the vast majority of websites should be able to handle with no trouble at all.
    - simiones 5y ago
      DoS is different from DDoS. A DoS attack is any kind of attack in which the attacker seeks not to gain access to a system, but to simply deny others access. This can include sending large payloads, maintaining idle connections, causing crashes, logging out other users etc. A DDoS is a particular kind of DoS attack - one in which the attack is conducted using a distributed set of machines, usually helping to (a) hide the attack from monitoring tools and (b) make it hard to block using IPs. Obviously this was not a DDoS, but could have looked like a plain DoS. The definition of a DoS attack depends on the system being attacked, there is no hard and fast limit to how much traffic a system is supposed to handle. For a badly written system like the one described here, the actions of the scraper were indistinguishable from the point of view of the library from the actions of a malicious actor. Hopefully, since the intent was not malicious in this case, thing may have been cleared up - though of course it's also possible that, unfortunately, they haven't.
dncornholio 5y ago
Last week I got a call from some random recruiter. I was wondering how they'd get my telephone number, because I can't recall putting my phonenumber publicly anywhere. Apparently they scrape profiles from linkedin, then put those through some Google Chrome extension (should've asked which one), and it spills out phone numbers! Contemplating if I should switch phone number now.
generic-husky 5y ago
It has kinda always baffled me, why some people think web scraping wouldn't/shouldn't be legal. I mean the second you put something out on the internet, you should expect it to be public, and as a consequence of that, to be scraped.
- simiones 5y ago
  Well, I can think of at least a few possible reasons. Not claiming those are right at all, but I do think they are the reasons why some people think/thought that scraping could be illegal. 1. It's hard to distinguish scrapers from DoS attackers, which we do expect to be illegal. 2. Depending on the use of the data after scraping, and the nature of the data being scraped, you may expect copyright laws and things like the GDPR to prevent any possible use of the data, so the very collection of it may be suspect. 3. Even for public data, a site may have Terms of Service that don't permit scraping, and some may expect to be able to enforce these
Helloyello 5y ago
iav 5y ago
Scraping is legal, but the legal framework requires scraping to abide by three rules: 1. The page is not behind a login. If you have to create an account to access the content, then you have to agree to abide by the terms of service which may ban scraping. 2. What you scrape is still protected by copyright. If you scrape a clip from Star Wars from a web page, you still can't redistribute it without a license. 3. Your activity may not impede access to the site in anyway i.e. no DDOSing. If there is a robots.txt file, you are supposed to abide by it but the courts do not literally say that. Must rate meter your requests, scrape at night, so on. Gray area if the robots.txt bans every page and you end up having to ignore it.
- oauea 5y ago
  Since when are terms of service legally binding?
  - colejohnson66 5y ago
    They never weren’t. After all, they’re contracts that you agree to when you check the box. Some clauses may be unenforceable, but voiding a clause does not necessarily void the whole contract.
  - maglite77 5y ago
    Always have been. See: https://en.wikipedia.org/wiki/Terms_of_service https://en.wikipedia.org/wiki/Terms_of_service
  - sodality2 5y ago
    Legally binding =/= enforceable with penalties or illegal to break (excepting revocation of service). Breaking TOS is not a crime [0] [0]: https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-down-violating-websites-terms-service-not-crime https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...
- tssva 5y ago
  The Supreme Court decision referenced in this decision involved a case where the information accessed was protected by a login and accessing the information violated the terms of service. The court ruled that is not a violation of the CFAA. I fail to see that it is therefore necessarily required that the page is not behind a login for scraping to be legal. There may be other issues regarding the type of information or repeatedly creating accounts to skirt account terminations which affect whether it is legal, but simply violating terms of service for accessing a page behind a login doesn't appear to make it illegal by itself.

1040 points by spenvo 5y ago

altdataseller 5y ago

Does the ruling make it illegal to block scrapers?

postalrat 5y ago
Doubt it. And it should never be illegal to attempt to block scrapers. Many sites do try.

ricardo81 5y ago

Interesting as I've seen a few search engine start ups that seem to scrape other search engine results, depending on your definition of scraping. My definition would be a user agent that doesn't uniquely identify itself that isn't using an authorised API.

CWuestefeld 5y ago

While I have sympathy for what the scrapers are trying to do in many cases, it bothers me that this doesn't seem to address what happens when badly-behaved scrapers cause, in effect, a DOS on the site. For the family of sites I'm responsible for, bot traffic comprises a majority of traffic - that is, to a first approximation, the lion's share of our operational costs are from needing to scale to handle the huge amount of bot traffic. Even when it's not as big as a DOS, it doesn't seem right to me that I can't tell people they're not welcome to cause this additional system load. Or even if there was some standardized way that we could provide a dumb API, just giving them raw data so we don't need to incur the additional expense of the processing for creature comforts on the page designed to make our users happier but the bots won't notice.

svnpenn 5y ago
Implement TLS fingerprinting on your server. People can still fake that if they are determined, but it should cut the abuse way down.
- CWuestefeld 5y ago
  No, nor can we just do it by IP. The bots are MUCH more sophisticated than that. More often than not, it's a cooperating distributed net of hundreds of bots, coming from multiple AWS, Azure, and GCP addresses. So they can pop up anywhere, and that IP could wind up being a real customer next week. And they're only recognizable as a botnet with sophisticated logic looking at the gestalt of web logs. We do use a 3rd party service to help with this - but that on its own is imposing a 5- to 6-digit annual expense on our business.
  - z3c0 5y ago
    Have you considered setting up an API to allow the bots to get what they want without hammering your front-end servers?
    - kayodelycaon 5y ago
      What makes you think they would use it?
      - z3c0 5y ago
        It's in their interest. I've scraped a lot, and it's not easy to build a reliable process on. Why parse a human interface when there's an application interface available?
    - CWuestefeld 5y ago
      Yes. And if I could get the perpetrators to raise their hands so I could work out an API for them, it would be the path of least resistance. But they take great pains to be anonymous, although I know from circumstantial evidence that at least a good chunk of it is various competitors (or services acting on behalf of competitors) scraping price data. IANAL, but I also wonder if, given that I'd be designing something specifically for competitors to query our prices in order to adjust their own prices, this would constitute some form of illegal collusion.
      - 5y ago
borski 5y ago
You could ban their IPs?
- KMnO4 5y ago
  IP bans are equivalent to residential door locks. They’re only deterring the most trivial attacks. In school I needed to scrape a few hundred thousand pages of a proteomics database website. For some reason you had to view each entry one at a time. There was IP throttling which banned you if you made requests too quickly. But slowing the script to 1 request per second would have taken days to scrape the site. So I paid <$5 for a list of 500 proxy servers and distributed it, completing the task in under half an hour.
  - l33t2328 5y ago
    Using proxies to hide your identity to get around a denial of access seems to get awfully close to violating the Computer Fraud and Abuse Act(in USA, at least). I’m surprised your school was okay with it.
    - KMnO4 5y ago
      Don't worry, I don't live in the USA. Thanks for your concern though.
  - borski 5y ago
    I agree it’s not perfect. It’s also significantly better than nothing.
    - Gigachad 5y ago
      It's also completely insufficient despite being better. There are so many scraper services that its just a matter of paying a small amount of money to make use of tens of thousands of IPs spread across ISPs and countries.
  - inshadows 5y ago
    Can you share where did you get such a nice proxy for 500 proxies? TIA
forgotmypw17 5y ago
Yes, I think working to accommodate the non-humans along with the humans is the right approach here. Scrapers have a limited range of IPs, so rate-limiting them and stalling (or dropping) request responses is one way to deal with the DoS scenario. For my sites, I have placed the majority behind HTTP Basic Auth...
- KptMarchewa 5y ago
  You realistically can't. There are services like [0][1] that mean any IP could be a scraper node. [0] https://brightdata.com/proxy-types/residential-proxies https://brightdata.com/proxy-types/residential-proxies [1] https://oxylabs.io/products/residential-proxy-pool https://oxylabs.io/products/residential-proxy-pool
  - orlp 5y ago
    > How does Bright Data acquire its residential IPs? > Bright Data has built a unique consumer IP model by which all involved parties are fairly compensated for their voluntary participation. App owners install a unique Software Development Kit (SDK) to their applications and receive monthly remuneration based on the number of users who opt-in. App users can voluntarily opt-in and are compensated through an ad-free user experience or enjoy an upgraded version of the app they are using for free. These consumers or ‘peers’ serve as the basis of our network and can opt-out at any time. This model has brought into existence an unrivaled, first of its kind, ethically sound, and compliant network of real consumers. I don't know how they can say with a straight face that this is 'ethically sound'. They have, essentially, created a botnet, but apparently because it's "AdTech" and the user "opts-in" (read: they click on random buttons until they hit one that makes the banner/ad go away) it's suddenly not malware.
    - TedDoesntTalk 5y ago
      NordVPN (Tesonet) has another business doing the same thing. They sell the IP addresses/bandwidth of their NordVPN customers to anyone who needs bulk mobile or residential IP addresses. That's right, installing their VPN software adds your IP address to a pool that NordVPN then resells. Xfinity/Comcast sort of pioneered this with their wifi routers that automatically expose an isolated wifi network called 'xfinity' (IIRC) whether you agree or not.
Nextgrid 5y ago
But don't you already have countermeasures to deter DoS attacks or malicious human users (what if someone pays or convinces people to open your site and press F5 repeatedly)? If not, you should, and the badly-behaved scrapers are actually a good wake-up call.
rmbyrro 5y ago
I have sympathy for your operational issues and costs, but isn't this kind of complaint the same as a shopping mall/center complaining of people who go in, check some info and go out without buying? I understand that bots have leverage and automation, but so does you to reach a larger audience. Should we continue to benefit from one side of the leverage, while complaining about the other side?
- wlesieutre 5y ago
  It's more like a mall complaining that while they're trying to serve 1000 customers, someone has gone and dumped 10000000 roombas throughout the stores which are going around scanning all the price tags.
- CWuestefeld 5y ago
  No. When I say that bots exceed the amount of real traffic, I'm including people "window shopping" on the good side. My complaint is more like, somebody wants to know the prices of all our products, and that we have roughly X products (where X is a very large number). They get X friends to all go into the store almost simultaneously, each writing down the price of the particular product they've been assigned to research. When they do this, there's scant space left in the store for even the browsing kind of customers to walk in. (of course I exaggerate a bit, but that's the idea)
  - mrobins 5y ago
    I’m sympathetic to the complaints about “rude” scraping behavior but there’s an easy solution. Rather than make people consume boatloads of resources they don’t want (individual page views, images, scripts, etc.) just build good interoperability tools that give the people what they want. In the physical example above that would be a product catalog that’s easily replicated with a CSV product listing or an API.
    - jensensbutton 5y ago
      You don't know why any random scraper is scraping you and thus you don't know what api to build that will do them from scraping. Also, it's likely easier for them to contribute scraping than write a bunch of code to integrate with your API so it there's no incentive for them to do so either.
linuxdude314 5y ago
If most of your traffic is bots, is the site even worth running? This really is akin to the question, “Should others be allowed to take my photo or try to talk to me in public?” Of course the answer should be yes, the internet is the digital equivalent of a public space. If you make it accessible, anyone should be able to consume. If you don’t want it scraped add auth!
- simiones 5y ago
  Do you also believe someone running a drone to follow and photograph you, personally, wherever you go in public would be fair and legal?
kstrauser 5y ago
I've told this story before, but it was fun, so I'm sharing it again: I'll skip the details, but a previous employer dealt with a large, then-new .mil website. Our customers would log into the site to check on the status of their invoices, and each page load would take approximately 1 minute. Seriously. It took about 10 minutes to log in and get to the list of invoices available to be checked, then another minute to look at one of them, then another minute to get out of it and back into the list, and so on. My job was to write a scraper for that website. It ran all night to fetch data into our DB, and then our website could show the same information to our customers in a matter of milliseconds (or all at once if they wanted one big aggregate report). Our customers loved this. The .mil website's developer hated it, and blamed all sorts of their tech problems on us, although: - While optimizing, I figured out how to skip lots of intermediate page loads and go directly to the invoices we wanted to see. - We ran our scraper at night so that it wouldn't interfere with their site during the day. - Because each of our customers had to check each one of their invoices every day if they wanted to get paid, and we were doing it more efficiently, our total load on their site was lower than the total load of our customers would be. Their site kept crashing, and we were there scapegoat. It was great fun when they blamed us in a public meeting, and we responded that we'd actually disabled our crawler for the past week, so the problem was still on their end. Eventually, they threatened to cut off all our access to the site. We helpfully pointed out that their brand new site wasn't ADA compliant, and we had vision-impaired customers who weren't able to use it. We offered to allow our customers to run the same reports from our website, for free, at no cost to the .mil agency, so that they wouldn't have to rebuild their website from the ground up. They saw it our way and begrudgingly allowed us to keep scraping.
- dwater 5y ago
  I have worked with .mil customers who paid us to scrape and index their website because they didn't have a better way to access their official, public documents.
  - jll29 5y ago
    This is not .mil specific: I've been told of a case where an airline first legally attacked a flight search engine (Skyscanner) for scraping, and then told them to continue when they realized that their own search engine couldn't handle all the traffic, and even if it could, it was more expensive per query than routing via Skyscanner.
bequanna 5y ago
As someone that has been on the other end, I can tell you devs don’t want to use selenium or inspect requests to reverse engineer your UI and wish there were more clean APIs. Have you tried making your UI more challenging to scrape and adding a simple API that requires free registration?
- JimBlackwood 5y ago
  So much this! I work in E-Commerce and (needless to say) we scrape a lot of websites. Due to our growth and the increase in scrapers we require, I’ve been writing a proposal to a higher up to talk to our biggest competitors to all set up a public API that batches the data for a smaller amount of requests. It would save everyone quite some traffic and effort.
userbinator 5y ago
That's what rate-limiting is for. Don't be so aggressive with it that you start hitting the faster visitors, however, or they may soon go somewhere else (has happened to me a few times).
- loceng 5y ago
  Do you know if there's a way to rate limit logged-in users differently than visitors of a site?
  - minusf 5y ago
    nginx can be set up to do that using the session cookie.
  - rolph 5y ago
    rate limiting can be a double edged sword, you can be better off giving a scraper highest bandwidth so they are gone sooner, otherwise somthing like making a zip or other sort of compilation of the site available may be an option. just what kind of scraper you have is a concern. does scraper just want a bunch of stock images; or does scraper have FOMO on web trinkets; or does scraper want to mirror/impersonate your site. the last option is the most concerning because then; scraper is mirroring bcz your site is cool and local UI/UX is wanted; or is scraper phishing smishing or otherwise duping your users.
    - loceng 5y ago
      Yeah, good points to consider. I think the sites that would be scrapped the most would be where the data is regularly and reliably up-to-date, and a large volume of it at that - so not just one scraper but many different parties may on a daily or weekly basis try to scrap every page. I feel that ruling should have the caveat that if a fair cost paid API version for getting publicly listed data then the scrapers must legally use that (say no more than 5% more than cost of CPU/bandwidth/etc of the scraping behaviour); ideally a rule too that at minimum there be a delay if they are republishing that data without your permission, so at least you as the platform/source/reason for the data being up-to-date aren't harmed too - which may then kill the source platform over time if regular visitors somehow start going to the competitor publishing the data.
ryan_j_naughton 5y ago
As other have said (A) there are plenty of countermeasures you can take, but also (B) you are frustrated that you are providing something free to the public and then annoyed at the "wrong" customers are using your product and costing you money. I'm sorry, but this is a failure of your business model. If we were to analogize this to a non-internet example: (1) A company throws a free concert/event and believes they will make money by alcohol sales. (2) A bunch of sober/non-drinking folks attend the concert but only drink water (3) Company blames the concert attendees for "taking advantage" of them when they really just had poor company policies and a bad business model. Put things behind authentication and authorization. Add a paywall. Implement DDOS and detection and banning approaches for scrapers. Etc etc. But don't make something public and then get mad at THE PUBLIC for using it. Behind that machine is a person, who happens to be a member of the public.
- noisenotsignal 5y ago
  There are certain classes of websites where the proposed solutions aren’t a great fit. For example, a shopping site hiding their catalog behind paywalls or authentication would raise barriers to entry such that a lot of genuine customers would be lost. I don’t think the business model is in general to be blamed here and it’s ok to acknowledge the unfortunate overhead and costs added by site usage patterns (e.g. scraping) that are counter to the expectation.
- inlined 5y ago
  Alternatively it could be seen that your juice company offers free samples. Then somebody abuses free and takes gallons home with them to bottle and sell as their own. That’s what it feels like when someone is scraping your network to bootstrap a competitor.
  - ryan_j_naughton 5y ago
    Again, what you call abuse of free samples, someone else calls a savvy strategy tailored to you're poorly crafted business plan. Have ways to limit the free samples or else it's your fault...
deleted 5y ago
[deleted]
gdulli 5y ago
When the original ruling in favor of HiQ came out, it still allowed for LinkedIn to block certain kinds of malicious scraping. LinkedIn had been specifically blocking HiQ, and was ordered to stop doing that.
brailsafe 5y ago
You probably can. On the protocol level with JSON-LD or other rich data packages that generate xml or standardized json endpoints. I did this for an open data portal, and this is something most G7 governments do with their federal open data portals using off the shelf packages (that are worth researching a bit obviously first), particularly in the python and flask world. We were still getting hammered by China at our Taiwanese language subdomain, but that was a different concern
YPCrumble 5y ago
Reading your comment my impression is that this is either an exaggeration or a very unique type of site if bots make up the majority of traffic to the point that scrapers are anywhere near the primary load factor. Would someone let me know if I’m just plain wrong in this assumption? I’ve run many types of sites and scrapers have never been anywhere close to the main source of traffic or even particularly noticeable compared to regular users. Even considering a very commonly scraped site like LinkedIn or Craigslist - for any site of any magnitude like this public pages are going to be cached so additional scrapers are going to have negligible impact. And a rate limit is probably one line of config. I’m not saying you are necessarily wrong, but I can’t imagine a scenario that you’re describing and would love to hear of one.
- nojito 5y ago
  Bandwidth isn't free.
  - YPCrumble 5y ago
    I didn’t say it is free, I said that the bandwidth for bots is negligible compared to that of regular users.
    - nojito 5y ago
      Negligible isn’t free either.
- VWWHFSfQ 5y ago
  > a very unique type of site if bots make up the majority of traffic Pretty much Twitter and the majority of such websites.
  - YPCrumble 5y ago
    Do you really believe bots make up a significant amount of Twitter’s operating cost? Like I said they’re just accessing cached tweets and are rate limited. How can the bot usage possibly be more than a small part of twitter’s operating cost?
throw10920 5y ago
Have you considered serving a proof-of-work challenge to clients accessing your website? Minimal cost on legit users, but large costs on large-scale web-scraping operations, and it doesn't matter if they split up their efforts across a bunch of IP addresses - they're still going to have to do those computations. https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash
- nyuszika7h 5y ago
  No thanks, as a user I would stay far away from such websites. This is akin to crypto miners. I don't need them to drive up my electricity costs and also contribute to global warming in the process. It's not worth the cost.
  - fouric 5y ago
    This is completely absurd - anti-spam PoW is not remotely comparable to crypto miners, and the electricity cost will be so far below the noise floor of owning a computer in the first place that you will literally not notice (and neither will the environment), unless website owners are completely insane and set up multi-second challenges (which, they won't). And, it's absolutely worth the cost - as a website owner, you get to impose costs on botting operations with minimal penalties for normal users and minimal environmental impact. Bots work because the costs of renting an AWS server and scraping websites (or sending spam, whatever) are extremely tiny - adding PoW challenges to everything that could be spammed suddenly massively changes the cost of running those spam operations, and would result in noticeably less spam if deployed widely. In fact, the net "environmental impact" would be negative, as botters start to shut down operations due to greatly increased operational costs.
  - danamit 5y ago
    You do PoW every time you send an email.
colinmhayes 5y ago
I'm sympathetic to this. I built a search engine for my senior project and my half baked scraper ended up taking down duke law's site during their registration period. Ended up getting a not so kindly worded email from them, but honestly this wasn't an especially hard problem to solve. All of my traffic was coming from the cluster that was on my university's subnet, it wouldn't have been that hard to for them to IP address timeouts when my crawler started scraping thousands of pages a second on their site. Not to victim blame, this was totally my fault, but I was a bit surprised that they hadn't experienced this before with how much automated scraping goes on.
- brightball 5y ago
  I’m honestly more interested in bot detection than anything else at this point. It seems like it should be perfectly legal to detect and then hold the connection open for a long period of time without giving a useful response. Or even send highly compressed gzip responses designed to fill their drives. Legal or not, I can’t see any good reason that we can’t make it painful.
  - fjabre 5y ago
    Make it painful if they abuse the site. We all benefit from open data. Polite scrapers are just fine and a natural part of the web ecosystem. Google has been scraping the web all day every day for decades now.
deleted 5y ago
[deleted]
hardtke 5y ago
The problem with many sites (and LinkedIn in particular) is that they whitelist a bunch of specific websites, presumably based on the business interests, but disallow everyone else in their robots.txt. You should either allow all scrapers that respect certain load requirements or allow none. Anything that Google is allowed to see and include in their search results should be fair game. Here's the end of LinkedIn's robots.txt: User-agent: * Disallow: / # Notice: If you would like to crawl LinkedIn, # please email whitelist-crawl@linkedin.com to apply # for white listing.
- diamondage 5y ago
  And this is what the HiQ case hinged on. LinkedIn were essentially selectively applying the computer fraud and abuse act based on their business interests - that was never going to sit well with judges.
- JimBlackwood 5y ago
  Btw, LinkedIn does have an API for things like Sales Navigator. It used some weird partnership program (SNAP) to get into it and it starts at (I think) 1500$/year per user. Still pretty cheap though, I think you’d get the value out of that quite quickly for a >300 people company.
car_analogy 5y ago
> Even when it's not as big as a DOS, it doesn't seem right to me that I can't tell people they're not welcome to cause this additional system load. You can tell them. You just can't prosecute them if they don't obey.
voxic11 5y ago
The court just ruled that scraping on its own isn't a violation of the CFAA. Meaning it doesn't count as the crime of "accessing a protected computer without authorization or exceeding authorized access and obtaining information". However presumably all the other provisions of the CFAA still apply, so if your scraping damages the functioning of a internet service then you still would have committed the crime of "Damaging a protected computer by intentional access". Negligently damaging a protected computer is punishable by 1 year in prison on the first offence. Recklessly damaging a protected computer is punishable by 1-5 years on the first offense. And intentionally damaging a protected computer is punishable by 1-10 years for the first offense. These penalties can go up to 20 years for repeated offenses.
davidhyde 5y ago
I don't know what kind of data you serve up but perhaps you could serve low quality or inaccurate content from addresses that are guessed from your api. I.e. endpoints not normally reachable in the normal functioning of your web app should return reasonable junk. A mixture of accurate and inaccurate data becomes worthless for bots and worthless data is not worth scraping. Just an idea!
bastardoperator 5y ago
Have you considered using a cache service like cloudflare?
KennyBlanken 5y ago
> While I have sympathy for what the scrapers are trying to do in many cases, it bothers me that this doesn't seem to address what happens when badly-behaved scrapers cause, in effect, a DOS on the site. Like when Aaron Swartz spent months hammering JSTOR causing it to become so slow it was almost unusuable, and despite knowing that he was causing widespread problems (including the eventual banning of MIT's entire IP range) actually worked to add additional laptops and improve his scraping speed...all the while going out of his way to subvert MIT's netops group trying to figure out where he was on the network. JSTOR, by the way, is a non-profit that provides aggregate access to their cataloged archive of journals, for schools and libraries to access journals they would otherwise never be able to afford. In many cases, free access.
- uncletammy 5y ago
  The effect on Jstore's revenue would have been negligible. I'm surprised to see someone so cold and unfeeling about Aaron Swartz. Especially considering the massive injustice with regards to application of the law and sentencing. > Federal prosecutors, led by Carmen Ortiz, later charged him with two counts of wire fraud and eleven violations of the Computer Fraud and Abuse Act, carrying a cumulative maximum penalty of $1 million in fines, 35 years in prison, asset forfeiture, restitution, and supervised release.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
asdfasgasdgasdg 5y ago
Seems like you could sue the scraper for that, then? If they cause you damages by their unapproved actions, you have a tort claim.
FrenchDevRemote 5y ago
What have you done to protect the site? Most automation libraries are detectable(puppeteer, even with extra-stealth, selenium, playwright...) The only library that I know that is more or less undetectable is used by a just a few hundred people...
- JimBlackwood 5y ago
  Colour me interested in this library! :)
- collegeburner 5y ago
  Curious, what library is that?

lazyjeff 5y ago

Now I wonder whether “retrieving” your own OAuth token from an app to make REST calls that extract your own data from cloud services is legal. It seems to fall under the same guideline, that exceeding authorization is not unauthorized access, so even though it's usually against the terms of service it doesn't violate CFAA?

TedDoesntTalk 5y ago

This is great news. A win for the Internet Archive and other archivists.

ghaff 5y ago
IANAL but it's not immediately obvious to me that this ruling covers bulk scraping and republishing untransformed. I'm genuinely curious about this personally. I presumably can't just grab anything I feel like off the web, curate it, and sell it.
- cphoover 5y ago
  Why not isn't that how google has become one of the largest companies in the world?
  - simiones 5y ago
    Google became huge with the public by providing a way to search for data; and they became huge as a business by using this mediation to sell targeted ads. It's only after they had become truly huge that they started actually serving data directly (news snippets, song lyrics, factual answers, maps, reviews etc).
    - cphoover 5y ago
      .... I dunno what the distinction is...They clearly were scraping all data, in terms of display they extracted pieces of publicly available websites to use as descriptions and titles, and images. They displayed cached versions of websites. So the amount of content you take of the larger body makes the distinction? But as you get larger those rules no longer apply? It doesn't make sense to me.
      - ghaff 5y ago
        >So the amount of content you take of the larger body makes the distinction? Of course. The amount of the original work is one of the key criteria on whether something is fair use or not. And there have been debates over whether Google crosses that line or not. But no one except the most ardent anti-copyright maximalists would argue that so long as something is available on a public web page, anyone can just reproduce it verbatim and distribute it on their own site. Scrapers do a lot of this anyway but doing anything about it is a wack-a-mole game of publications often don't bother.

bobajeff 5y ago

I had no idea this was even being discussed. I'm glad they are reasonable on this. Wish they had been as reasonable on breaking encryption/DRM schemes.

jakelazaroff 5y ago

If I have to stake out a binary position here, I’m pro scraping. But I really wish we could find a way to be more nuanced here. The scraper in question is looking at public LinkedIn profiles so that it can snitch to employers about which employees might be looking for new jobs. That’s not at all the same as archival; it’s using my data to harm me.

brians 5y ago
Yes, but it’s specifically using data you published to harm you. Compare Blind, which is engineered to not be attributable in this way.
- jakelazaroff 5y ago
  I understand that I published it. That doesn’t mean I should accept that hostile parties will use it against me. This is kinda like telling someone who is being harassed on social media that they’re consenting to it by having a public account. We should strive to make our digital personae safe from bad actors, not throw our hands up and say “if you put yourself out there, you have no recourse”.
xboxnolifes 5y ago
It's a public page. Your employer could just as well check your page theirself. It may be a tragedy of efficiency, but it's not like the scraper is grabbing hidden data. The issue is in something else. Maybe it's the fact that your current employer would punish you for looking for a new job. Or maybe LinkedIn's public "looking for job" status is not sustainable in it's current form.
- notch656a 5y ago
  Weev was charged, and eventually convicted based merely on scraping from AT&T [0]. When the charge was vacated, it was only on venue/jurisdiction, not on the basis of the scraping being legal. Seems there's precedent merely scraping this information is felonious behavior. https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-beginning-end https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-begi...

ketzu 5y ago

I am not versed in law, especially not in US law, but this case seems to be very specific that scraping is no violation of the CFAA. I do support this interpretation. However, the case of scraping I personally find more problematic is the use of personal data I provide to one side, then used by scrapers without my knowledge or permission. I truly wonder which way we are better off on that issue as a society. Independent of the current law, should anything that is accessible be essentially free-for-all or should there be limitations on what you are allowed to do. Cases highlighted in the article: Facial recognition by third parties on social media profiles, facebook scraping for personal data, search engines, journalists or archives. (Not all need to have the same answer to the question "do we want this") Besides that, the point I care slightly less about is the idea that allowing scaping with very leisure limits leads to even more closed up systems.

henryfjordan 5y ago
"I gave my data to Linkedin and now scrapers are reading it from the public web". Be mad at Linkedin before you are mad at the scraper.
- ketzu 5y ago
  This seems to have various angles to it. First, one question is if the intent of the original owner of the data important? When I put data on linkedin (or facebook, my private website, hackenews or my employers website) I might have an opinion on who gets to do what with my data (see also GDPR discussions). Should I blame linkedin (or meta/myself/my employer) to do what I expected them to do, or should I blame those that do what I don't want them to do? Should I just be blamed directly because I even want to make a distinction between those? If I didn't want my data I could just not provide it (or participate in/surf the web at all if we extend the idea to more general data collection). Secondly, it touches on the idea that linkedin should not make the data publicly available (i.e., without authentication), and we end up with a less open system. Is that better? Is it what we want? Maybe there are also other ways that I am not aware just now. (Competing purely on value added is probably futile for data aggregators.)

cloudyporpoise 5y ago

These anti-scraping corporate activists need to get with the times and allow access to their data, legitimately, through an API. Third parties will scrape and sell the data regardless, so why not just cut them out and even charge for individuals to legitimately use the API. API keys could be tied to an individual and at least LinkedIn would know who was conducting what action. Make it easier to get the data through an API than having to scrape it.

8bitsrule 5y ago

Wondered how this works WRT copyright (since the article did not contain the word). Here's Kent State's (short) IPR advice [https://libguides.library.kent.edu/data-management/copyright https://libguides.library.kent.edu/data-management/copyright] It says "Data are considered 'facts' under U.S. law. They are not copyrightable.... Creative arrangement, annotation, or selection of data can be protected by copyright."

1vuio0pswjnm7 5y ago

"On LinkedIn, our members trust us with their information, which is why we prohibit unauthorized scraping on our platform." This is an unpersuasive argument because it ignores all the computer users who are not "members". Whether or not "members" trust LinkedIn should have no bearning on whether other computer users who may or may not be "members" can retrieve others' public information. Even more, this statement does not decry so-called scraping only "unauthorised" scraping. Who provides "authorisation". Surely not the LinkedIn members. It is presumptuous if not ridiculous for "tech" companies to claim computer users "trust" them. Most of these companies recieve no feedback from the majority of their "members". Tech companies generally have no "customer service" for the members they target with data collection. Further, there is an absence of meaningful choice. It is like saying people "trust" credit bureaus with their information. History shows these data collection intermediaries could not be trusted and that is why Americans have the Fair Credit Reporting Act.

MWil 5y ago
Great point about non-members/members
maximilianroos 5y ago
Relatedly — the court found that LinkedIn was selling a very similar dataset to the one they were attempting to cut off — so the claim that they were protecting customers' privacy was stretched
- deleted 5y ago
  [deleted]
- unityByFreedom 5y ago
  Haha what a joke. Do you have a link for that?
  - FrenchDevRemote 5y ago
    I think he's talking about the fact that with a premium membership you can have access to turnover rates stats etc...

MWil 5y ago

Next up, we just need all public court records to be freely available to BE scraped and not $3 per page https://patentlyo.com/patent/2020/08/court-pacer-should.html https://patentlyo.com/patent/2020/08/court-pacer-should.html

KennyBlanken 5y ago
Really the problem is that PACER has been turned into a cash cow for the federal court system, with fees and profits growing despite costs being virtually nill. But yeah, the irony of the federal court system legalizing screen scraping, something PACER contractually prohibits. Court records should be public record, not encumbered in terms of use/access/reproduction/copyright.

infiniteL0Op 5y ago

nomoreusernames 5y ago

so i dont have the right to not be scraped? thats like sending radiowaves and making me pay a license for a radio i dont use. same with spam. i should give you a token to send me emails i maybe want to look at your stuff.

notch656a 5y ago

An interesting departure, considering weev was convicted on merely scraping [0] AT&T. Although his charge was vacated, it was on the venue/jurisdiction, not that scraping was found to be legal. [0] https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-beginning-end https://www.eff.org/deeplinks/2013/07/weevs-case-flawed-begi...

whatever1 5y ago

Is it legal though to login to a website and then scrape data (that would not be accessible if I was just browsing as a guest)?

cphoover 5y ago
Plaid seems to think so

amelius 5y ago

Where is the lobbying? Something seems wrong ...

ricardo81 5y ago

For what it's worth, Linkedin was incredibly easy to scrape back in the day, wrt profile/email correlation. Can't buy any aggressive stance they may have against 'scrapers'. 2 options. Their Linkedin ID's are base 12, and would redirect you if you simply wanted to enumerate them. You could also upload your 'contacts', 200-300 at a time and it'd leak profile IDs (Twitter and Facebook mitigated this ~5 years ago). I still have a @pornhub or some such "contact" that I can't delete from testing this.

hendersoon 5y ago

Simple answer, require users to login to view information then apply captchas and a series of bans with increasing durations when they act like a scraper rather than a human. LinkedIn already does the behavior detection, captchas, and bans.

fatih-erikli 5y ago

Technically doing a copy-paste and crawling is the same thing. If copy/pasting something is legal, then crawling it should be legal too. If not, vice versa. In any case, it ends up with copyright related laws, I think.

_eigenfoo 5y ago

"Web scraping is legal" seems to be an overbroad interpretation - the Ninth Circuit is merely reaffirming a preliminary injunction in light of a recent and separate ruling by the Supreme Court. The opinion [1] only weighs the merits hiQ's request for injunctive relief, and doesn't say anything about the meatier topic of web scraping legality. I think the real action will be the ruling of the district court. [1] https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/1...

kemitchell 5y ago
First Thought: That headline seems wropng. People are probably gonna get burned just reading that. Second Thought: I should really read the opinion before I make any comments. Clicked through, skimmed around, looked for a link to the slip looking for the opinion link ... couldn't find it. Thank you, sir.
_eigenfoo 5y ago
Upon more digging [1], the district court trial is currently scheduled for Feb 27 2023. It'll take a while! [1] https://storage.courtlistener.com/recap/gov.uscourts.cand.312704/gov.uscourts.cand.312704.236.0.pdf https://storage.courtlistener.com/recap/gov.uscourts.cand.31...

tastysandwich 5y ago

I'm (obviously) not a lawyer. Does "publicly accessible" also apply to websites that are totally free, but require registration (which is open to anyone)? And even though this is in the US, do cases like these set precedents for other western nations? asking for a friend ...

wraptile 5y ago
AFAIK anything that binds you a ToS (e.g. registration) is offlimit for legal scraping in USA as USA can literally put you in prison for violating _terms of service_, which is pretty crazy. Other countries are much better about this, though you should always consult a lawyer if something you want to scrape is behind a login page.

xtat 5y ago

Ok now do private data

pkdpic 5y ago

So glad I caught this post. I need to write more scrapers.

KieranMac 5y ago

This headline is horribly misleading. The Ninth Circuit said that web scraping is not a violation of the CFAA, but that other causes of action, such as trespass to chattels, breach of contract, misappropriation, may still apply. Just super irresponsible writing. Still, a very good day for scraping.

neycoda 5y ago

I mean, it's public data... you scrape it with your eyes whenever you view a web page, and your computer scrapes it in the browser. Saying web scraping should be illegal is like saying taking photos of buildings should be illegal.

IAmEveryone 5y ago
Taking a photo of a copyrighted photo (and distributing it) is illegal. You're just saying that anything that's technically possible should be legal, since there is no need to prohibit the technically impossible.

xrd 5y ago

Can anyone tell me what this means for sites that proxy and display content from other sites? For example, if this were applied to a RSS reader (something like Google Reader), how would this ruling affect that service?

nurettin 5y ago

This makes me feel less like an information pirate. Not cool.

WYepQ4dNnG 5y ago

If the data is publicly available then it should be legal to scrape it and use it. LinkedIn can decide, if they want, to put all the data behind login. Problem solved.

cphoover 5y ago
I agree. to me this is like people complaining someone is taking a photograph of their front yard... If you don't want something visible, don't display it to the world.

1vuio0pswjnm7 5y ago

Here is the opinion for anyone who is interested https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/1...

IAmEveryone 5y ago

Lots of people here seem to believe that scraping should be legal because it is possible. The flaw in that argument is that a law is never needed, by definition, to prevent things that can easily be avoided by technical means. The laws against burglary protect your property and health but, more importantly, they allow us all to live in peace without investing vast amounts of resources into physical protection.

Drakula2k 5y ago

Wasn't it already the case (https://en.wikipedia.org/wiki/Web_scraping#United_States https://en.wikipedia.org/wiki/Web_scraping#United_States)? That's why I usually recommend using US-based web scraping tools and services (such as https://webscraping.ai https://webscraping.ai). In other countries, it still may be a grey area.

etewiah 5y ago

Great news for the real estate sector. I actually built an open source project for scraping real estate websites several years ago: https://github.com/RealEstateWebTools/property_web_scraper https://github.com/RealEstateWebTools/property_web_scraper Might be time to go back and update it ;)

alkonaut 5y ago

Excellent. And the only ruling that makes any sense. What's important here though is to confirm that this also can't somehow be waivered by "Terms Of Service". The law needs to clearly say that web scraping is legal and that sites can't just circumvent the law by agreement with users. Even better if it could also clearly state that it includes any and all digital content and is not limited to (say) hypertext. I.e. if I as a user can reach content in any way, then I can also scrape/save/archive/whatever using the tech I want. E.g. I can save my youtube videos, spotify songs and so on.

hamasho 5y ago

Every time I hear news like this, I remember this local news in Japan [0]. An old bureaucratic company developed a system for a local library. It was very slow, so a user started scraping it and developed his own library system. A few weeks later, he was arrested. The system was so shit that it became unusable by only hundreds of requests per hour. The local government filed a claim, and the officer arrested him for the "dangerous cyber attack." [0] In Japanese: https://www.wikiwand.com/ja/%E5%B2%A1%E5%B4%8E%E5%B8%82%E7%AB%8B%E4%B8%AD%E5%A4%AE%E5%9B%B3%E6%9B%B8%E9%A4%A8%E4%BA%8B%E4%BB%B6 https://www.wikiwand.com/ja/%E5%B2%A1%E5%B4%8E%E5%B8%82%E7%A...

dahfizz 5y ago
Isn't that literally a DOS attack?
- grardb 5y ago
  I'm not sure that "hundreds of requests per hour" meets the criteria for a DDOS. Assuming 999 requests per hour, that's about one request every four seconds, which the vast majority of websites should be able to handle with no trouble at all.
  - simiones 5y ago
    DoS is different from DDoS. A DoS attack is any kind of attack in which the attacker seeks not to gain access to a system, but to simply deny others access. This can include sending large payloads, maintaining idle connections, causing crashes, logging out other users etc. A DDoS is a particular kind of DoS attack - one in which the attack is conducted using a distributed set of machines, usually helping to (a) hide the attack from monitoring tools and (b) make it hard to block using IPs. Obviously this was not a DDoS, but could have looked like a plain DoS. The definition of a DoS attack depends on the system being attacked, there is no hard and fast limit to how much traffic a system is supposed to handle. For a badly written system like the one described here, the actions of the scraper were indistinguishable from the point of view of the library from the actions of a malicious actor. Hopefully, since the intent was not malicious in this case, thing may have been cleared up - though of course it's also possible that, unfortunately, they haven't.

dncornholio 5y ago

Last week I got a call from some random recruiter. I was wondering how they'd get my telephone number, because I can't recall putting my phonenumber publicly anywhere. Apparently they scrape profiles from linkedin, then put those through some Google Chrome extension (should've asked which one), and it spills out phone numbers! Contemplating if I should switch phone number now.

generic-husky 5y ago

It has kinda always baffled me, why some people think web scraping wouldn't/shouldn't be legal. I mean the second you put something out on the internet, you should expect it to be public, and as a consequence of that, to be scraped.

simiones 5y ago
Well, I can think of at least a few possible reasons. Not claiming those are right at all, but I do think they are the reasons why some people think/thought that scraping could be illegal. 1. It's hard to distinguish scrapers from DoS attackers, which we do expect to be illegal. 2. Depending on the use of the data after scraping, and the nature of the data being scraped, you may expect copyright laws and things like the GDPR to prevent any possible use of the data, so the very collection of it may be suspect. 3. Even for public data, a site may have Terms of Service that don't permit scraping, and some may expect to be able to enforce these

Helloyello 5y ago

iav 5y ago

Scraping is legal, but the legal framework requires scraping to abide by three rules: 1. The page is not behind a login. If you have to create an account to access the content, then you have to agree to abide by the terms of service which may ban scraping. 2. What you scrape is still protected by copyright. If you scrape a clip from Star Wars from a web page, you still can't redistribute it without a license. 3. Your activity may not impede access to the site in anyway i.e. no DDOSing. If there is a robots.txt file, you are supposed to abide by it but the courts do not literally say that. Must rate meter your requests, scrape at night, so on. Gray area if the robots.txt bans every page and you end up having to ignore it.

oauea 5y ago
Since when are terms of service legally binding?
- colejohnson66 5y ago
  They never weren’t. After all, they’re contracts that you agree to when you check the box. Some clauses may be unenforceable, but voiding a clause does not necessarily void the whole contract.
- maglite77 5y ago
  Always have been. See: https://en.wikipedia.org/wiki/Terms_of_service https://en.wikipedia.org/wiki/Terms_of_service
- sodality2 5y ago
  Legally binding =/= enforceable with penalties or illegal to break (excepting revocation of service). Breaking TOS is not a crime [0] [0]: https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-down-violating-websites-terms-service-not-crime https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...
tssva 5y ago
The Supreme Court decision referenced in this decision involved a case where the information accessed was protected by a login and accessing the information violated the terms of service. The court ruled that is not a violation of the CFAA. I fail to see that it is therefore necessarily required that the page is not behind a login for scraping to be legal. There may be other issues regarding the type of information or repeatedly creating accounts to skirt account terminations which affect whether it is legal, but simply violating terms of service for accessing a page behind a login doesn't appear to make it illegal by itself.