8 ms·

Scraping Isn't Hacking

108 points by danso 6y ago

scarygliders 6y ago
Scraping isn't hacking, sure. But I've witnessed some particularly aggressive scrapers bring a shopping site to its knees. I've watched as multiple scrapers were launched at this site from different blocks of IP addresses. I spent some hours blocking these. After spending even more time, I tracked down these IP blocks - from multiple countries - back to the one scraping company, who had obviously been hired by a competitor to scrape the entire site for its prices for the ranges of goods sold. I consider such aggressive scraping to be a DDoS attack.
- andyfleming 6y ago
  It’s more like service slaughter (which isn’t a term AFAIK) than denial of service (since denial isn’t the goal). It’s a tough problem though. In an ideal world, I think scrapers should pay for the compute/resources required to serve their requests. I think that’s one of the only ways to scale for this fairly. (This assumes you agree that scraping of public content should be allowed in the first place.)
  - jka 6y ago
    That's an interesting idea. Sometimes a lot of the (service provision) cost depends on how efficient the data access is at the host. For example, let's take an API that includes a 'GET /data?start=0&end=100' endpoint that retrieves items with key between 0 and 100. If items are stored in sorted key order, that could be an efficient and inexpensive query that retrieves exactly the 100 relevant items and doesn't perform any other unnecessary work. If, on the other hand, the database contains a billion items and they aren't stored in key order, then with a caller-pays pricing model, it's (probably, for the sake of argument) not worth running that query. In some cases it might be reasonable to continue to pay for the expensive queries. But what if all you need to do is convince the host to 'CREATE INDEX ... ON (key)'? (all very hypothetical - I don't have any explicit answers. if the host service were open source, the answer might be 'open a pull request and talk to them, and hopefully reduce everyone's costs')
    - ethanwillis 6y ago
      If I was evil and the law was that scrapers had access BUT they had to pay their fair share of the cost. I would just make their costs so absurdly high that it wouldn't be beneficial to them.
      - andyfleming 6y ago
        Yeah, you’d almost need some way to genetically price this. You’d almost want websites to opt into it. You can either accept the $x/request price or not participate in the program. Though, I guess at that point you might as well just sell API access.
    - andyfleming 6y ago
      Yeah, a hard part of this is that it depends on the efficiency of the implementation by the website creators/maintainers. At least at that point you’re on the same side. You’re both trying to be fair and you’re both trying to help the website work well.
bernardlunn 6y ago
Lousy title. Not as clickbaity might be sometimes scraping is legal and sometimes it is ethical. Creator of original content here.
jjice 6y ago
Does anyone have tips for polite scraping? Just making sure not to hurt a site (or get IP blocked)?
- Nextgrid 6y ago
  Start by assuming good faith from the website regarding scrapers and clearly identify yourself in your user-agent (provide a contact email or something) and do it from a single IP. Submit a reasonable amount of requests/second and ramp it up slowly depending on the response time and size/popularity of the website (if it's a national e-commerce brand you can hit it with hundreds per second with no expected ill effects because that's a drop in the bucket compared to their overall traffic, but a very small website on shared hosting might warrant a more cautious approach). Of course, if they are hostile to it and don't like the "being nice" approach then fight fire with fire and try to get what you want as fast as possible while blending into their overall traffic so they can't identify nor block it. It might sometimes even involve paying real people (on Mechanical Turk or similar) to just browse the website as they normally would and scrape it manually.
- abdusco 6y ago
  Scrapy (a popular python scraping framework) has an auto-throttling feature that lets you limit concurrent requests and requests per minute for a domain. I tweak these values to what I estimate the website could handle (depending on how large it is). The page also outlines how throttling algorithm works, which you use as a starting point for your own work. https://docs.scrapy.org/en/latest/topics/autothrottle.html https://docs.scrapy.org/en/latest/topics/autothrottle.html
- dgb23 6y ago
  I assume that there is merit in throttling the request frequency.
- andyfleming 6y ago
  This is a tough problem because if you asked most sites/admins if you could scrape their site, they don’t really have much motivation to say yes. For some it feels like you are stealing their hard work. For others they don’t want to pay for the requests your scraper will make to their site, etc. Regardless, I’m an ideal world, some polite things would be to: - ask for permission before scraping, explaining how it’s neutral or positive for their business (so they are more likely to support your continued scraping or even potentially provide a more cost-effective format/API for you) - scrape at a reasonable pace - scrape off hours - make requests with a clear user agent so they know where the requests are coming from
superkuh 6y ago
Well, the article is right. But what happens in practice in cases like these is that everything has been made illegal and it's just arbitrarily enforced. All behavior that grandpa judge can't understand, like, say, anything beyond using a browser (say, wget?) is a hacking tool and as long as the person you annoyed has deep pockets or connections you'll go to prison. The problem isn't with the details of any particular scraping or subtly. The problem is with the law, CFAA 1030, and how it can be interpreted however the particular DA wants. The judges are too ignorant of technology to stop them.
- ghaff 6y ago
  >All behavior that grandpa judge can't understand, Stop with the ageist crap.
  - mellosouls 6y ago
    Downvoters should seriously consider if they would feel the same if gender/race/whatever was inserted instead...
    - drewcoo 6y ago
      You mean if all the judges were old white men and they were somehow unable to understand what seemed obvious to young women of color so their rulings were discriminatory? Ok, I'm seriously considering it.
      - mellosouls 6y ago
        I mean the comment I replied to was down-voted (greyed at the time) for rightly calling out the ageism in the parent. I'm not entirely sure what you are saying in response.
        superkuh 6y ago
        I thought about this sequence of comments seriously for quite a while. I accept that "grandpa judge" was inappropriate. But how would you phrase the problem of our judges not understanding technology? Is it a judge thing? Lawyers understand. But lawyer tend to be younger. There are few young judges. I think age, and the lack of life experience with tech because they grew up before it, probably is the factor that matters here.
vmception 6y ago
Well it shouldn't fall under the current language of the CFAA Weak minded sheltered programmers will unnecessarily commit suicide if you levy that law against them for scraping!
- myself248 6y ago
  That's just cold.
paulryanrogers 6y ago
Hacking as in a crime. For some reason I read this as people who scrape websites aren't real 'hackers', in the no True Scotsman sense.
- Sebb767 6y ago
  Same here. Maybe 'Scraping should not fall under hacking laws' would be a more clear title.
  - jbob2000 6y ago
    More accurately, “The Markup believes that its web scraping activities aren’t considered ‘unauthorized access of a computer’”
    - CrazyStat 6y ago
      "The Markup believes [...]" is underselling it. The 9th circuit, for one, agrees with them [1]. In the case at question here the 11th circuit ruled the other way. When circuit courts come to opposite conclusions the Supreme Court will sometimes step in to provide a ruling so that there's unified precedent. Hopefully they'll agree with the 9th circuit. [1] https://arstechnica.com/tech-policy/2019/09/web-scraping-doesnt-violate-anti-hacking-law-appeals-court-rules/ https://arstechnica.com/tech-policy/2019/09/web-scraping-doe...
- caymanjim 6y ago
  The term "hacking" long ago shifted to mean "illegal activity" unless otherwise specified or obvious from the context. In the context of this title it's clear what was meant. I'm an ancient, pedantic nerd, and even I don't cling to the archaic 1970s meaning of "hacking".
  - caymanjim 6y ago
    I'm aware of the irony of claiming this on "Hacker News".
  - deleted 6y ago
dafoex 6y ago
I might be thinking about this the wrong way, but, scrapers as described by this article are more-or-less targeted web crawlers, right? Don't we have a de-facto standard for handling this? User-agent: scraperbot Disallow: /
- FastedCoyote 6y ago
  It's trivial to change the User Agent.
  - dafoex 6y ago
    It is, but what I'm suggesting is that a scraper identifies itself as such in the user agent and complies with the robots exclusion standard. Google's crawlers recognise that `googlebot` - among other strings - refers to them, and a scraper should behave similarly if it doesn't want to be seen as malicious. I think scrapers should be legal, but just like crawlers, they should comply with a directive that says they aren't welcome. And on the topic of trivial workarounds, it's even more trivial to just ignore the robots.txt and scrape away, but that's what malicious bots performing questionable activities do. One would hope a newsbot is being civil and not behaving in a shady manner, but alas, I don't for one moment think any of these news publishers have even heard of a robots.txt
deleted 6y ago
[deleted]
swyx 6y ago
i'm a little confused why this article doesn't also bring up the other high profile supreme court scraping case - the HiQ LinkedIn case (https://www.natlawreview.com/article/hiq-files-opposition-brief-supreme-court-linkedin-cfaa-data-scraping-dispute https://www.natlawreview.com/article/hiq-files-opposition-br...) Why does the Supreme Court seem to be considering these two related cases together? seems like duplication?
- artembugara 6y ago
  This case is really really specific. LinkedIn did not owe the data of people who posted it there.
goldenkey 6y ago
It's funny how scraping can be illegal yet Google and Microsoft and Yahoo and DuckDuckGo, and every other search engine have a monopoly on it. It's legal for them, because .....? Money money money! Google even hotlinks images for Google images... pretty hilarious stuff.
- Google234 6y ago
  You can block them in your robot.txt if you desire... it’s not the same thing, websites want to be indexed
- dafoex 6y ago
  Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.
danbmil99 6y ago
This case https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn Is relevant here. LinkedIn is invoking the Computer Fraud and Abuse Act for scraping a website without logging in. While hiQ won a preliminary injunction, the case is ongoing and LinkedIn / Microsoft has asked the Supreme Court to weigh in.
- patagurbon 6y ago
  Surely Microsoft would want Bing to have access to most websites right? Bing probably doesn't bring in the same revenue though...
  - Google234 6y ago
    It’s in those websites interested to allow bing to index them. bing respects the robot.txt
    - artembugara 6y ago
      are you sure about respecting robots.txt?
      - Google234 6y ago
        Yes they do. Does newscatcherapi respect them?

108 points by danso 6y ago

Scraping isn't hacking, sure. But I've witnessed some particularly aggressive scrapers bring a shopping site to its knees. I've watched as multiple scrapers were launched at this site from different blocks of IP addresses. I spent some hours blocking these. After spending even more time, I tracked down these IP blocks - from multiple countries - back to the one scraping company, who had obviously been hired by a competitor to scrape the entire site for its prices for the ranges of goods sold. I consider such aggressive scraping to be a DDoS attack.

andyfleming 6y ago
It’s more like service slaughter (which isn’t a term AFAIK) than denial of service (since denial isn’t the goal). It’s a tough problem though. In an ideal world, I think scrapers should pay for the compute/resources required to serve their requests. I think that’s one of the only ways to scale for this fairly. (This assumes you agree that scraping of public content should be allowed in the first place.)
- jka 6y ago
  That's an interesting idea. Sometimes a lot of the (service provision) cost depends on how efficient the data access is at the host. For example, let's take an API that includes a 'GET /data?start=0&end=100' endpoint that retrieves items with key between 0 and 100. If items are stored in sorted key order, that could be an efficient and inexpensive query that retrieves exactly the 100 relevant items and doesn't perform any other unnecessary work. If, on the other hand, the database contains a billion items and they aren't stored in key order, then with a caller-pays pricing model, it's (probably, for the sake of argument) not worth running that query. In some cases it might be reasonable to continue to pay for the expensive queries. But what if all you need to do is convince the host to 'CREATE INDEX ... ON (key)'? (all very hypothetical - I don't have any explicit answers. if the host service were open source, the answer might be 'open a pull request and talk to them, and hopefully reduce everyone's costs')
  - ethanwillis 6y ago
    If I was evil and the law was that scrapers had access BUT they had to pay their fair share of the cost. I would just make their costs so absurdly high that it wouldn't be beneficial to them.
    - andyfleming 6y ago
      Yeah, you’d almost need some way to genetically price this. You’d almost want websites to opt into it. You can either accept the $x/request price or not participate in the program. Though, I guess at that point you might as well just sell API access.
  - andyfleming 6y ago
    Yeah, a hard part of this is that it depends on the efficiency of the implementation by the website creators/maintainers. At least at that point you’re on the same side. You’re both trying to be fair and you’re both trying to help the website work well.

bernardlunn 6y ago

Lousy title. Not as clickbaity might be sometimes scraping is legal and sometimes it is ethical. Creator of original content here.

jjice 6y ago

Does anyone have tips for polite scraping? Just making sure not to hurt a site (or get IP blocked)?

Nextgrid 6y ago
Start by assuming good faith from the website regarding scrapers and clearly identify yourself in your user-agent (provide a contact email or something) and do it from a single IP. Submit a reasonable amount of requests/second and ramp it up slowly depending on the response time and size/popularity of the website (if it's a national e-commerce brand you can hit it with hundreds per second with no expected ill effects because that's a drop in the bucket compared to their overall traffic, but a very small website on shared hosting might warrant a more cautious approach). Of course, if they are hostile to it and don't like the "being nice" approach then fight fire with fire and try to get what you want as fast as possible while blending into their overall traffic so they can't identify nor block it. It might sometimes even involve paying real people (on Mechanical Turk or similar) to just browse the website as they normally would and scrape it manually.
abdusco 6y ago
Scrapy (a popular python scraping framework) has an auto-throttling feature that lets you limit concurrent requests and requests per minute for a domain. I tweak these values to what I estimate the website could handle (depending on how large it is). The page also outlines how throttling algorithm works, which you use as a starting point for your own work. https://docs.scrapy.org/en/latest/topics/autothrottle.html https://docs.scrapy.org/en/latest/topics/autothrottle.html
dgb23 6y ago
I assume that there is merit in throttling the request frequency.
andyfleming 6y ago
This is a tough problem because if you asked most sites/admins if you could scrape their site, they don’t really have much motivation to say yes. For some it feels like you are stealing their hard work. For others they don’t want to pay for the requests your scraper will make to their site, etc. Regardless, I’m an ideal world, some polite things would be to: - ask for permission before scraping, explaining how it’s neutral or positive for their business (so they are more likely to support your continued scraping or even potentially provide a more cost-effective format/API for you) - scrape at a reasonable pace - scrape off hours - make requests with a clear user agent so they know where the requests are coming from

superkuh 6y ago

Well, the article is right. But what happens in practice in cases like these is that everything has been made illegal and it's just arbitrarily enforced. All behavior that grandpa judge can't understand, like, say, anything beyond using a browser (say, wget?) is a hacking tool and as long as the person you annoyed has deep pockets or connections you'll go to prison. The problem isn't with the details of any particular scraping or subtly. The problem is with the law, CFAA 1030, and how it can be interpreted however the particular DA wants. The judges are too ignorant of technology to stop them.

ghaff 6y ago
>All behavior that grandpa judge can't understand, Stop with the ageist crap.
- mellosouls 6y ago
  Downvoters should seriously consider if they would feel the same if gender/race/whatever was inserted instead...
  - drewcoo 6y ago
    You mean if all the judges were old white men and they were somehow unable to understand what seemed obvious to young women of color so their rulings were discriminatory? Ok, I'm seriously considering it.
    - mellosouls 6y ago
      I mean the comment I replied to was down-voted (greyed at the time) for rightly calling out the ageism in the parent. I'm not entirely sure what you are saying in response.
      - superkuh 6y ago
        I thought about this sequence of comments seriously for quite a while. I accept that "grandpa judge" was inappropriate. But how would you phrase the problem of our judges not understanding technology? Is it a judge thing? Lawyers understand. But lawyer tend to be younger. There are few young judges. I think age, and the lack of life experience with tech because they grew up before it, probably is the factor that matters here.

vmception 6y ago

Well it shouldn't fall under the current language of the CFAA Weak minded sheltered programmers will unnecessarily commit suicide if you levy that law against them for scraping!

myself248 6y ago
That's just cold.

paulryanrogers 6y ago

Hacking as in a crime. For some reason I read this as people who scrape websites aren't real 'hackers', in the no True Scotsman sense.

Sebb767 6y ago
Same here. Maybe 'Scraping should not fall under hacking laws' would be a more clear title.
- jbob2000 6y ago
  More accurately, “The Markup believes that its web scraping activities aren’t considered ‘unauthorized access of a computer’”
  - CrazyStat 6y ago
    "The Markup believes [...]" is underselling it. The 9th circuit, for one, agrees with them [1]. In the case at question here the 11th circuit ruled the other way. When circuit courts come to opposite conclusions the Supreme Court will sometimes step in to provide a ruling so that there's unified precedent. Hopefully they'll agree with the 9th circuit. [1] https://arstechnica.com/tech-policy/2019/09/web-scraping-doesnt-violate-anti-hacking-law-appeals-court-rules/ https://arstechnica.com/tech-policy/2019/09/web-scraping-doe...
caymanjim 6y ago
The term "hacking" long ago shifted to mean "illegal activity" unless otherwise specified or obvious from the context. In the context of this title it's clear what was meant. I'm an ancient, pedantic nerd, and even I don't cling to the archaic 1970s meaning of "hacking".
- caymanjim 6y ago
  I'm aware of the irony of claiming this on "Hacker News".
- deleted 6y ago

dafoex 6y ago

I might be thinking about this the wrong way, but, scrapers as described by this article are more-or-less targeted web crawlers, right? Don't we have a de-facto standard for handling this? User-agent: scraperbot Disallow: /

FastedCoyote 6y ago
It's trivial to change the User Agent.
- dafoex 6y ago
  It is, but what I'm suggesting is that a scraper identifies itself as such in the user agent and complies with the robots exclusion standard. Google's crawlers recognise that `googlebot` - among other strings - refers to them, and a scraper should behave similarly if it doesn't want to be seen as malicious. I think scrapers should be legal, but just like crawlers, they should comply with a directive that says they aren't welcome. And on the topic of trivial workarounds, it's even more trivial to just ignore the robots.txt and scrape away, but that's what malicious bots performing questionable activities do. One would hope a newsbot is being civil and not behaving in a shady manner, but alas, I don't for one moment think any of these news publishers have even heard of a robots.txt

deleted 6y ago

[deleted]

swyx 6y ago

i'm a little confused why this article doesn't also bring up the other high profile supreme court scraping case - the HiQ LinkedIn case (https://www.natlawreview.com/article/hiq-files-opposition-brief-supreme-court-linkedin-cfaa-data-scraping-dispute https://www.natlawreview.com/article/hiq-files-opposition-br...) Why does the Supreme Court seem to be considering these two related cases together? seems like duplication?

artembugara 6y ago
This case is really really specific. LinkedIn did not owe the data of people who posted it there.

goldenkey 6y ago

It's funny how scraping can be illegal yet Google and Microsoft and Yahoo and DuckDuckGo, and every other search engine have a monopoly on it. It's legal for them, because .....? Money money money! Google even hotlinks images for Google images... pretty hilarious stuff.

Google234 6y ago
You can block them in your robot.txt if you desire... it’s not the same thing, websites want to be indexed
dafoex 6y ago
Maybe scrapers should respect robots.txt and go away if there's a `scraperbot` entry, the same way Google goes away if you put `googlebot` in there.

danbmil99 6y ago

This case https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn Is relevant here. LinkedIn is invoking the Computer Fraud and Abuse Act for scraping a website without logging in. While hiQ won a preliminary injunction, the case is ongoing and LinkedIn / Microsoft has asked the Supreme Court to weigh in.

patagurbon 6y ago
Surely Microsoft would want Bing to have access to most websites right? Bing probably doesn't bring in the same revenue though...
- Google234 6y ago
  It’s in those websites interested to allow bing to index them. bing respects the robot.txt
  - artembugara 6y ago
    are you sure about respecting robots.txt?
    - Google234 6y ago
      Yes they do. Does newscatcherapi respect them?