12 ms·

Web Scraping Is Vital to Democracy

426 points by atg_abhishek 6y ago

JoeAltmaier 6y ago
Clearly its time to come up with rules for this activity. Picking an apple from a tree in a public park is clearly harmless. Bringing in a combine harvester to take every scrap is not. How do we set a boundary on scraping?
- booleandilemma 6y ago
  They should sell scraping licenses, right? /s
- slim 6y ago
  a website would be like an apple tree if there was an infinite number of apples in it. so you could bring a harvester and still leave enough apples for everyone.
mikewarot 6y ago
Not only is web scraping in danger, the trend towards HTTPS makes it harder to log traffic, and manage the flow of data to/from browsers on your own computer.
- steelbrain 6y ago
  Can you please explain how you got to that conclusion?
  - kupopuffs 6y ago
    Obviously makes it harder for others to log your traffic
  - mschuster91 6y ago
    Not OP, but it's very hard to intercept and monitor the traffic of an app that uses code signing, obfuscation and certificate pinning properly. What used to be a five minute "put tcpdump on the access point / router to work" job pre-HTTPS everywhere now is a many days worth job of messing around - out of reach for all but the really dedicated, and as a result it is very hard for an user to have actual visibility over where their data flows.
    - captn3m0 6y ago
      I'd say this is a consequence of making our phones opaque rather than HTTPS-everywhere. Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Doing the same on our mobile devices is much more tougher, because we've let feature phones become walled gardens.
      - mschuster91 6y ago
        > Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. As for the walled garden part: I agree with the general sentiment, but on the other hand I also see the lengths malware authors go to gather data from people. There really is no one-fits-all solution here, because anything that allows the user to intercept and monitor SSL communication can automatically be used by an attacker! :(
        captn3m0 6y ago
        > Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. Are there desktop apps that behave this way? Atleast in my experience - I haven't come across anything like this on Linux.
        mschuster91 6y ago
        Dropbox does: https://knowledge.broadcom.com/external/article/169397/dropbox-ssl-interception-and-issues-when.html https://knowledge.broadcom.com/external/article/169397/dropb... And I seriously hope cert pinning gets adopted by more applications.
        dylz 6y ago
        Many desktop apps behave this way, yes. Anything from a major tech co should at least
pessimizer 6y ago
It's pretty typical and sad how the US is prosecuting a dirty cop who looks up confidential data in exchange for money: on behalf of the system itself rather than on behalf of the victims' privacy, any limitations on the actions of law enforcement officers in general, or public corruption. In libertarianism/liberalism's final form, the only two kinds of laws left are lèse majesté and violations of terms of service.
1vuio0pswjnm7 6y ago
"In both of those instances, the pages and data scraped are publicly available on the internet-no hacking necessary-but sites involved could easily change the fine print on their terms of service to label the aggregation of that information "unauthorized."" They could. But if they filed a claim under the CFAA without ever sending a cease and desist letter to the alleged intruder, I think the claim would be dismissed. Is simply changing TOS enough for a CFAA claim to have a reasonable chance of success. I could be wrong, but I believe for every CFAA claim we have seen so far based on "scraping", there was some notice to stop directed specifically at the respondent. If someone thinks just changing TOS (public notice) is enough for a CFAA claim to survive a Motion to Dismiss and there is no need to also contact the alleged offender asking them to stop, then let's hear about the precedent that supports that idea. I do not think there is any such precedent, but I could be wrong.
- amelius 6y ago
  So robots.txt has no meaning?
  - fogihujy 6y ago
    I think the main issue is with scrapers that don't adhere to robots.txt/X-robot headers.
    - 9HZZRfNlpR 6y ago
      Google doesn't either, they just don't index the paths written in robots.txt but bots still hit them.
  - krageon 6y ago
    No, because robots.txt is a friendly request and not a demand. It can never be a demand.
  - err4nt 6y ago
    It's not binding, but it's a hint they may listen to where you can say things like if there's a certain URL structure where your server would keep serving content infinitely and just waste the crawlers time and resources, you can hint to them they maybe they should avoid certain URLs, but hey if they wanna waste their resources crawling that anyway be my guest!
wraptile 6y ago
This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and fingerprints to fight the nature of the internet. Finally everyone wants benefits of public data (search engine indexing etc) but don't really want to give anything back to the ecosystem. It's just pure greed and law, our society and government shouldn't aid it in any way, shape or form.
- IfOnlyYouKnew 6y ago
  You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do. Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them access becomes legally binding. In the offline world, we derive great benefit from following the second model. Indeed, if only measures that successfully prevent people from entering your house without permissions were to count, you wouldn't need laws in the first place! You would, however, need a bunker. Which is quite a bit more expansive than a functioning legal system. In that offline world, we have created all sorts of additional rules to balance rights for specific situations, and we rely on a canon of expectations that say, for example, that it's usually not ok to enter a private house, but you don't need explicit permission to enter a supermarket. These are still developing for the online world, and your idea of the "ecosystem" hints at that. But you're just taking from those contradictory ideas above to arrive at the outcome you intuitively feel is "just": a bit of might-is-right when it comes to "whatever is online is fair game", followed by principled ideas of rights and obligations when websites try to defend themselves in that jungle of yours. What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road.
  - 6y ago
charliebrownau 6y ago
Ever considered Democracy doesnt work Look at the last 5-25-55-255 years
mnmkng 6y ago
I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as well. Instead of hiring 100 interns to collect some data manually, you hire one programmer to automate that data collection process. I think that the ability to automate tasks on the internet is absolutely crucial to further development of our society and limiting it in any way will be detrimental to the world as a whole. The amount of information these days is so vast, that no human labor force could possibly analyze that information and use it to drive our progress. Disclaimer: We run a web scraping platform (https://apify.com https://apify.com)
- sokoloff 6y ago
  While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”. I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s going to be frowned upon.
  - simplecto 6y ago
    This is orthogonal to the purpose of scraping. If I run a search engine which potentially links real human eyes back to you, then should I pay the "ad" toll as well? I don't believe so. I do believe there is an agreeable middle-ground, but Google walked away from that conversation years ago.
  - krageon 6y ago
    Advertisements are harmful to every layer of society, mostly because they prey on you to instill desires that you probably wouldn't have had on your own (it being the case that this is their entire added value proposition). They should not be tolerated, and the fact that they ever were is a travesty. That being the case, any argument founded on "but advertisements" does not hold water.
bzb6 6y ago
You should also have called it a “human right” for improved virtue signalling.
- norwegianwhale 6y ago
  "Human right" is one of those sacred terms that are supposed to mean so much but mean nothing. "Human rights" are just superstitious conventions about what's good/bad. "No one can deprive you of housing" oh look I am gonna set up my tent here in this warm shop and claim human rights protection.
wslh 6y ago
The corollary would be that robots.txt exclusion for certain web crawlers should be removed.
tobyhinloopen 6y ago
We use scrapers to wrap and combine 3rd party tools and increase the usability and decrease amount of work required to get something done. Sometimes to do some task at a 3rd party tool, it requires many clicks and page loads. My scraper wrapping the tool automates most of the task and only keeps the manual task, reducing the amount of required clicks (and time) required to get the task done. I use a mix of headless browsers (for JS-heavy apps) and raw API calls. Sometimes I even use the browser to login, trigger a single sample APi call, extract all headers and content, close the browser, and re-run that api call directly using a HTTP lib. The request body obviously gets modified on the user’s requirements. We’re bypassing the slow login process as long as the session is valid. We’re also sharing sessions/logins/accounts this way without exposing credentials to users. It can also be done to bypass the only-one-session-per-user systems. This is done with permission from 3rd parties. They’re fine with it, they just didn’t want to provide a proper API or let us bypass the only-one-session rule because it requires code changes they’re not willing to make only for us. Sometimes the tool breaks when the html Or api changes but it usually only takes a few minutes to modify the code to fix it.
agustif 6y ago
Last weekend I made a little playwright script to checkout stock item status of decathlon products. The bike -Rodillo- Has been out of stock for several weeks. While coding it it was briefly on stock but I was too slow with my card input hahaha and so I neeed to now automate the buy part as soon as it's available, or make that run every 5min and mail/notify me if available... Btw this was super easy, playwright as a puppeteer successor really rocks, another thing hard to hate from MS like TS or VS Code PS: I brought down script execcution from 7 secocnds to 2 seocnds, by not loading any unneeded stuff css images external js fonts. I felt like a good scraper citizen doing so, it was just two linnes to block by request type on the network level, and it worked like a charm to speed up the process.
- bangtoyib 6y ago
  nice
crispyambulance 6y ago
I use web scraping on my employer's Oracle EBS instance because coordinating with the battle-axe personalities that run ERP's is such a pain in the ass I would rather have a root-canal. With Web Scraping, I don't even have to talk to them.
- appleflaxen 6y ago
  For others: it looks like EBS in this context means "oracle e-business suite" and ERP means "enterprise resource planning" (maybe everyone else knows these, but just in case) https://en.wikipedia.org/wiki/Oracle_Applications#Oracle_E-Business_Suite https://en.wikipedia.org/wiki/Oracle_Applications#Oracle_E-B... https://en.wikipedia.org/wiki/Enterprise_resource_planning https://en.wikipedia.org/wiki/Enterprise_resource_planning
  - crispyambulance 6y ago
    Yes, that's right, and those of us that don't have to deal with these dreadful systems are better off :-)
_wldu 6y ago
If you post a file to the WWW (World Wide Web) do not be shocked that the whole world can read and copy it. That's how it works. You're welcome.
SergeAx 6y ago
Okay, great point, but who will pay for that? Serving content costs money: traffic, extra load on CPU and storages, and now on serverless functions too. One spider may be equivalent of hundreds of real users. And it spoils caches too.
- emiliobumachar 6y ago
  Crackpot regulation proposal: scraping is legal if you favor mirror sites and afterwards create a mirror yourself and maintain it to some arbitrary standard.
werber 6y ago
I was trying to write a scraping script for a non profit to automate the work they were giving to law students to do manually earlier this year but the court system had agreed captchas that made it impossible. In my (legally uniformed) mind it felt like they were actively undermining people of lesser means by not allowing a simple script to run and essentially mandating an expensive legal team to do menial work
smarx007 6y ago
Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally). Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on the website to say: I am giving this tool my cloud power of attorney (best if signed through a digital govt id system, e.g. https://en.wikipedia.org/wiki/BankID https://en.wikipedia.org/wiki/BankID) and I take responsibility for what it does; you may not block it or erect CAPTCHAs for it, it scrapes/takes actions on your website on my behalf for my personal needs. This would make running a bunch of scripts on your own NUC or Pi an inalienable right while still allowing companies to fight unfair competition and plain simple DoS attacks.
- sundarurfriend 6y ago
  > It should be possible for any company to ban bots but then for a person with a account on the website to say: I am giving this tool my cloud power of attorney That's basically what API keys are, aren't they?
  - smarx007 6y ago
    No, many providers require to apply for API access, to pay for it, to be bound by T&C (some APIs require to you to cache data for max 24h) and also provide a crippled subset of actions through it. What I was saying is that none of that shall be allowed once a power of attorney has been given to a tool (obviously unless the website access is paid, then it's perfectly fine to charge for API access but then it should be offered on something like FRAND terms). Think of Chrome extensions detected and blocked by some websites, this would not be allowed under my proposal.
- kordlessagain 6y ago
  Building a distributed crowd sourced screen crawler.
  - 6y ago
bangtoyib 6y ago
nice
naringas 6y ago
and so is free (FSF-style) and open source software. but apparently a healthy economy (with a 'free' market) is more important... o̶h̶,̶ ̶a̶l̶s̶o̶ ̶n̶a̶t̶i̶o̶n̶a̶l̶ ̶s̶e̶c̶u̶r̶i̶t̶y̶ ̶a̶n̶d̶ ̶s̶o̶m̶e̶t̶h̶i̶n̶g̶ ̶a̶b̶o̶u̶t̶ ̶t̶h̶e̶ ̶c̶h̶i̶l̶d̶r̶e̶n̶.̶
jkingsbery 6y ago
The connection between web scraping and the case cited seems pretty tenuous. I can't say I know much about the definition of Computer Fraud and Abuse Act, but employment agreements commonly stipulate that the employee can only access information to the extent needed for his or her job. So defining this as unauthorized use wouldn't at all be surprising given how that term is used in practice. The article then goes on to defend web scraping which is very different because (1) the person in question accessed data manually, (2) the person in question had access to confidential information, (3) the person in question used the data in a way he must have agreed not to as part of his employment. It's hard to see how someone could connect whatever precedent this case sets to web scraping.
- perpetualpatzer 6y ago
  >The connection between web scraping and the case cited seems pretty tenuous. The link is another case, Linkedin v hiQ, that is being held pending this case and presents the same question of what counts as "accessing without authorization or exceeding authorization" under the CFAA. The dispute there is whether hiQ could scrape public LinkedIn pages. The crux of the issue is that if instructions on how to use data that someone has access to without "breaking and entering" don't count as revoking "authorization", this law doesn't cover this officer's actions (though other laws / job requirements may). If just breaking verbal or written terms of use is enough to criminalize it, that covers a whole bunch of things we'd think of as not federal crimes, like lying about your age to set up a facebook account.
dimitrios1 6y ago
The internet is not vital to democracy, and perhaps as we are learning more now, antithetical to it. So then it logically follows that web scraping is not vital to democracy.
- PestoDiRucola 6y ago
  I don't think the internet is antithetical do democracy. Social media on the other hand...
notahaterbut 6y ago
Web-scraping can do harm to a business if the scraper does not implement things like timeouts or sleeps between processes. Websites and businesses do not factor bandwidth for the rogue scraper.
- PestoDiRucola 6y ago
  You can easily block a harmful scraper like that from the website owner's side.
  - notahaterbut 6y ago
    what if the scraper uses ip rotation or serverless workers ?
  - kingdomcome50 6y ago
    You make the target. I’ll make the bot. Let’s see who wins. In all seriousness, your suggestion is woefully naive in the face of any serious entity.
- kordlessagain 6y ago
  This only applies to crawlers who are set to recurse into URLs.
slashblake 6y ago
Web scraping does for the web what TikTok does for music. It enables creativity. You can take a meme (i.e., song/web site) and change/combine it to make a new meme.
artur_makly 6y ago
We at https://VisualSitemaps.com https://VisualSitemaps.com feel the same way. Tons of our customers use us not just to archive a snapshot of the site but also to keep tabs on any visual and scope changes over time. All without any coding. 100% Automated.
kordlessagain 6y ago
Someone tell this to Bloomberg.
forgotmypw17 6y ago
Bonus, if your site is scrapable, it is easier to preserve.
- gitweb 6y ago
  With Puppeteer and Playwright, essentially every site is scrapeable. Doesn't mean it is easy, but it can be done.
  - forgotmypw17 6y ago
    It's scrapeable if you try hard, but it probably won't get archived well.
est 6y ago
Another proof why semantic Web is dead, or is it really never there? Now all we have is SPA 30MB main.min.js mess.
ChrisArchitect 6y ago
"ScrapingIsNotACrime" t-shirts caught my eye and shared the amicus brief a few days ago here https://news.ycombinator.com/item?id=25254499 https://news.ycombinator.com/item?id=25254499 Can't really fault The Markup for looking out for their own techniques/strategies
eli 6y ago
I don't particularly have a problem with scraping, but I could really do without whoever keeps trying to scrape lots of pages very quickly from hundreds of IPs and random user agents.
- vorpalhex 6y ago
  I think as a scraper, we should generally respect rate-limits. Generally the issue is that it's hard to enforce that against bad users. The few times I've written scrapers, one of the things I've done is put contact information in the user-agent so that if there is an issue, the site admins can reach out. So far that's worked out ok for me.
specialist 6y ago
Why didn't data.gov usher in a new era of transparency? I really thought Obama Admin's data.gov initiative signaled a sea change. Share your data. Show your work. Science, journalism, others, face this same crisis. Legitimacy, credibility, authenticity, accountability, etc. It's this obvious? Why are we still talking about it?
crb002 6y ago
Web scraping == 1st Amendment.
heavyset_go 6y ago
Major platforms don't feel that way, and they plan on using feature detection to make sure that unauthorized, or competitors', browsers can't access the content that they host. These are the same companies that walk all over the ADA in an attempt to stop scraping.
Drakula2k 6y ago
It's good to see that web scraping is not associated only with "hackers" and spammy websites anymore. I'm working on simplifying web scraping for developers with https://webscraping.ai https://webscraping.ai and see how important it is for almost any business or researcher.

426 points by atg_abhishek 6y ago

Clearly its time to come up with rules for this activity. Picking an apple from a tree in a public park is clearly harmless. Bringing in a combine harvester to take every scrap is not. How do we set a boundary on scraping?

booleandilemma 6y ago
They should sell scraping licenses, right? /s
slim 6y ago
a website would be like an apple tree if there was an infinite number of apples in it. so you could bring a harvester and still leave enough apples for everyone.

mikewarot 6y ago

Not only is web scraping in danger, the trend towards HTTPS makes it harder to log traffic, and manage the flow of data to/from browsers on your own computer.

steelbrain 6y ago
Can you please explain how you got to that conclusion?
- kupopuffs 6y ago
  Obviously makes it harder for others to log your traffic
- mschuster91 6y ago
  Not OP, but it's very hard to intercept and monitor the traffic of an app that uses code signing, obfuscation and certificate pinning properly. What used to be a five minute "put tcpdump on the access point / router to work" job pre-HTTPS everywhere now is a many days worth job of messing around - out of reach for all but the really dedicated, and as a result it is very hard for an user to have actual visibility over where their data flows.
  - captn3m0 6y ago
    I'd say this is a consequence of making our phones opaque rather than HTTPS-everywhere. Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Doing the same on our mobile devices is much more tougher, because we've let feature phones become walled gardens.
    - mschuster91 6y ago
      > Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. As for the walled garden part: I agree with the general sentiment, but on the other hand I also see the lengths malware authors go to gather data from people. There really is no one-fits-all solution here, because anything that allows the user to intercept and monitor SSL communication can automatically be used by an attacker! :(
      - captn3m0 6y ago
        > Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. Are there desktop apps that behave this way? Atleast in my experience - I haven't come across anything like this on Linux.
        mschuster91 6y ago
        Dropbox does: https://knowledge.broadcom.com/external/article/169397/dropbox-ssl-interception-and-issues-when.html https://knowledge.broadcom.com/external/article/169397/dropb... And I seriously hope cert pinning gets adopted by more applications.
        dylz 6y ago
        Many desktop apps behave this way, yes. Anything from a major tech co should at least

pessimizer 6y ago

It's pretty typical and sad how the US is prosecuting a dirty cop who looks up confidential data in exchange for money: on behalf of the system itself rather than on behalf of the victims' privacy, any limitations on the actions of law enforcement officers in general, or public corruption. In libertarianism/liberalism's final form, the only two kinds of laws left are lèse majesté and violations of terms of service.

1vuio0pswjnm7 6y ago

"In both of those instances, the pages and data scraped are publicly available on the internet-no hacking necessary-but sites involved could easily change the fine print on their terms of service to label the aggregation of that information "unauthorized."" They could. But if they filed a claim under the CFAA without ever sending a cease and desist letter to the alleged intruder, I think the claim would be dismissed. Is simply changing TOS enough for a CFAA claim to have a reasonable chance of success. I could be wrong, but I believe for every CFAA claim we have seen so far based on "scraping", there was some notice to stop directed specifically at the respondent. If someone thinks just changing TOS (public notice) is enough for a CFAA claim to survive a Motion to Dismiss and there is no need to also contact the alleged offender asking them to stop, then let's hear about the precedent that supports that idea. I do not think there is any such precedent, but I could be wrong.

amelius 6y ago
So robots.txt has no meaning?
- fogihujy 6y ago
  I think the main issue is with scrapers that don't adhere to robots.txt/X-robot headers.
  - 9HZZRfNlpR 6y ago
    Google doesn't either, they just don't index the paths written in robots.txt but bots still hit them.
- krageon 6y ago
  No, because robots.txt is a friendly request and not a demand. It can never be a demand.
- err4nt 6y ago
  It's not binding, but it's a hint they may listen to where you can say things like if there's a certain URL structure where your server would keep serving content infinitely and just waste the crawlers time and resources, you can hint to them they maybe they should avoid certain URLs, but hey if they wanna waste their resources crawling that anyway be my guest!

wraptile 6y ago

This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and fingerprints to fight the nature of the internet. Finally everyone wants benefits of public data (search engine indexing etc) but don't really want to give anything back to the ecosystem. It's just pure greed and law, our society and government shouldn't aid it in any way, shape or form.

IfOnlyYouKnew 6y ago
You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do. Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them access becomes legally binding. In the offline world, we derive great benefit from following the second model. Indeed, if only measures that successfully prevent people from entering your house without permissions were to count, you wouldn't need laws in the first place! You would, however, need a bunker. Which is quite a bit more expansive than a functioning legal system. In that offline world, we have created all sorts of additional rules to balance rights for specific situations, and we rely on a canon of expectations that say, for example, that it's usually not ok to enter a private house, but you don't need explicit permission to enter a supermarket. These are still developing for the online world, and your idea of the "ecosystem" hints at that. But you're just taking from those contradictory ideas above to arrive at the outcome you intuitively feel is "just": a bit of might-is-right when it comes to "whatever is online is fair game", followed by principled ideas of rights and obligations when websites try to defend themselves in that jungle of yours. What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road.
- 6y ago

charliebrownau 6y ago

Ever considered Democracy doesnt work Look at the last 5-25-55-255 years

mnmkng 6y ago

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as well. Instead of hiring 100 interns to collect some data manually, you hire one programmer to automate that data collection process. I think that the ability to automate tasks on the internet is absolutely crucial to further development of our society and limiting it in any way will be detrimental to the world as a whole. The amount of information these days is so vast, that no human labor force could possibly analyze that information and use it to drive our progress. Disclaimer: We run a web scraping platform (https://apify.com https://apify.com)

sokoloff 6y ago
While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”. I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s going to be frowned upon.
- simplecto 6y ago
  This is orthogonal to the purpose of scraping. If I run a search engine which potentially links real human eyes back to you, then should I pay the "ad" toll as well? I don't believe so. I do believe there is an agreeable middle-ground, but Google walked away from that conversation years ago.
- krageon 6y ago
  Advertisements are harmful to every layer of society, mostly because they prey on you to instill desires that you probably wouldn't have had on your own (it being the case that this is their entire added value proposition). They should not be tolerated, and the fact that they ever were is a travesty. That being the case, any argument founded on "but advertisements" does not hold water.

bzb6 6y ago

You should also have called it a “human right” for improved virtue signalling.

norwegianwhale 6y ago
"Human right" is one of those sacred terms that are supposed to mean so much but mean nothing. "Human rights" are just superstitious conventions about what's good/bad. "No one can deprive you of housing" oh look I am gonna set up my tent here in this warm shop and claim human rights protection.

wslh 6y ago

The corollary would be that robots.txt exclusion for certain web crawlers should be removed.

tobyhinloopen 6y ago

We use scrapers to wrap and combine 3rd party tools and increase the usability and decrease amount of work required to get something done. Sometimes to do some task at a 3rd party tool, it requires many clicks and page loads. My scraper wrapping the tool automates most of the task and only keeps the manual task, reducing the amount of required clicks (and time) required to get the task done. I use a mix of headless browsers (for JS-heavy apps) and raw API calls. Sometimes I even use the browser to login, trigger a single sample APi call, extract all headers and content, close the browser, and re-run that api call directly using a HTTP lib. The request body obviously gets modified on the user’s requirements. We’re bypassing the slow login process as long as the session is valid. We’re also sharing sessions/logins/accounts this way without exposing credentials to users. It can also be done to bypass the only-one-session-per-user systems. This is done with permission from 3rd parties. They’re fine with it, they just didn’t want to provide a proper API or let us bypass the only-one-session rule because it requires code changes they’re not willing to make only for us. Sometimes the tool breaks when the html Or api changes but it usually only takes a few minutes to modify the code to fix it.

agustif 6y ago

Last weekend I made a little playwright script to checkout stock item status of decathlon products. The bike -Rodillo- Has been out of stock for several weeks. While coding it it was briefly on stock but I was too slow with my card input hahaha and so I neeed to now automate the buy part as soon as it's available, or make that run every 5min and mail/notify me if available... Btw this was super easy, playwright as a puppeteer successor really rocks, another thing hard to hate from MS like TS or VS Code PS: I brought down script execcution from 7 secocnds to 2 seocnds, by not loading any unneeded stuff css images external js fonts. I felt like a good scraper citizen doing so, it was just two linnes to block by request type on the network level, and it worked like a charm to speed up the process.

bangtoyib 6y ago
nice

crispyambulance 6y ago

I use web scraping on my employer's Oracle EBS instance because coordinating with the battle-axe personalities that run ERP's is such a pain in the ass I would rather have a root-canal. With Web Scraping, I don't even have to talk to them.

appleflaxen 6y ago
For others: it looks like EBS in this context means "oracle e-business suite" and ERP means "enterprise resource planning" (maybe everyone else knows these, but just in case) https://en.wikipedia.org/wiki/Oracle_Applications#Oracle_E-Business_Suite https://en.wikipedia.org/wiki/Oracle_Applications#Oracle_E-B... https://en.wikipedia.org/wiki/Enterprise_resource_planning https://en.wikipedia.org/wiki/Enterprise_resource_planning
- crispyambulance 6y ago
  Yes, that's right, and those of us that don't have to deal with these dreadful systems are better off :-)

_wldu 6y ago

If you post a file to the WWW (World Wide Web) do not be shocked that the whole world can read and copy it. That's how it works. You're welcome.

SergeAx 6y ago

Okay, great point, but who will pay for that? Serving content costs money: traffic, extra load on CPU and storages, and now on serverless functions too. One spider may be equivalent of hundreds of real users. And it spoils caches too.

emiliobumachar 6y ago
Crackpot regulation proposal: scraping is legal if you favor mirror sites and afterwards create a mirror yourself and maintain it to some arbitrary standard.

werber 6y ago

I was trying to write a scraping script for a non profit to automate the work they were giving to law students to do manually earlier this year but the court system had agreed captchas that made it impossible. In my (legally uniformed) mind it felt like they were actively undermining people of lesser means by not allowing a simple script to run and essentially mandating an expensive legal team to do menial work

smarx007 6y ago

Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally). Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on the website to say: I am giving this tool my cloud power of attorney (best if signed through a digital govt id system, e.g. https://en.wikipedia.org/wiki/BankID https://en.wikipedia.org/wiki/BankID) and I take responsibility for what it does; you may not block it or erect CAPTCHAs for it, it scrapes/takes actions on your website on my behalf for my personal needs. This would make running a bunch of scripts on your own NUC or Pi an inalienable right while still allowing companies to fight unfair competition and plain simple DoS attacks.

sundarurfriend 6y ago
> It should be possible for any company to ban bots but then for a person with a account on the website to say: I am giving this tool my cloud power of attorney That's basically what API keys are, aren't they?
- smarx007 6y ago
  No, many providers require to apply for API access, to pay for it, to be bound by T&C (some APIs require to you to cache data for max 24h) and also provide a crippled subset of actions through it. What I was saying is that none of that shall be allowed once a power of attorney has been given to a tool (obviously unless the website access is paid, then it's perfectly fine to charge for API access but then it should be offered on something like FRAND terms). Think of Chrome extensions detected and blocked by some websites, this would not be allowed under my proposal.
kordlessagain 6y ago
Building a distributed crowd sourced screen crawler.
- 6y ago

bangtoyib 6y ago

nice

naringas 6y ago

and so is free (FSF-style) and open source software. but apparently a healthy economy (with a 'free' market) is more important... o̶h̶,̶ ̶a̶l̶s̶o̶ ̶n̶a̶t̶i̶o̶n̶a̶l̶ ̶s̶e̶c̶u̶r̶i̶t̶y̶ ̶a̶n̶d̶ ̶s̶o̶m̶e̶t̶h̶i̶n̶g̶ ̶a̶b̶o̶u̶t̶ ̶t̶h̶e̶ ̶c̶h̶i̶l̶d̶r̶e̶n̶.̶

jkingsbery 6y ago

The connection between web scraping and the case cited seems pretty tenuous. I can't say I know much about the definition of Computer Fraud and Abuse Act, but employment agreements commonly stipulate that the employee can only access information to the extent needed for his or her job. So defining this as unauthorized use wouldn't at all be surprising given how that term is used in practice. The article then goes on to defend web scraping which is very different because (1) the person in question accessed data manually, (2) the person in question had access to confidential information, (3) the person in question used the data in a way he must have agreed not to as part of his employment. It's hard to see how someone could connect whatever precedent this case sets to web scraping.

perpetualpatzer 6y ago
>The connection between web scraping and the case cited seems pretty tenuous. The link is another case, Linkedin v hiQ, that is being held pending this case and presents the same question of what counts as "accessing without authorization or exceeding authorization" under the CFAA. The dispute there is whether hiQ could scrape public LinkedIn pages. The crux of the issue is that if instructions on how to use data that someone has access to without "breaking and entering" don't count as revoking "authorization", this law doesn't cover this officer's actions (though other laws / job requirements may). If just breaking verbal or written terms of use is enough to criminalize it, that covers a whole bunch of things we'd think of as not federal crimes, like lying about your age to set up a facebook account.

dimitrios1 6y ago

The internet is not vital to democracy, and perhaps as we are learning more now, antithetical to it. So then it logically follows that web scraping is not vital to democracy.

PestoDiRucola 6y ago
I don't think the internet is antithetical do democracy. Social media on the other hand...

notahaterbut 6y ago

Web-scraping can do harm to a business if the scraper does not implement things like timeouts or sleeps between processes. Websites and businesses do not factor bandwidth for the rogue scraper.

PestoDiRucola 6y ago
You can easily block a harmful scraper like that from the website owner's side.
- notahaterbut 6y ago
  what if the scraper uses ip rotation or serverless workers ?
- kingdomcome50 6y ago
  You make the target. I’ll make the bot. Let’s see who wins. In all seriousness, your suggestion is woefully naive in the face of any serious entity.
kordlessagain 6y ago
This only applies to crawlers who are set to recurse into URLs.

slashblake 6y ago

Web scraping does for the web what TikTok does for music. It enables creativity. You can take a meme (i.e., song/web site) and change/combine it to make a new meme.

artur_makly 6y ago

We at https://VisualSitemaps.com https://VisualSitemaps.com feel the same way. Tons of our customers use us not just to archive a snapshot of the site but also to keep tabs on any visual and scope changes over time. All without any coding. 100% Automated.

kordlessagain 6y ago

Someone tell this to Bloomberg.

forgotmypw17 6y ago

Bonus, if your site is scrapable, it is easier to preserve.

gitweb 6y ago
With Puppeteer and Playwright, essentially every site is scrapeable. Doesn't mean it is easy, but it can be done.
- forgotmypw17 6y ago
  It's scrapeable if you try hard, but it probably won't get archived well.

est 6y ago

Another proof why semantic Web is dead, or is it really never there? Now all we have is SPA 30MB main.min.js mess.

ChrisArchitect 6y ago

"ScrapingIsNotACrime" t-shirts caught my eye and shared the amicus brief a few days ago here https://news.ycombinator.com/item?id=25254499 https://news.ycombinator.com/item?id=25254499 Can't really fault The Markup for looking out for their own techniques/strategies

eli 6y ago

I don't particularly have a problem with scraping, but I could really do without whoever keeps trying to scrape lots of pages very quickly from hundreds of IPs and random user agents.

vorpalhex 6y ago
I think as a scraper, we should generally respect rate-limits. Generally the issue is that it's hard to enforce that against bad users. The few times I've written scrapers, one of the things I've done is put contact information in the user-agent so that if there is an issue, the site admins can reach out. So far that's worked out ok for me.

specialist 6y ago

Why didn't data.gov usher in a new era of transparency? I really thought Obama Admin's data.gov initiative signaled a sea change. Share your data. Show your work. Science, journalism, others, face this same crisis. Legitimacy, credibility, authenticity, accountability, etc. It's this obvious? Why are we still talking about it?

crb002 6y ago

Web scraping == 1st Amendment.

heavyset_go 6y ago

Major platforms don't feel that way, and they plan on using feature detection to make sure that unauthorized, or competitors', browsers can't access the content that they host. These are the same companies that walk all over the ADA in an attempt to stop scraping.

Drakula2k 6y ago

It's good to see that web scraping is not associated only with "hackers" and spammy websites anymore. I'm working on simplifying web scraping for developers with https://webscraping.ai https://webscraping.ai and see how important it is for almost any business or researcher.