12 ms·

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that come

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

brink 7y ago
I think you mean contradiction, not paradox.
- ramblerman 7y ago
  Not really. Both are correct, and paradox fits better in OPs context, paradox: a statement or proposition that seems self-contradictory or absurd but in reality expresses a possible truth.
  - brink 7y ago
    A paradox is something that initially looks like a contradiction, but after closer inspection, is not.
- foxx-boxx 7y ago
  Probably he or she was black listed on Facebook
yodon 7y ago
> People want their data to be public People don't want their data to be public. People want other people's data to be public. One's own data everyone thinks should be private and tightly controlled. This applies to people and businesses equally.
- ppseafield 7y ago
  I think the GP in this context means, e.g. LinkedIn wants their information to be public in the cases where it benefits them as a business. But then they want it to not be public when it doesn't benefit them. There is no such thing as "public information, except ..." - information is public, or it's not. If none of LinkedIn's data was public, they would have a much harder time getting people to sign up, and having as many users signed up as possible is part of their business model. From a copyright perspective (since that's what LinkedIn's lawyers claimed): imagine if a newspaper sued another newspaper, saying that - not just the content of its paper - the information in the newspaper was copyrighted and could not be accessed by "unauthorized" third party companies. Either you print it, or you don't!
- derefr 7y ago
  In this case, LinkedIn users kind of do want their “public profiles” to be public. They’re online CVs; by definition, if you make one, your goal is to get it into the hands of anyone who asks for it! LinkedIn, likewise, has built its business model on an implicit contract with its users that it’s going to show their CV to anyone who asks for it. I think LinkedIn users would be surprised that LinkedIn doesn’t let bots read (scrape) their public CV. A CV (individually, rather than in aggregate) is ultimately useful for only one thing: marketing the CV’s author’s skills. Why wouldn’t I want my marketable skills scraped into some private “talent matchmaking” agency’s databases, such that someone could find me—and hire me—when I show up as a result of some fancy OLAP query they paid that agency to run on their scraped data? It’s more roundabout than them just finding my CV on LinkedIn, but I’m still glad they found it!
  - barrkel 7y ago
    Not so sure about that. Messages on LinkedIn are mediated in a single place, and you have a measure of control over how your profile shows up in searches. If your CV is scraped, you could end up anywhere and now you're getting recruiter spam from all over when you're not interested.
    - derefr 7y ago
      > If your CV is scraped, you could end up anywhere and now you're getting recruiter spam from all over when you're not interested. I would point out that this is still possible (even probable!) without any bots being involved at all. Back before CVs were online, humans working for recruitment agencies would “scrape” information from local, physical job boards by hand into their company’s databases (where “database” here could just mean a filing cabinet.) IMHO, the real solution to that is a spam filter (or an “agent”, in the old world.) Just because a lot of people want to talk to you, and most of them aren’t very interesting, doesn’t mean they need to be prevented from accessing you—they just need to be prioritized by interesting-ness, which is something you can do yourself, or hire a service to do for you.
  - Dayshine 7y ago
    >I think LinkedIn users would be surprised that LinkedIn doesn’t let bots read (scrape) their public CV. LinkedIn is really really clear that: - They won't share you information with 3rd parties - You're not allowed to use information on LinkedIn for commercial purposes without their permission - Other users can view your personal data So, why would I expect random third party companies to be able to scrape and sell my personal information? My personal information is there for the individual use of others, and for authorised use by recruiters (who are vetted/managed by LinkedIn). I've chased down the convention spam mail I get using my GDPR rights, and surprise surprise, they got my details by scraping LinkedIn. That is absolutely not expected nor acceptable use of my data...
Wowfunhappy 7y ago
I want web scraping to be legal—but, is it really contradictory to say "I want this data to be accessible to real humans only"? Any person can post on Hacker News. However, if someone made a bot to post to Hacker News, I think most of us would be pretty upset.
- alaaf 7y ago
  Isn’t this different from read access? Apart from server resources, downloading content doesn’t effect a web site as much as posting on it.
- nathanlied 7y ago
  Isn't that a different paradigm, though? A posting bot set loose on a forum/platform will (normally) visibly degrade service in a much more visible and impactful way than a scraping bot. And in either case, writing (and running!) a bot that posts on HN is not illegal behaviour in itself.
- danShumway 7y ago
  I dunno. I'm on Hackernews to see interesting articles and read interesting conversation. If a bot can post interesting articles and make interesting conversation, I'm not sure I care that it's a bot. And if a human can't do those things, I'm not sure I care whether or not they're 'real'. https://xkcd.com/810/ https://xkcd.com/810/ That focus on "we don't care if you self-promote, we don't care why you're here, we just want you to be a good citizen" is part of why I like HN. It's not completely black and white, but in general I believe that users online have the Right to Delegate[0]. That right should only be legally taken away if there's a really, unbelievably compelling social justification for doing so. I am pretty skeptical that banning web scraping has that kind of justification. [0]: https://anewdigitalmanifesto.com/#right-to-delegate https://anewdigitalmanifesto.com/#right-to-delegate
- manaskarekar 7y ago
  Makes me wonder, how many users on HN are actually very convincing bots?
- stubish 7y ago
Nitramp 7y ago
"public" is not the right concept here I think. E.g. imagine a composer conducting a public airing of some work of music (e.g. on some festival). That you were able to hear the music in public doesn't mean the {composer,artists,...} give up their copyrights.
- mokus 7y ago
  I think copyright as you mention here is the right concept, or at least a lot closer. In particular, the limits on copyright. If someone is reciting a list of facts in public, they can’t expect people not to record those facts, because copyright doesn’t apply to that. Reciting the list in public using computers shouldn’t change that.
- davorak 7y ago
  > public doesn't mean the {composer,artists,...} give up their copyrights. Copyright law protects the original work of the composer and artists in your example. User profile data on Linkedin is not Linkedin original work. Additionally user profiles are mostly made up of facts, which are not copyrightable.
tyingq 7y ago
There's plenty of grey here. For example, scrapers that try to check people in for flights to get better seats. Some that tried to charge for that. That creates problems, where some customers benefit at the expense of others, high load on a "locking type" piece of code, etc. Similar for ticket sales for concerts, and probably other spaces. There are also companies that provide added value by compiling and correlating "public info" in a useful way that creates value. If Google let me scrape their search and remove ads, it would be popular, but is it "legal"? Or maybe Google Maps?
- yuters 7y ago
  Another problem with web scraping is a B2B website offering services for B2C companies to better reach consumers. It can be a tricky thing to do without basically giving your clients list to all your competitors.
- LameRubberDucky 7y ago
  I would think, and of course could be wrong, it would be as legal as Google scraping all of the web sites that they do in order to create their search engine in the first place. In particular, Google provides cached versions of web pages. That's pretty hardcore scraping.
  - deleted 7y ago
    [deleted]
  - tyingq 7y ago
    I put "legal" in quotes because I assume it's not criminal. It would, though, likely result in some pretty quick civil action.
TrackerFF 7y ago
I can understand why some do not want scrapers - increased traffic (with practically zero benefits to the owners) is one obvious reason. (Some people will then say "But why not just offer APIs", but that's a lot of extra work and maintenance). It's like with instagram and other social media platforms. The content creators put in the hard work, while the leeches are stealing content for their own benefit, giving zero credits to the original content creators.
- xnyan 7y ago
  There’s a very effective way out, don’t your data on the public web.
  - kerkeslager 7y ago
    [decided to delete because I misunderstood the context]
    - deleted 7y ago
      [deleted]
    - ppseafield 7y ago
      Let's not conflate a) a person's personal data, and b) a business's dataset. The GP and the article are clearly referring to the latter. Preventing web scraping won't protect users from businesses collecting their data.
- buboard 7y ago
  Instagram is not the content creator tho
  - TrackerFF 7y ago
    What I'm trying to say is: For every popular content creator on IG, there are tens and hundreds of (more or less automatic) content curators that do nothing more than scrape content with lots of likes, and re-post on their own channels. Then when they get sufficient followers, they make money through paid product placements, account flipping, pay-to-play sharing, and what not. More often than not, there's no linking to the original pages / creators.
sillyquiet 7y ago
Yeah, I have nothing at all against scraping per se, it's more about the huge bot traffic the commercial scrapers generate, which would ALSO be fine, but 1. it can be hard to tell scrapers from malicious DDOS bots sometimes and 2.) the person being scraped literally pays for that scraping traffic.
- a1369209993 7y ago
  So, apply DOS/DDOS mitigations? You need to have those anyway, so you might as well use them for this too.
  - sillyquiet 7y ago
    Yeah, but where do you draw the line between 'oh sorry we dropped your request because of rate limiting' (or whatever mitigation strategy) and 'oh, we dropped your request because u scraping us bro' legally? IANAL, but this lawsuit seems to indicated that putting barriers in front of scraping attempts is a no-no.
    - a1369209993 7y ago
      "You issued more than $X requests per $Y seconds. We don't care whether you're scraping or not; try again later." for any values of X and Y.
DougWebb 7y ago
One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet to scrape all of the account information for a particular county. My client doesn't care so much about the data; it's the server load that's a problem. Normal activity for this site is a few dozen account searches per minute, but when the botnet gets through our blockade it sends hundreds of search requests per second, overwhelmimg the site. The operator of the botnet has NEVER tried to contact my client to ask for an efficient api to access the data, which they'd probably provide for a minimal fee. Data hosting isn't free, even if the data is.
- gbasin 7y ago
  Can you just put it behind a CDN that protects you?
- vorpalhex 7y ago
  Wouldn't the solution be to offer a streamlined download (maybe even as a torrent if you're worried about bandwidth) of all the data then?
  - TremendousJudge 7y ago
    probably, but as GP said, "The operator of the botnet has NEVER tried to contact my client to ask for an efficient api to access the data" Some people just don't care for the commons
  - Symbiote 7y ago
    I work on a fully open data repository. The website has the API linked in 3 places, so when I find inappropriate scraping I block it with "HTTP 420 ... see <API link> or contact <email>". Some people probably switch to using the API, but no-one has ever contacted us. They either give up, or run their scraper on a different computer -- I've seen the same scraper move between university computers, departments, then (in the evening) to a consumer broadband IP.
kerkeslager 7y ago
> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want is, put simply, organizational transparency, personal privacy. I believe humans have a right to privacy, but I don't believe organizations have rights, period, and I believe radical transparency within an organization prevents organizations from trampling the rights of individuals. Organizations in this case include corporations, governments, and nonprofits.
- whatshisface 7y ago
  What if the organization is one person in an LLC? Do they get rights? If so then a big company can hire a bunch of little LLCs to act as rights-having proxies for any task that requires them.
  - kerkeslager 7y ago
    I'm going to assume you're asking in good faith and try to address the confusion here. The human does get rights, the organization doesn't. In some cases, believing that humans have rights and believing that organizations have rights might lead one to the same action. In those cases, I'd take the action. I wouldn't want to violate a human's rights out of some vindictive dislike of organizations: that's not the point. The point is that I'd take that action because I believe in human rights, not because I believe that the organization has rights. And with organizational transparency: the entire point of organizational transparency is to protect human rights. In cases where organizational transparency would trample human rights to privacy, I would go with human rights every time. Violating the privacy of humans to achieve organizational transparency would defeat the entire purpose.
    - whatshisface 7y ago
      Let's say that individual humans have the right to keep secrets. Let's also say that they have the right to keep secrets with their associates, and to tell them to who they please. Now, doesn't that make it legal for a group of people to keep secrets about you? What about selling them? I just don't see what doing away with the legal fiction of corporate personage would do about Facebook.
levosmetalo 7y ago
Sure. My data is still my data, and if I publish it on my platform for free, that still shouldn't automatically give you the right to copy the data and provide on your platform. It's basically the same as a TV broadcasting a film for free, and then going after you legally if you recorded that film and uploaded it to your website.
- sjg007 7y ago
  It's not the same thing at all. The film is broadcast without re-transmission rights.
  - levosmetalo 7y ago
    How is that different? Where on my website I gave anyone "re-transmission" or "re-publishing" rights?
    - sjg007 7y ago
      It's different for broadcast films and tv https://en.wikipedia.org/wiki/Retransmission_consent https://en.wikipedia.org/wiki/Retransmission_consent
- jayd16 7y ago
  This is a bad analogy as scraped but copyrighted works are still protected by copyright. Whats in question is whether you should have exclusive rights to information you have shared but did not copyright. Seems people also think this noncopyrighted data should also be protected even if its munged and added to an original work by the scraper.
- hamiltont 7y ago
  This does not legalize theft, it says sites cannot respond to suspected scrapers differently than they respond to non-suspected scrapers. You can still rate limit, as long as you do it universally to all site users Copyright law is unchanged. If someone scrapes your blog and then re-uses your posts on their own blog, you still have possible copyright infringement claim
soup10 7y ago
Billions are being made collecting data and content for free and publishing it along with targeted advertisements. Web-scrapers can collect that aggregated data and redistribute it or create competing services. This is terrifying to certain tech giants and threatens their moat's and lock-in/network advantage. They will lobby hard against it with PR campaigns citing everything from security, privacy, copyrights and all sorts of other exaggerated bogeymen. At the end of the day its mostly about preserving their monopolies though.
AgloeDreams 7y ago
People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it. By People you mean business or people people? Because I don't think people want everything to be public, many in fact use various networks to avoid oversharing and even then many people don't want their old bosses or exes looking at their profiles, there just don't exist tools to limit access that granularly.
jakelazaroff 7y ago
I want to be able to use LinkedIn to network with colleagues and people in my industry. If someone wants to scrape my profile to make a report on industry trends, I’m fine with it. What I don’t want is hiQ vacuuming up my data so they can snitch to my employer if they think I’m job hunting. How is this a paradox? Tech — the web in particular — is supposed to be an equalizing force, but HiQ is clearly trying to give my employer more power over me. We are an industry that prides itself on solving difficult problems — how is our response here to just throw up our hands and say “it’s all or nothing”?
- gph 7y ago
  >Tech — the web in particular — is supposed to be an equalizing force Don't take the Google PR so seriously, the tech industry wants to make money like every other industry.
- Nasrudith 7y ago
  We haven't solved that in the same way we haven't "solved" encryption not having a magical good people only door despite spook tantrums. There fundamentally isn't a possible mechanism and really wanting it doesn't change that. It is a result of equality - not of outcome but rules. Open for everyone but those whose applications you don't like isn't open. On a technical level trying to prevent it is like the "evil bit" as a solution to malware.
  - jakelazaroff 7y ago
    Of course there are possible mechanisms. There are heuristics to detect bots. The whole reason for this lawsuit is that LinkedIn blocked hiQ from scraping their website. I'm also not necessarily talking about a technical defense against unwanted scraping. Write a law makes it illegal to do something like "scraping personally identifiable information and storing or presenting it non–anonymized", and prosecute companies who break it. I'm sure there are loopholes in that particular example, but the point is we can absolutely add shades of gray here. > Open for everyone but those whose applications you don't like isn't open. Openness should be a means, not an end. If we make something "not open" but it prevents 95% of undesirable uses and only 5% of desirable ones, is that not a tradeoff worth discussing?
stanferder 7y ago
I don't think the direction would ever be clear, even if the legality were clearly established. The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field. Companies want to provide some information to some people; but providing all information to all people is analogous to allowing customers to make a meal of free food samples, on a recurring basis.
- mindcrime 7y ago
  The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field. It already is. There are entire companies, like Distil Networks, who exist solely to protect companies from bots/scrapers/etc. Actually, looks like Distil got acquired and are now part of Imperva, but anyway, the idea is the same. This is definitely an existing field. https://www.imperva.com/products/bot-management/ https://www.imperva.com/products/bot-management/ Disclosure: former Distil employee, but I have no financial stake in this discussion, and have mixed feelings about scraping. Clearly it can be beneficial in some situations, but when I think about having to pay exorbitant prices to scalpers for tickets to an event, because they used a bot to buy up all the tickets, that is less appealing.
Nasrudith 7y ago
It sounds like the same old issue - lack of understanding of the fundamentals underlying technologies. Otherwise they would realize what they demand is contradictory and incoherent like demanding to be both viewable by all and not viewable. DRM is one fundamental example of it.
Tactic 7y ago
The issue here for some, if not many, is a matter of scale. It is one thing if an end-user, whom I am trying to service, comes to my site and gets my publicly available data. Maybe I monetize with ads, maybe not. It doesn't matter, that is the audience I am trying to service, regardless of size. But when you scrape it my load goes up dramatically. A load I have to pay for. It is analogous to the privacy debates going on with one said saying "hey, don't track everywhere I go and tag me with facial recognition" and the other side saying "hey, you are in public and people can see you." The issue is not complete privacy, but one of scale. And of intent. I believe society is soon going to have to come to grips with the scale of things and legislate what are acceptable scales of action as it seems to be becoming a large issue in a growing number are areas.
- orf 7y ago
  Provide an API for public data to reduce the costs associated with rendering a full blown page, and deliver just the information needed.
  - Tactic 7y ago
    Entirely feasible. Also reasonable for you to pay me for the service as it is taking my development efforts to meet your business model. The advantage to you is you have a defined interface that I won't prevent.
    - orf 7y ago
      I guess you missed the comment I was replying to: it may cost you more money, in bandwidth and per page resources, to not provide an API than it does for you to provide one. So no, I won’t pay you for the privilege of you saving money.
    - oauea 7y ago
      No, I'll happily just scrape your site instead. But if you'd rather not have that happen, provide an API.
  - briandear 7y ago
Psyladine 7y ago
People love when information is free, but as soon as monetization appears they see it like taking food from their plate.
cirno 7y ago
> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it Can you really not imagine a world where a person accidentally or in poor judgment uploads something private to their own site (their real name, home address, credit card#, or any piece of highly damaging information that could cost them their careers) and wishes for it to be removed? (but can't because many of these scraping sites never respond to takedown requests) People make mistakes and post things they shouldn't. A mistake from someone many years in the past that they've made amends for shouldn't haunt someone for the rest of their lives. But it does when we decide that every single line of text ever uttered online must be preserved and easily accessible by anyone for all eternity. Blocking scrapers is an arms-race escalation because these sites refuse to remove content, and it's used as a tool for character assassination by bad actors. It's a proactive defense.
make3 7y ago
they want there data to be public for a specific use, I feel like that's pretty easy to understand. LinkedIn: they want their info to be public -> to get jobs

brink 7y ago

I think you mean contradiction, not paradox.

ramblerman 7y ago
Not really. Both are correct, and paradox fits better in OPs context, paradox: a statement or proposition that seems self-contradictory or absurd but in reality expresses a possible truth.
- brink 7y ago
  A paradox is something that initially looks like a contradiction, but after closer inspection, is not.
foxx-boxx 7y ago
Probably he or she was black listed on Facebook

yodon 7y ago

> People want their data to be public People don't want their data to be public. People want other people's data to be public. One's own data everyone thinks should be private and tightly controlled. This applies to people and businesses equally.

ppseafield 7y ago
I think the GP in this context means, e.g. LinkedIn wants their information to be public in the cases where it benefits them as a business. But then they want it to not be public when it doesn't benefit them. There is no such thing as "public information, except ..." - information is public, or it's not. If none of LinkedIn's data was public, they would have a much harder time getting people to sign up, and having as many users signed up as possible is part of their business model. From a copyright perspective (since that's what LinkedIn's lawyers claimed): imagine if a newspaper sued another newspaper, saying that - not just the content of its paper - the information in the newspaper was copyrighted and could not be accessed by "unauthorized" third party companies. Either you print it, or you don't!
derefr 7y ago
In this case, LinkedIn users kind of do want their “public profiles” to be public. They’re online CVs; by definition, if you make one, your goal is to get it into the hands of anyone who asks for it! LinkedIn, likewise, has built its business model on an implicit contract with its users that it’s going to show their CV to anyone who asks for it. I think LinkedIn users would be surprised that LinkedIn doesn’t let bots read (scrape) their public CV. A CV (individually, rather than in aggregate) is ultimately useful for only one thing: marketing the CV’s author’s skills. Why wouldn’t I want my marketable skills scraped into some private “talent matchmaking” agency’s databases, such that someone could find me—and hire me—when I show up as a result of some fancy OLAP query they paid that agency to run on their scraped data? It’s more roundabout than them just finding my CV on LinkedIn, but I’m still glad they found it!
- barrkel 7y ago
  Not so sure about that. Messages on LinkedIn are mediated in a single place, and you have a measure of control over how your profile shows up in searches. If your CV is scraped, you could end up anywhere and now you're getting recruiter spam from all over when you're not interested.
  - derefr 7y ago
    > If your CV is scraped, you could end up anywhere and now you're getting recruiter spam from all over when you're not interested. I would point out that this is still possible (even probable!) without any bots being involved at all. Back before CVs were online, humans working for recruitment agencies would “scrape” information from local, physical job boards by hand into their company’s databases (where “database” here could just mean a filing cabinet.) IMHO, the real solution to that is a spam filter (or an “agent”, in the old world.) Just because a lot of people want to talk to you, and most of them aren’t very interesting, doesn’t mean they need to be prevented from accessing you—they just need to be prioritized by interesting-ness, which is something you can do yourself, or hire a service to do for you.
- Dayshine 7y ago
  >I think LinkedIn users would be surprised that LinkedIn doesn’t let bots read (scrape) their public CV. LinkedIn is really really clear that: - They won't share you information with 3rd parties - You're not allowed to use information on LinkedIn for commercial purposes without their permission - Other users can view your personal data So, why would I expect random third party companies to be able to scrape and sell my personal information? My personal information is there for the individual use of others, and for authorised use by recruiters (who are vetted/managed by LinkedIn). I've chased down the convention spam mail I get using my GDPR rights, and surprise surprise, they got my details by scraping LinkedIn. That is absolutely not expected nor acceptable use of my data...

Wowfunhappy 7y ago

I want web scraping to be legal—but, is it really contradictory to say "I want this data to be accessible to real humans only"? Any person can post on Hacker News. However, if someone made a bot to post to Hacker News, I think most of us would be pretty upset.

alaaf 7y ago
Isn’t this different from read access? Apart from server resources, downloading content doesn’t effect a web site as much as posting on it.
nathanlied 7y ago
Isn't that a different paradigm, though? A posting bot set loose on a forum/platform will (normally) visibly degrade service in a much more visible and impactful way than a scraping bot. And in either case, writing (and running!) a bot that posts on HN is not illegal behaviour in itself.
danShumway 7y ago
I dunno. I'm on Hackernews to see interesting articles and read interesting conversation. If a bot can post interesting articles and make interesting conversation, I'm not sure I care that it's a bot. And if a human can't do those things, I'm not sure I care whether or not they're 'real'. https://xkcd.com/810/ https://xkcd.com/810/ That focus on "we don't care if you self-promote, we don't care why you're here, we just want you to be a good citizen" is part of why I like HN. It's not completely black and white, but in general I believe that users online have the Right to Delegate[0]. That right should only be legally taken away if there's a really, unbelievably compelling social justification for doing so. I am pretty skeptical that banning web scraping has that kind of justification. [0]: https://anewdigitalmanifesto.com/#right-to-delegate https://anewdigitalmanifesto.com/#right-to-delegate
manaskarekar 7y ago
Makes me wonder, how many users on HN are actually very convincing bots?
stubish 7y ago

Nitramp 7y ago

"public" is not the right concept here I think. E.g. imagine a composer conducting a public airing of some work of music (e.g. on some festival). That you were able to hear the music in public doesn't mean the {composer,artists,...} give up their copyrights.

mokus 7y ago
I think copyright as you mention here is the right concept, or at least a lot closer. In particular, the limits on copyright. If someone is reciting a list of facts in public, they can’t expect people not to record those facts, because copyright doesn’t apply to that. Reciting the list in public using computers shouldn’t change that.
davorak 7y ago
> public doesn't mean the {composer,artists,...} give up their copyrights. Copyright law protects the original work of the composer and artists in your example. User profile data on Linkedin is not Linkedin original work. Additionally user profiles are mostly made up of facts, which are not copyrightable.

tyingq 7y ago

There's plenty of grey here. For example, scrapers that try to check people in for flights to get better seats. Some that tried to charge for that. That creates problems, where some customers benefit at the expense of others, high load on a "locking type" piece of code, etc. Similar for ticket sales for concerts, and probably other spaces. There are also companies that provide added value by compiling and correlating "public info" in a useful way that creates value. If Google let me scrape their search and remove ads, it would be popular, but is it "legal"? Or maybe Google Maps?

yuters 7y ago
Another problem with web scraping is a B2B website offering services for B2C companies to better reach consumers. It can be a tricky thing to do without basically giving your clients list to all your competitors.
LameRubberDucky 7y ago
I would think, and of course could be wrong, it would be as legal as Google scraping all of the web sites that they do in order to create their search engine in the first place. In particular, Google provides cached versions of web pages. That's pretty hardcore scraping.
- deleted 7y ago
  [deleted]
- tyingq 7y ago
  I put "legal" in quotes because I assume it's not criminal. It would, though, likely result in some pretty quick civil action.

TrackerFF 7y ago

I can understand why some do not want scrapers - increased traffic (with practically zero benefits to the owners) is one obvious reason. (Some people will then say "But why not just offer APIs", but that's a lot of extra work and maintenance). It's like with instagram and other social media platforms. The content creators put in the hard work, while the leeches are stealing content for their own benefit, giving zero credits to the original content creators.

xnyan 7y ago
There’s a very effective way out, don’t your data on the public web.
- kerkeslager 7y ago
  [decided to delete because I misunderstood the context]
  - deleted 7y ago
    [deleted]
  - ppseafield 7y ago
    Let's not conflate a) a person's personal data, and b) a business's dataset. The GP and the article are clearly referring to the latter. Preventing web scraping won't protect users from businesses collecting their data.
buboard 7y ago
Instagram is not the content creator tho
- TrackerFF 7y ago
  What I'm trying to say is: For every popular content creator on IG, there are tens and hundreds of (more or less automatic) content curators that do nothing more than scrape content with lots of likes, and re-post on their own channels. Then when they get sufficient followers, they make money through paid product placements, account flipping, pay-to-play sharing, and what not. More often than not, there's no linking to the original pages / creators.

sillyquiet 7y ago

Yeah, I have nothing at all against scraping per se, it's more about the huge bot traffic the commercial scrapers generate, which would ALSO be fine, but 1. it can be hard to tell scrapers from malicious DDOS bots sometimes and 2.) the person being scraped literally pays for that scraping traffic.

a1369209993 7y ago
So, apply DOS/DDOS mitigations? You need to have those anyway, so you might as well use them for this too.
- sillyquiet 7y ago
  Yeah, but where do you draw the line between 'oh sorry we dropped your request because of rate limiting' (or whatever mitigation strategy) and 'oh, we dropped your request because u scraping us bro' legally? IANAL, but this lawsuit seems to indicated that putting barriers in front of scraping attempts is a no-no.
  - a1369209993 7y ago
    "You issued more than $X requests per $Y seconds. We don't care whether you're scraping or not; try again later." for any values of X and Y.

DougWebb 7y ago

One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet to scrape all of the account information for a particular county. My client doesn't care so much about the data; it's the server load that's a problem. Normal activity for this site is a few dozen account searches per minute, but when the botnet gets through our blockade it sends hundreds of search requests per second, overwhelmimg the site. The operator of the botnet has NEVER tried to contact my client to ask for an efficient api to access the data, which they'd probably provide for a minimal fee. Data hosting isn't free, even if the data is.

gbasin 7y ago
Can you just put it behind a CDN that protects you?
vorpalhex 7y ago
Wouldn't the solution be to offer a streamlined download (maybe even as a torrent if you're worried about bandwidth) of all the data then?
- TremendousJudge 7y ago
  probably, but as GP said, "The operator of the botnet has NEVER tried to contact my client to ask for an efficient api to access the data" Some people just don't care for the commons
- Symbiote 7y ago
  I work on a fully open data repository. The website has the API linked in 3 places, so when I find inappropriate scraping I block it with "HTTP 420 ... see <API link> or contact <email>". Some people probably switch to using the API, but no-one has ever contacted us. They either give up, or run their scraper on a different computer -- I've seen the same scraper move between university computers, departments, then (in the evening) to a consumer broadband IP.

kerkeslager 7y ago

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want is, put simply, organizational transparency, personal privacy. I believe humans have a right to privacy, but I don't believe organizations have rights, period, and I believe radical transparency within an organization prevents organizations from trampling the rights of individuals. Organizations in this case include corporations, governments, and nonprofits.

whatshisface 7y ago
What if the organization is one person in an LLC? Do they get rights? If so then a big company can hire a bunch of little LLCs to act as rights-having proxies for any task that requires them.
- kerkeslager 7y ago
  I'm going to assume you're asking in good faith and try to address the confusion here. The human does get rights, the organization doesn't. In some cases, believing that humans have rights and believing that organizations have rights might lead one to the same action. In those cases, I'd take the action. I wouldn't want to violate a human's rights out of some vindictive dislike of organizations: that's not the point. The point is that I'd take that action because I believe in human rights, not because I believe that the organization has rights. And with organizational transparency: the entire point of organizational transparency is to protect human rights. In cases where organizational transparency would trample human rights to privacy, I would go with human rights every time. Violating the privacy of humans to achieve organizational transparency would defeat the entire purpose.
  - whatshisface 7y ago
    Let's say that individual humans have the right to keep secrets. Let's also say that they have the right to keep secrets with their associates, and to tell them to who they please. Now, doesn't that make it legal for a group of people to keep secrets about you? What about selling them? I just don't see what doing away with the legal fiction of corporate personage would do about Facebook.

levosmetalo 7y ago

Sure. My data is still my data, and if I publish it on my platform for free, that still shouldn't automatically give you the right to copy the data and provide on your platform. It's basically the same as a TV broadcasting a film for free, and then going after you legally if you recorded that film and uploaded it to your website.

sjg007 7y ago
It's not the same thing at all. The film is broadcast without re-transmission rights.
- levosmetalo 7y ago
  How is that different? Where on my website I gave anyone "re-transmission" or "re-publishing" rights?
  - sjg007 7y ago
    It's different for broadcast films and tv https://en.wikipedia.org/wiki/Retransmission_consent https://en.wikipedia.org/wiki/Retransmission_consent
jayd16 7y ago
This is a bad analogy as scraped but copyrighted works are still protected by copyright. Whats in question is whether you should have exclusive rights to information you have shared but did not copyright. Seems people also think this noncopyrighted data should also be protected even if its munged and added to an original work by the scraper.
hamiltont 7y ago
This does not legalize theft, it says sites cannot respond to suspected scrapers differently than they respond to non-suspected scrapers. You can still rate limit, as long as you do it universally to all site users Copyright law is unchanged. If someone scrapes your blog and then re-uses your posts on their own blog, you still have possible copyright infringement claim

soup10 7y ago

Billions are being made collecting data and content for free and publishing it along with targeted advertisements. Web-scrapers can collect that aggregated data and redistribute it or create competing services. This is terrifying to certain tech giants and threatens their moat's and lock-in/network advantage. They will lobby hard against it with PR campaigns citing everything from security, privacy, copyrights and all sorts of other exaggerated bogeymen. At the end of the day its mostly about preserving their monopolies though.

AgloeDreams 7y ago

People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it. By People you mean business or people people? Because I don't think people want everything to be public, many in fact use various networks to avoid oversharing and even then many people don't want their old bosses or exes looking at their profiles, there just don't exist tools to limit access that granularly.

jakelazaroff 7y ago

I want to be able to use LinkedIn to network with colleagues and people in my industry. If someone wants to scrape my profile to make a report on industry trends, I’m fine with it. What I don’t want is hiQ vacuuming up my data so they can snitch to my employer if they think I’m job hunting. How is this a paradox? Tech — the web in particular — is supposed to be an equalizing force, but HiQ is clearly trying to give my employer more power over me. We are an industry that prides itself on solving difficult problems — how is our response here to just throw up our hands and say “it’s all or nothing”?

gph 7y ago
>Tech — the web in particular — is supposed to be an equalizing force Don't take the Google PR so seriously, the tech industry wants to make money like every other industry.
Nasrudith 7y ago
We haven't solved that in the same way we haven't "solved" encryption not having a magical good people only door despite spook tantrums. There fundamentally isn't a possible mechanism and really wanting it doesn't change that. It is a result of equality - not of outcome but rules. Open for everyone but those whose applications you don't like isn't open. On a technical level trying to prevent it is like the "evil bit" as a solution to malware.
- jakelazaroff 7y ago
  Of course there are possible mechanisms. There are heuristics to detect bots. The whole reason for this lawsuit is that LinkedIn blocked hiQ from scraping their website. I'm also not necessarily talking about a technical defense against unwanted scraping. Write a law makes it illegal to do something like "scraping personally identifiable information and storing or presenting it non–anonymized", and prosecute companies who break it. I'm sure there are loopholes in that particular example, but the point is we can absolutely add shades of gray here. > Open for everyone but those whose applications you don't like isn't open. Openness should be a means, not an end. If we make something "not open" but it prevents 95% of undesirable uses and only 5% of desirable ones, is that not a tradeoff worth discussing?

stanferder 7y ago

I don't think the direction would ever be clear, even if the legality were clearly established. The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field. Companies want to provide some information to some people; but providing all information to all people is analogous to allowing customers to make a meal of free food samples, on a recurring basis.

mindcrime 7y ago
The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field. It already is. There are entire companies, like Distil Networks, who exist solely to protect companies from bots/scrapers/etc. Actually, looks like Distil got acquired and are now part of Imperva, but anyway, the idea is the same. This is definitely an existing field. https://www.imperva.com/products/bot-management/ https://www.imperva.com/products/bot-management/ Disclosure: former Distil employee, but I have no financial stake in this discussion, and have mixed feelings about scraping. Clearly it can be beneficial in some situations, but when I think about having to pay exorbitant prices to scalpers for tickets to an event, because they used a bot to buy up all the tickets, that is less appealing.

Nasrudith 7y ago

It sounds like the same old issue - lack of understanding of the fundamentals underlying technologies. Otherwise they would realize what they demand is contradictory and incoherent like demanding to be both viewable by all and not viewable. DRM is one fundamental example of it.

Tactic 7y ago

The issue here for some, if not many, is a matter of scale. It is one thing if an end-user, whom I am trying to service, comes to my site and gets my publicly available data. Maybe I monetize with ads, maybe not. It doesn't matter, that is the audience I am trying to service, regardless of size. But when you scrape it my load goes up dramatically. A load I have to pay for. It is analogous to the privacy debates going on with one said saying "hey, don't track everywhere I go and tag me with facial recognition" and the other side saying "hey, you are in public and people can see you." The issue is not complete privacy, but one of scale. And of intent. I believe society is soon going to have to come to grips with the scale of things and legislate what are acceptable scales of action as it seems to be becoming a large issue in a growing number are areas.

orf 7y ago
Provide an API for public data to reduce the costs associated with rendering a full blown page, and deliver just the information needed.
- Tactic 7y ago
  Entirely feasible. Also reasonable for you to pay me for the service as it is taking my development efforts to meet your business model. The advantage to you is you have a defined interface that I won't prevent.
  - orf 7y ago
    I guess you missed the comment I was replying to: it may cost you more money, in bandwidth and per page resources, to not provide an API than it does for you to provide one. So no, I won’t pay you for the privilege of you saving money.
  - oauea 7y ago
    No, I'll happily just scrape your site instead. But if you'd rather not have that happen, provide an API.
- briandear 7y ago

Psyladine 7y ago

People love when information is free, but as soon as monetization appears they see it like taking food from their plate.

cirno 7y ago

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it Can you really not imagine a world where a person accidentally or in poor judgment uploads something private to their own site (their real name, home address, credit card#, or any piece of highly damaging information that could cost them their careers) and wishes for it to be removed? (but can't because many of these scraping sites never respond to takedown requests) People make mistakes and post things they shouldn't. A mistake from someone many years in the past that they've made amends for shouldn't haunt someone for the rest of their lives. But it does when we decide that every single line of text ever uttered online must be preserved and easily accessible by anyone for all eternity. Blocking scrapers is an arms-race escalation because these sites refuse to remove content, and it's used as a tool for character assassination by bad actors. It's a proactive defense.

make3 7y ago

they want there data to be public for a specific use, I feel like that's pretty easy to understand. LinkedIn: they want their info to be public -> to get jobs