6 ms·

Being a programmer not a lawyer, I like the idea of more rights for scrapers. I don't want to see the internet partitioned away and owned by a few companies, e

Being a programmer not a lawyer, I like the idea of more rights for scrapers. I don't want to see the internet partitioned away and owned by a few companies, especially when that information is often called a "public profile".

FLUX-YOU 9y ago
Can they claim a tax credit for supporting that bandwidth usage and handling abuse?
- tomc1985 9y ago
  Not needed. Cost of doing business.
  - FLUX-YOU 9y ago
    I don't trust the US government to write good rights for scrapers. They can't even do computer crime sentences well. At best, it's a burden for no solid gain for society. At worst, there will be loopholes used to DoS businesses because they can't shut down individuals due to law-given rights, and that will lead to court fights. These rights would do nothing but save scraper authors from learning to obfuscate their actions.
    - tomc1985 9y ago
      Information wants to be free. People should stop fighting it! If one makes information "public" but don't really want to share it, then the public is fully justified in taking it.
      - biocomputation 9y ago
        << Information wants to be free. People should stop fighting it! That's why you routinely publish your bank account, social insurance, and credit card details online, right?
        tomc1985 9y ago
        Not like I have a choice... and how are those things explicitly declared 'public'?
djsumdog 9y ago
It gets into the incredibly murky water of how the web works. You're just issuing a request and getting things back. Sometimes in a web browser, sometimes not. But the content itself may still be copyright. You can't just take it, even though for now, the publisher/server is allowing you to view it for free. But what if you only chose to view some of the content (e.g. block ads). What if you apply your own styles to change the way that information is displayed? You're just changing the way the browser represents that data. You're not redistributing it as your own at this point. What if you store that data, but don't republish it; just used it in some each algorithms? There are a whole lot of interesting grey areas here, but many that already have precedents that side more with the copyright holders.
- swiley 9y ago
  Maybe the whole idea of copyright is flawed and harmful?
  - bitJericho 9y ago
    There's nothing wrong with copyrights; 14 year copyrights.
    - int_19h 9y ago
      Why 14, though? I'm well aware of the historical precedent, but that number was rather arbitrary even then - it was what a bunch of people agreed upon, based on their ideas and experience, and given the environment. It's doubly arbitrary today, considering how much the environment has changed. Is a 14-year copyright on software reasonable, for example, or too long. Rather than making it a hard cut-off point, it would be interesting to come up with a scheme that attempts to capture the spirit of term limits. Consider: why are copyright terms even a thing? Well, copyright is a monopoly on a thing that is not naturally restricted; it does not exist in the absence of society, and is therefore a privilege granted by that society. By itself, copyright is meant to encourage creativity in the interest of public good, and at the same time, to provide some means to derive profit from one's creative expression. So there are two conflicting interests at play here - the desire of the creator to be rewarded for the fruits of his labor, and the desire of the society to enjoy growing, constantly enriched culture. The copyright term, then, marks the point at which the latter trumps the former. Instead, what we could do is capture the fact that the interests conflict. For as long as you hold copyright, you're effectively denying society the ability to freely enjoy the culture that you have enriched. Why, then, not tax the copyright accordingly? You could consider it a kind of intellectual property tax, but with a twist: the longer copyright is held, the more the interests of society are infringed, and the larger the compensatory payment required to maintain the copyright. So we could start with a grace period of a couple of years that is completely free, then it starts growing steadily. For some really popular work that makes significant profits, the author could easily afford payments to maintain copyright for a decade or two (or however long - that is something that can be dialed arbitrarily). For things that are too obscure, payments would cease shortly, and they would fall to public domain. There wouldn't be such a thing as "abandonware" anymore. What use to put the money to? Many possibilities there. Publicly sponsored arts and art education is an obvious choice. Another interesting example would be offering bulk sums of money to authors of culturally important works to surrender their copyrights sooner, so that the public can enjoy them.
smegel 9y ago
If a website puts something on the public internet, it should not even be aware if it is being accessed by a scraper or a human. Maybe we should just ban User Agent strings and be done with it.
- rev_bird 9y ago
  I'd be willing to bet that the user-agent field isn't the problem; it's patterns that everything looks for now, right? People have been lying in the request headers for decades.
  - Maultasche 9y ago
    Yeah, I remember a plugin in the early days of Firefox that allowed Firefox pretend to be Internet Explorer. It would send request headers that made it appear to be IE. There were websites at the time that would display just fine in Firefox, but would refuse to display anything if they detected a non-IE browser.
- sbarre 9y ago
  You call it the "public internet" but it's most definitely not a public space or anything like it. Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained. I'm not saying I agree with Linkedin in this particular scenario, but this is about two commercial for-profit entities arguing over money, so let's not make it about something it's not.
  - smegel 9y ago
    > Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained And are MORE than happy to send the content of their servers to unsolicited, uninvited, anonymous guests on mere request. No-one is forcing them to do so!
    - sbarre 9y ago
      And that's exactly LinkedIn's position here. No one should be forcing them to send their content to anyone. They claim they should be allowed to discriminate at their discretion who they respond to, since they own and operate the servers. This "no one is forcing you to send your content" goes both ways.. If one side is going to say they're entitled to receive the content on request, the other side wants to be able to say they're entitled to refuse to answer that request..
dahart 9y ago
> Being a programmer not a lawyer, I like the idea of more rights for scrapers. What rights should scrapers have that they don't right now? Keep in mind that a lot of the scraping going on is just some other private company abusing access and hoping to gather and use the information for their own private profit. How many companies are scraping StackOverflow for example and doing nothing but attempting to copy it and draw traffic to their own site? I can't stand copycat sites, they fill my search results with junk. I would assume the majority of scraping that is currently happening is not doing the public any good. > I don't want to see the internet partitioned away and owned by a few companies, especially when that information is often called a "public profile". This sounds like you're suggesting that LinkedIn or Facebook calling your profile a 'public profile' means that the law should treat it as a public service due to use of the word 'public', is that what you mean? The word public may be overloaded here. I can see why tax funded projects should be publicly accessible, but I have a hard time seeing why private companies should be compelled to provide access to anything at their own expense.
- ocdtrekkie 9y ago
  Consider that should anything ever happen to the sites they scrape, suddenly they become super valuable to the rest of us. Decentralized information is not a bad thing. And arguably, if monetizing those scraped sites pay for them duplicating the data to additional places, I think that's probably fine. Arguably, if they do a better job getting that information in results to people who need it in search, they may be performing a service there as well. (A lot of decently informative sites have absolutely awful search/visibility.)
  - dahart 9y ago
    > Consider that should anything ever happen to the sites they scrape, suddenly they become super valuable to the rest of us. That may well be true, but that value doesn't mean anyone should just be able to take that value from the company that put up the effort and investment to collect the data, and turn around an use it for their own profit. Nor does it mean that a company shouldn't be able to serve the data to whomever it wants and/or restrict access from whomever it wants. Value to the consumer is still not a reason to compel private companies to offer public services. It would be valuable to both of us if Google gave us free money, but no court is going to compel them to do so just because of the potential value to you and me. It seems bad, btw, if we choose to rely on private companies to keep the only backups of our personal data. If a site going down has a negative effect on my life, and takes down data with it that I need, it might be an indication that I shouldn't have kept my data there. Also true that decentralized information is not a bad thing, as a generic ideal or a data backup plan. But for a business, decentralization in this context means loss of profit, as well as possibly theft, cheating, and copyright violation. Also, the increased value that comes from a company folding will be used against you by these private, for profit scrapers. They can and will hold their copy ransom for more money, if possible. > Arguably, if they do a better job getting that information in results to people who need it in search, they may be performing a service there as well. (A lot of decently informative sites have absolutely awful search/visibility.) What is the argument in favor of this being legal? It is currently not legal, and the law currently does not give any credit for 'doing it better'. Why should it?
ipnon 9y ago
This case is much more significant than that. The ruling will determine whether or not breaching the terms and agreements of a website constitutes a crime under the Computer Fraud and Abuse Act (CFAA). LinkedIn is arguing that since scraping is not defined by the company itself to be a permitted use, a federal felony is being committed by anyone using a script to access its website. The language used in the CFAA is "unauthorized access" and this case will set a legal precedent for the meaning of that term.
- qarioz 9y ago
  So some company can sue me if I use `wget somecompany.com` if LinkedIn win?
- razwall 9y ago
  We can hope that this case will set a legal precedent, but it may, like the Craigslist case, end in a settlement (or some other disposition) before that point is reached.
harrygeez 9y ago
The way I see it, if you want to make a 'public' profile, host it yourself, create your own website. LinkedIn has spent a lot of resources to create a service/network that helps to connect you with recruiters (or the opposite), whose information may not be easily available to you or can be found easily in the 'public' Internet. If you think from a product owner's standpoint, why should I let just anyone scrape the content collection that I built with so much effort?
RHSman2 9y ago
Agree. However, I think there is a difference between harvesting a database's contents and observing trends, data and creating algorithms based around that observed data.

FLUX-YOU 9y ago

Can they claim a tax credit for supporting that bandwidth usage and handling abuse?

tomc1985 9y ago
Not needed. Cost of doing business.
- FLUX-YOU 9y ago
  I don't trust the US government to write good rights for scrapers. They can't even do computer crime sentences well. At best, it's a burden for no solid gain for society. At worst, there will be loopholes used to DoS businesses because they can't shut down individuals due to law-given rights, and that will lead to court fights. These rights would do nothing but save scraper authors from learning to obfuscate their actions.
  - tomc1985 9y ago
    Information wants to be free. People should stop fighting it! If one makes information "public" but don't really want to share it, then the public is fully justified in taking it.
    - biocomputation 9y ago
      << Information wants to be free. People should stop fighting it! That's why you routinely publish your bank account, social insurance, and credit card details online, right?
      - tomc1985 9y ago
        Not like I have a choice... and how are those things explicitly declared 'public'?

djsumdog 9y ago

It gets into the incredibly murky water of how the web works. You're just issuing a request and getting things back. Sometimes in a web browser, sometimes not. But the content itself may still be copyright. You can't just take it, even though for now, the publisher/server is allowing you to view it for free. But what if you only chose to view some of the content (e.g. block ads). What if you apply your own styles to change the way that information is displayed? You're just changing the way the browser represents that data. You're not redistributing it as your own at this point. What if you store that data, but don't republish it; just used it in some each algorithms? There are a whole lot of interesting grey areas here, but many that already have precedents that side more with the copyright holders.

swiley 9y ago
Maybe the whole idea of copyright is flawed and harmful?
- bitJericho 9y ago
  There's nothing wrong with copyrights; 14 year copyrights.
  - int_19h 9y ago
    Why 14, though? I'm well aware of the historical precedent, but that number was rather arbitrary even then - it was what a bunch of people agreed upon, based on their ideas and experience, and given the environment. It's doubly arbitrary today, considering how much the environment has changed. Is a 14-year copyright on software reasonable, for example, or too long. Rather than making it a hard cut-off point, it would be interesting to come up with a scheme that attempts to capture the spirit of term limits. Consider: why are copyright terms even a thing? Well, copyright is a monopoly on a thing that is not naturally restricted; it does not exist in the absence of society, and is therefore a privilege granted by that society. By itself, copyright is meant to encourage creativity in the interest of public good, and at the same time, to provide some means to derive profit from one's creative expression. So there are two conflicting interests at play here - the desire of the creator to be rewarded for the fruits of his labor, and the desire of the society to enjoy growing, constantly enriched culture. The copyright term, then, marks the point at which the latter trumps the former. Instead, what we could do is capture the fact that the interests conflict. For as long as you hold copyright, you're effectively denying society the ability to freely enjoy the culture that you have enriched. Why, then, not tax the copyright accordingly? You could consider it a kind of intellectual property tax, but with a twist: the longer copyright is held, the more the interests of society are infringed, and the larger the compensatory payment required to maintain the copyright. So we could start with a grace period of a couple of years that is completely free, then it starts growing steadily. For some really popular work that makes significant profits, the author could easily afford payments to maintain copyright for a decade or two (or however long - that is something that can be dialed arbitrarily). For things that are too obscure, payments would cease shortly, and they would fall to public domain. There wouldn't be such a thing as "abandonware" anymore. What use to put the money to? Many possibilities there. Publicly sponsored arts and art education is an obvious choice. Another interesting example would be offering bulk sums of money to authors of culturally important works to surrender their copyrights sooner, so that the public can enjoy them.

smegel 9y ago

If a website puts something on the public internet, it should not even be aware if it is being accessed by a scraper or a human. Maybe we should just ban User Agent strings and be done with it.

rev_bird 9y ago
I'd be willing to bet that the user-agent field isn't the problem; it's patterns that everything looks for now, right? People have been lying in the request headers for decades.
- Maultasche 9y ago
  Yeah, I remember a plugin in the early days of Firefox that allowed Firefox pretend to be Internet Explorer. It would send request headers that made it appear to be IE. There were websites at the time that would display just fine in Firefox, but would refuse to display anything if they detected a non-IE browser.
sbarre 9y ago
You call it the "public internet" but it's most definitely not a public space or anything like it. Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained. I'm not saying I agree with Linkedin in this particular scenario, but this is about two commercial for-profit entities arguing over money, so let's not make it about something it's not.
- smegel 9y ago
  > Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained And are MORE than happy to send the content of their servers to unsolicited, uninvited, anonymous guests on mere request. No-one is forcing them to do so!
  - sbarre 9y ago
    And that's exactly LinkedIn's position here. No one should be forcing them to send their content to anyone. They claim they should be allowed to discriminate at their discretion who they respond to, since they own and operate the servers. This "no one is forcing you to send your content" goes both ways.. If one side is going to say they're entitled to receive the content on request, the other side wants to be able to say they're entitled to refuse to answer that request..

dahart 9y ago

> Being a programmer not a lawyer, I like the idea of more rights for scrapers. What rights should scrapers have that they don't right now? Keep in mind that a lot of the scraping going on is just some other private company abusing access and hoping to gather and use the information for their own private profit. How many companies are scraping StackOverflow for example and doing nothing but attempting to copy it and draw traffic to their own site? I can't stand copycat sites, they fill my search results with junk. I would assume the majority of scraping that is currently happening is not doing the public any good. > I don't want to see the internet partitioned away and owned by a few companies, especially when that information is often called a "public profile". This sounds like you're suggesting that LinkedIn or Facebook calling your profile a 'public profile' means that the law should treat it as a public service due to use of the word 'public', is that what you mean? The word public may be overloaded here. I can see why tax funded projects should be publicly accessible, but I have a hard time seeing why private companies should be compelled to provide access to anything at their own expense.

ocdtrekkie 9y ago
Consider that should anything ever happen to the sites they scrape, suddenly they become super valuable to the rest of us. Decentralized information is not a bad thing. And arguably, if monetizing those scraped sites pay for them duplicating the data to additional places, I think that's probably fine. Arguably, if they do a better job getting that information in results to people who need it in search, they may be performing a service there as well. (A lot of decently informative sites have absolutely awful search/visibility.)
- dahart 9y ago
  > Consider that should anything ever happen to the sites they scrape, suddenly they become super valuable to the rest of us. That may well be true, but that value doesn't mean anyone should just be able to take that value from the company that put up the effort and investment to collect the data, and turn around an use it for their own profit. Nor does it mean that a company shouldn't be able to serve the data to whomever it wants and/or restrict access from whomever it wants. Value to the consumer is still not a reason to compel private companies to offer public services. It would be valuable to both of us if Google gave us free money, but no court is going to compel them to do so just because of the potential value to you and me. It seems bad, btw, if we choose to rely on private companies to keep the only backups of our personal data. If a site going down has a negative effect on my life, and takes down data with it that I need, it might be an indication that I shouldn't have kept my data there. Also true that decentralized information is not a bad thing, as a generic ideal or a data backup plan. But for a business, decentralization in this context means loss of profit, as well as possibly theft, cheating, and copyright violation. Also, the increased value that comes from a company folding will be used against you by these private, for profit scrapers. They can and will hold their copy ransom for more money, if possible. > Arguably, if they do a better job getting that information in results to people who need it in search, they may be performing a service there as well. (A lot of decently informative sites have absolutely awful search/visibility.) What is the argument in favor of this being legal? It is currently not legal, and the law currently does not give any credit for 'doing it better'. Why should it?

ipnon 9y ago

This case is much more significant than that. The ruling will determine whether or not breaching the terms and agreements of a website constitutes a crime under the Computer Fraud and Abuse Act (CFAA). LinkedIn is arguing that since scraping is not defined by the company itself to be a permitted use, a federal felony is being committed by anyone using a script to access its website. The language used in the CFAA is "unauthorized access" and this case will set a legal precedent for the meaning of that term.

qarioz 9y ago
So some company can sue me if I use `wget somecompany.com` if LinkedIn win?
razwall 9y ago
We can hope that this case will set a legal precedent, but it may, like the Craigslist case, end in a settlement (or some other disposition) before that point is reached.

harrygeez 9y ago

The way I see it, if you want to make a 'public' profile, host it yourself, create your own website. LinkedIn has spent a lot of resources to create a service/network that helps to connect you with recruiters (or the opposite), whose information may not be easily available to you or can be found easily in the 'public' Internet. If you think from a product owner's standpoint, why should I let just anyone scrape the content collection that I built with so much effort?

RHSman2 9y ago

Agree. However, I think there is a difference between harvesting a database's contents and observing trends, data and creating algorithms based around that observed data.