7 ms·

This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logic

This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and fingerprints to fight the nature of the internet. Finally everyone wants benefits of public data (search engine indexing etc) but don't really want to give anything back to the ecosystem. It's just pure greed and law, our society and government shouldn't aid it in any way, shape or form.

IfOnlyYouKnew 6y ago
You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do. Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them access becomes legally binding. In the offline world, we derive great benefit from following the second model. Indeed, if only measures that successfully prevent people from entering your house without permissions were to count, you wouldn't need laws in the first place! You would, however, need a bunker. Which is quite a bit more expansive than a functioning legal system. In that offline world, we have created all sorts of additional rules to balance rights for specific situations, and we rely on a canon of expectations that say, for example, that it's usually not ok to enter a private house, but you don't need explicit permission to enter a supermarket. These are still developing for the online world, and your idea of the "ecosystem" hints at that. But you're just taking from those contradictory ideas above to arrive at the outcome you intuitively feel is "just": a bit of might-is-right when it comes to "whatever is online is fair game", followed by principled ideas of rights and obligations when websites try to defend themselves in that jungle of yours. What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road.
- wraptile 6y ago
  > You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There is no middle ground _in the protocol_ and that's why I explicitly said hacks. Any system can be rehashed into anything else with unlimited extra layers on top of it - by your definition everything is everything. > What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road. Sorry but that sounds uneforcable and rather absurd. We have the framework in place already - if you don't want something to be public don't put it out in the public _explicitly_. To add we already have legal framework in place for copyrighted and/or protected content like photos and against any sort of malicious attacks like ddos. Our web is getting extremely centralized and most of these majors are natural monopolies: google search becomes stronger the more data it has - google can scrape the entire web freely yet it's competition can't; facebook becomes stronger the more data is has etc. etc. One way to restore balance is to ensure that public data remains public and the ecosystem can have healthy competition and growth otherwise we're moving to a very dystopian corporate owned world.
- dec0dedab0de 6y ago
  I think a closer "offline world" analogy would be if a magazine tried to enforce rules to what you can do with it after it was delivered.
- hhjj 6y ago
  The offline world is the world you are by default, you can't choose to put your house elsewhere. On contrary, when you put information on internet you choose to do it there (at least for website owners...). If you want to put restrictions on the usage of your data, make people sign a contract before accessing it. Also a contract should not have "the public" as a party or a subset of, you should be able to identify the parties you have contracted with. Else you may end up with warrants targeting everyone or a subset of...
  - IfOnlyYouKnew 6y ago
nickff 6y ago
>"a complete logical oxymoron: they want data to be public but also select who gets to see it." I think you're proving too much here. Your argument applies to all published authors, and would strike a crippling blow to their copyright.
- aleksei 6y ago
  First of all I should think copyright only restricts publishing, not reading. Obviously if you put a price on your book I have to pay it, but that's a different issue. Secondly the Internet is best viewed as a public noticeboard purely because of the way the protocol works. There's just no getting around that. I think you'd agree that putting up a notice on a street corner and then getting offended when people read it would be viewed as rather odd, if not something else.
- tremon 6y ago
  But published authors don't have control over who gets to see their work -- only who gets to profit from it (distribution rights, not use rights).
matthewhartmans 6y ago
>>> Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Wouldn't it be ironic if the non-selective people used this as leverage for discrimination.
deleted 6y ago
[deleted]
snarf21 6y ago
I think web-scraping is ok but it should be treated as copyrighted data. Let's take a real example: We've built a company that creates ratings for doctors via a private process. We'd like the doctors to be able to show their good rating on their website via a linked widget. How do we stop a competitor from not only scraping our data but using it as their own and/or selling it to someone else?
- Siira 6y ago
  By network effects and brand loyalty.
  - snarf21 6y ago
    We're small with no funding. Our competitors have $100Ms of VC funding. How do we even get network effects and brand loyalty?

IfOnlyYouKnew 6y ago

You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do. Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them access becomes legally binding. In the offline world, we derive great benefit from following the second model. Indeed, if only measures that successfully prevent people from entering your house without permissions were to count, you wouldn't need laws in the first place! You would, however, need a bunker. Which is quite a bit more expansive than a functioning legal system. In that offline world, we have created all sorts of additional rules to balance rights for specific situations, and we rely on a canon of expectations that say, for example, that it's usually not ok to enter a private house, but you don't need explicit permission to enter a supermarket. These are still developing for the online world, and your idea of the "ecosystem" hints at that. But you're just taking from those contradictory ideas above to arrive at the outcome you intuitively feel is "just": a bit of might-is-right when it comes to "whatever is online is fair game", followed by principled ideas of rights and obligations when websites try to defend themselves in that jungle of yours. What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road.

wraptile 6y ago
> You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There is no middle ground _in the protocol_ and that's why I explicitly said hacks. Any system can be rehashed into anything else with unlimited extra layers on top of it - by your definition everything is everything. > What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road. Sorry but that sounds uneforcable and rather absurd. We have the framework in place already - if you don't want something to be public don't put it out in the public _explicitly_. To add we already have legal framework in place for copyrighted and/or protected content like photos and against any sort of malicious attacks like ddos. Our web is getting extremely centralized and most of these majors are natural monopolies: google search becomes stronger the more data it has - google can scrape the entire web freely yet it's competition can't; facebook becomes stronger the more data is has etc. etc. One way to restore balance is to ensure that public data remains public and the ecosystem can have healthy competition and growth otherwise we're moving to a very dystopian corporate owned world.
dec0dedab0de 6y ago
I think a closer "offline world" analogy would be if a magazine tried to enforce rules to what you can do with it after it was delivered.
hhjj 6y ago
The offline world is the world you are by default, you can't choose to put your house elsewhere. On contrary, when you put information on internet you choose to do it there (at least for website owners...). If you want to put restrictions on the usage of your data, make people sign a contract before accessing it. Also a contract should not have "the public" as a party or a subset of, you should be able to identify the parties you have contracted with. Else you may end up with warrants targeting everyone or a subset of...
- IfOnlyYouKnew 6y ago

nickff 6y ago

>"a complete logical oxymoron: they want data to be public but also select who gets to see it." I think you're proving too much here. Your argument applies to all published authors, and would strike a crippling blow to their copyright.

aleksei 6y ago
First of all I should think copyright only restricts publishing, not reading. Obviously if you put a price on your book I have to pay it, but that's a different issue. Secondly the Internet is best viewed as a public noticeboard purely because of the way the protocol works. There's just no getting around that. I think you'd agree that putting up a notice on a street corner and then getting offended when people read it would be viewed as rather odd, if not something else.
tremon 6y ago
But published authors don't have control over who gets to see their work -- only who gets to profit from it (distribution rights, not use rights).

matthewhartmans 6y ago

>>> Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Wouldn't it be ironic if the non-selective people used this as leverage for discrimination.

deleted 6y ago

[deleted]

snarf21 6y ago

I think web-scraping is ok but it should be treated as copyrighted data. Let's take a real example: We've built a company that creates ratings for doctors via a private process. We'd like the doctors to be able to show their good rating on their website via a linked widget. How do we stop a competitor from not only scraping our data but using it as their own and/or selling it to someone else?

Siira 6y ago
By network effects and brand loyalty.
- snarf21 6y ago
  We're small with no funding. Our competitors have $100Ms of VC funding. How do we even get network effects and brand loyalty?