5 ms·

The IETF is working on a solution to this, and Cloudlare is actively contributing and implementing; see https://blog.cloudflare.com/eliminating-captchas-on-ipho

by mnot 4y ago

The IETF is working on a solution to this, and Cloudlare is actively contributing and implementing; see https://blog.cloudflare.com/eliminating-captchas-on-iphones-and-macs-using-new-standard/ https://blog.cloudflare.com/eliminating-captchas-on-iphones-...

dceddia 4y ago
This plan looks even worse for privacy, and seems to rely on authenticating the unique device itself. > Know your user is coming from an authentic device and signed application, verified by the device vendor directly. If they manage to get wide adoption of something like this it seems like a very bad day for privacy, and a joyous day for advertisers, anyone who wants to prevent web scraping, Google and/or anyone who might want to make it hard to crawl the entire web…
- mnot 4y ago
  Read a bit more; that’s not how the cryptography works. They’re blinded tokens.
  - dceddia 4y ago
    I read further, and it sounds like they've designed it with privacy in mind. That's nice. I'm still having a hard time seeing how this doesn't eventually lead to a completely locked-down internet, where users can only use approved browsers and devices. They have a list of steps for how a request would be made, where steps 2 and 3 are: > 2. Safari supports PATs, so it will make an API call to Apple’s Attester, asking them to attest. > 3. The Apple attester will check various device components, confirm they are valid, and then make an API call to the Cloudflare Issuer (since Cloudflare acting as an Origin chooses to use the Cloudflare Issuer). In a theoretical future world where 99% of site operators have set this up for protection, and 99% of users are using approved browsers, how would one do something like... Create a competitor to Google? You'd need to crawl the web for that. Would you imagine Apple or Cloudflare would gladly let your device request millions of tokens per hour? Or would that be throttled or disallowed entirely? Use curl (or telnet, or [any other HTTP client]) to grab a page? Use yt-dlp to download a YouTube video? Scrape a bunch of data for an AI project? See this article from the front page where someone scraped a bunch of car listings from KBB and trained a model to estimate car prices and found some interesting results. https://blog.aqnichol.com/2022/12/31/large-scale-vehicle-classification/ https://blog.aqnichol.com/2022/12/31/large-scale-vehicle-cla... - would something like that be permitted under a system like this? Or might you need to own/rent an army of authorized devices with authorized browsers to do that experiment?
    - userbinator 4y ago
      The Apple attester will check various device components, confirm they are valid Here's the "you will be under our control" part of their scheme. Running any "unauthorised" software? Rooted/jailbroken? Certain "security" features disabled? Using third-party replacement parts? ... Social credit score too low? Too bad, you're now denied access.
    - jws 4y ago
      I've implemented PAT on a service. A server would only want to require PAT when it wants to see if it is dealing with a human. In the context of, say a blog, you would require PAT when someone wants to make an anonymous comment or create an account. For requests which are "reading" you just serve content, unless for some reason you only want human eyeballs to see your content. In your proposed scenario, reading the publicly accessible contents of the web, there should be no problems. (Of course some percentage of sites will accidentally have required PAT at any time and be unscannable, but presumably they figure that out and fix it.) Now for the good side: I, reluctantly, implemented a geolocation filter to control anonymous content additions to that service I was alluding to. I felt bad about it, but I also felt bad having to filter out content spam every day. It turned out that all my strange content spam came from one country, so I banned 143 million people from anonymous content creation for my convenience. With PAT I can remove the national ban and let any "probably human" in.
      - dceddia 4y ago
        > For requests which are "reading" you just serve content, unless for some reason you only want human eyeballs to see your content. I was going to cite the LinkedIn case where, last I had heard, the courts had decided that scraping was legal... Headline [0] from April: > Court rules that data scraping is legal in LinkedIn appeal > LinkedIn has lost its latest attempt to block companies from scraping information from its public pages, including member pages. ... but upon googling it, I found a more recent [1] ruling :/ > LinkedIn prevails in 6-year lawsuit against data scraper > The U.S. District Court for the Northern District of California sided with LinkedIn in its six year lawsuit against a firm that scraped data ... So that sure puts a nail into the argument I was going to make. But still, while I think your use case lines up with the spirit of this kind of system, I think the reality is that it also would be used by every single site with a signup wall to kill off the archive.ph's of the world. 0: https://www.zdnet.com/article/court-rules-that-data-scraping-is-legal-in-linkedin-appeal/ https://www.zdnet.com/article/court-rules-that-data-scraping... 1: https://iapp.org/news/a/linkedin-prevails-in-six-year-lawsuit-against-data-scraper/ https://iapp.org/news/a/linkedin-prevails-in-six-year-lawsui...
- charcircuit 4y ago
  Stopping ad fraud only hurts people trying to make money from it. Preventing web scraping is good for privacy. This is implemented in a privacy preserving way.
  - nibbleshifter 4y ago
    > Preventing web scraping is good for privacy. You will need to elaborate on that.
    - gattilorenz 4y ago
      I assume the OP means that scraping is used to collect public data, including that of individuals, which can even be linked across different websites. There’s at least a couple of services that try to connect somebody’s Instagram account to their FB/Twitter/LinkedIn etc. I assume some of those rely on scraping (+username checking), since the TOS for the APIs of those social networks probably prohibit this use case.
      - charcircuit 4y ago
        Yes, big datasets of user data can be created and sold. This user data can be joined across multiple sites to build up profiles on people. These datasets floating around can harm the reputation of a site.
  - userbinator 4y ago
    Preventing web scraping is good for privacy It's bad for freedom. Very, very very bad. Talking about the "privacy" of what has been made publicly available makes no sense.
    - charcircuit 4y ago
      >It's bad for freedom So are laws in the real world about stealing. >Talking about the "privacy" of what has been made publicly available makes no sense. Yes, it does. Users often wish to be able to delete or make something that was once public private. For example someone could post a picture of themselves on twitter. A year later they are no longer comfortable with having pictures of themself online so they go and delete them. Despite the user deleting them malicious scrapers will not delete them and keep those images. Another example would be setting your real name to your twitter name. Later you aren't comfortable using your real name so you change it away. Scrapers may still have your real name despite you wanting it to be a secret.
thataintright 4y ago
Why is the IETF letting the CIA influence the design of network protocols?
- dane-pgp 4y ago
  If you think that's bad, you're not going to like the fact that they're letting the NSA design a system for "TPM-based Network Device Remote Integrity Verification": https://www.ietf.org/archive/id/draft-ietf-rats-tpm-based-network-device-attest-14.html https://www.ietf.org/archive/id/draft-ietf-rats-tpm-based-ne...
- RedShift1 4y ago
  If you think that's bad, don't look up what happened with IPsec.

by mnot 4y ago

dceddia 4y ago

This plan looks even worse for privacy, and seems to rely on authenticating the unique device itself. > Know your user is coming from an authentic device and signed application, verified by the device vendor directly. If they manage to get wide adoption of something like this it seems like a very bad day for privacy, and a joyous day for advertisers, anyone who wants to prevent web scraping, Google and/or anyone who might want to make it hard to crawl the entire web…

mnot 4y ago
Read a bit more; that’s not how the cryptography works. They’re blinded tokens.
- dceddia 4y ago
  I read further, and it sounds like they've designed it with privacy in mind. That's nice. I'm still having a hard time seeing how this doesn't eventually lead to a completely locked-down internet, where users can only use approved browsers and devices. They have a list of steps for how a request would be made, where steps 2 and 3 are: > 2. Safari supports PATs, so it will make an API call to Apple’s Attester, asking them to attest. > 3. The Apple attester will check various device components, confirm they are valid, and then make an API call to the Cloudflare Issuer (since Cloudflare acting as an Origin chooses to use the Cloudflare Issuer). In a theoretical future world where 99% of site operators have set this up for protection, and 99% of users are using approved browsers, how would one do something like... Create a competitor to Google? You'd need to crawl the web for that. Would you imagine Apple or Cloudflare would gladly let your device request millions of tokens per hour? Or would that be throttled or disallowed entirely? Use curl (or telnet, or [any other HTTP client]) to grab a page? Use yt-dlp to download a YouTube video? Scrape a bunch of data for an AI project? See this article from the front page where someone scraped a bunch of car listings from KBB and trained a model to estimate car prices and found some interesting results. https://blog.aqnichol.com/2022/12/31/large-scale-vehicle-classification/ https://blog.aqnichol.com/2022/12/31/large-scale-vehicle-cla... - would something like that be permitted under a system like this? Or might you need to own/rent an army of authorized devices with authorized browsers to do that experiment?
  - userbinator 4y ago
    The Apple attester will check various device components, confirm they are valid Here's the "you will be under our control" part of their scheme. Running any "unauthorised" software? Rooted/jailbroken? Certain "security" features disabled? Using third-party replacement parts? ... Social credit score too low? Too bad, you're now denied access.
  - jws 4y ago
    I've implemented PAT on a service. A server would only want to require PAT when it wants to see if it is dealing with a human. In the context of, say a blog, you would require PAT when someone wants to make an anonymous comment or create an account. For requests which are "reading" you just serve content, unless for some reason you only want human eyeballs to see your content. In your proposed scenario, reading the publicly accessible contents of the web, there should be no problems. (Of course some percentage of sites will accidentally have required PAT at any time and be unscannable, but presumably they figure that out and fix it.) Now for the good side: I, reluctantly, implemented a geolocation filter to control anonymous content additions to that service I was alluding to. I felt bad about it, but I also felt bad having to filter out content spam every day. It turned out that all my strange content spam came from one country, so I banned 143 million people from anonymous content creation for my convenience. With PAT I can remove the national ban and let any "probably human" in.
    - dceddia 4y ago
      > For requests which are "reading" you just serve content, unless for some reason you only want human eyeballs to see your content. I was going to cite the LinkedIn case where, last I had heard, the courts had decided that scraping was legal... Headline [0] from April: > Court rules that data scraping is legal in LinkedIn appeal > LinkedIn has lost its latest attempt to block companies from scraping information from its public pages, including member pages. ... but upon googling it, I found a more recent [1] ruling :/ > LinkedIn prevails in 6-year lawsuit against data scraper > The U.S. District Court for the Northern District of California sided with LinkedIn in its six year lawsuit against a firm that scraped data ... So that sure puts a nail into the argument I was going to make. But still, while I think your use case lines up with the spirit of this kind of system, I think the reality is that it also would be used by every single site with a signup wall to kill off the archive.ph's of the world. 0: https://www.zdnet.com/article/court-rules-that-data-scraping-is-legal-in-linkedin-appeal/ https://www.zdnet.com/article/court-rules-that-data-scraping... 1: https://iapp.org/news/a/linkedin-prevails-in-six-year-lawsuit-against-data-scraper/ https://iapp.org/news/a/linkedin-prevails-in-six-year-lawsui...
charcircuit 4y ago
Stopping ad fraud only hurts people trying to make money from it. Preventing web scraping is good for privacy. This is implemented in a privacy preserving way.
- nibbleshifter 4y ago
  > Preventing web scraping is good for privacy. You will need to elaborate on that.
  - gattilorenz 4y ago
    I assume the OP means that scraping is used to collect public data, including that of individuals, which can even be linked across different websites. There’s at least a couple of services that try to connect somebody’s Instagram account to their FB/Twitter/LinkedIn etc. I assume some of those rely on scraping (+username checking), since the TOS for the APIs of those social networks probably prohibit this use case.
    - charcircuit 4y ago
      Yes, big datasets of user data can be created and sold. This user data can be joined across multiple sites to build up profiles on people. These datasets floating around can harm the reputation of a site.
- userbinator 4y ago
  Preventing web scraping is good for privacy It's bad for freedom. Very, very very bad. Talking about the "privacy" of what has been made publicly available makes no sense.
  - charcircuit 4y ago
    >It's bad for freedom So are laws in the real world about stealing. >Talking about the "privacy" of what has been made publicly available makes no sense. Yes, it does. Users often wish to be able to delete or make something that was once public private. For example someone could post a picture of themselves on twitter. A year later they are no longer comfortable with having pictures of themself online so they go and delete them. Despite the user deleting them malicious scrapers will not delete them and keep those images. Another example would be setting your real name to your twitter name. Later you aren't comfortable using your real name so you change it away. Scrapers may still have your real name despite you wanting it to be a secret.

thataintright 4y ago

Why is the IETF letting the CIA influence the design of network protocols?

dane-pgp 4y ago
If you think that's bad, you're not going to like the fact that they're letting the NSA design a system for "TPM-based Network Device Remote Integrity Verification": https://www.ietf.org/archive/id/draft-ietf-rats-tpm-based-network-device-attest-14.html https://www.ietf.org/archive/id/draft-ietf-rats-tpm-based-ne...
RedShift1 4y ago
If you think that's bad, don't look up what happened with IPsec.