3 ms·

This is my only fear about this, honestly. I already had to put my websites behind a login as that's the only realistic defense against LLM crawlers. If the da

This is my only fear about this, honestly. I already had to put my websites behind a login as that's the only realistic defense against LLM crawlers. If the data collection is moved to the browser, though, then requiring a login would no longer be adequate protection. I'd have to also ban the use of Chrome itself. I'd have to seriously consider the possibility of just not having a web presence in any form.

ArinaS 2y ago
Use Anubis - https://anubis.techaro.lol/ https://anubis.techaro.lol/.
- JohnFen 2y ago
  There's very little actual information about Anubis there. At least, I couldn't find much, but perhaps I missed some docs. I'm skeptical about the effectiveness of PoW solutions for this sort of thing, but I'm open to learning about new approaches. How would PoW be effective when the adversary is the user's browser itself and the user is already authenticated?
  - xena 2y ago
    Most scrapers don't run browsers. Making the scrapers run real browsers changes the economics of scraping and makes it less cost-effective. I am working on making it allow more traffic by default and then applying challenges based on request pressure or other factors like system load. I also need to finish the WebAssembly PR and a few other important things. It's a work in progress, but it's used by the United Nations so it can't be that bad :)
    - JohnFen 2y ago
      > Most scrapers don't run browsers. The specific risk I was talking about was that if OpenAI buys Chrome, they could (and I think it's likely they would) use the contents of whatever pages the users browse to as training for their models. Basically, turning the browser into a disguised crawler that would be immune from the usual anti-crawling methods, including putting up a login page.
      - ArinaS 2y ago
        So literally spyware. Don't get me wrong, current Chrome is also spyware, but sending every page you visit to a third-party is just a whole new level.

ArinaS 2y ago

Use Anubis - https://anubis.techaro.lol/ https://anubis.techaro.lol/.

JohnFen 2y ago
There's very little actual information about Anubis there. At least, I couldn't find much, but perhaps I missed some docs. I'm skeptical about the effectiveness of PoW solutions for this sort of thing, but I'm open to learning about new approaches. How would PoW be effective when the adversary is the user's browser itself and the user is already authenticated?
- xena 2y ago
  Most scrapers don't run browsers. Making the scrapers run real browsers changes the economics of scraping and makes it less cost-effective. I am working on making it allow more traffic by default and then applying challenges based on request pressure or other factors like system load. I also need to finish the WebAssembly PR and a few other important things. It's a work in progress, but it's used by the United Nations so it can't be that bad :)
  - JohnFen 2y ago
    > Most scrapers don't run browsers. The specific risk I was talking about was that if OpenAI buys Chrome, they could (and I think it's likely they would) use the contents of whatever pages the users browse to as training for their models. Basically, turning the browser into a disguised crawler that would be immune from the usual anti-crawling methods, including putting up a login page.
    - ArinaS 2y ago
      So literally spyware. Don't get me wrong, current Chrome is also spyware, but sending every page you visit to a third-party is just a whole new level.