3 ms·

> When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server. "If you

by jraph 1y ago

> When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server. "If you don't consent to me entering your house, change its logic so that picking the door's lock doesn't let me open the door" Yeah, well… As if the LLM scrappers didn't try anything under the sun like using millions of different residential IP to prevent admins from "changing the logic of the server" so it doesn't "return a response with a 200-series status code" when they don't agree to this scrapping. As if there weren't broken assumptions that make "When you return a response with a 200-series status code, you've granted consent" very false. As if technical details were good carriers of human intents.

ryandrake 1y ago
The locked door is a ridiculous analogy when it comes to the open web. Pretty much all "door" analogies are flawed, but sure let's imagine your web server has a door. If you want to actually lock the door, you're more than welcome to put an authentication gate around your content. A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way.
- jraph 1y ago
  Any analogy is flawed and you can kill most analogies very fast. They are meant to illustrate a point hopefully efficiently, not to be mathematically true. They are not to everyone's taste, me included in most cases. They are mostly fine as long as they are not used to make a point, but only to illustrate it. I agree with this criticism of this analogy, I actually had this flaw in mind from the start. There are other flaws I have in mind as well. I have developed more without the analogy in the remaining of the comment. How about we focus on the crux of the matter? > A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way The point is that these scrappers use tricks so that it's difficult not to grant them access. What is unreasonable here is to think that 200 means consent, especially knowing about the tricks. Edit: > you're more than welcome to put an authentication gate around your content. I don't want to. Adding auth so llm providers don't abuse my servers and the work I meant to share publicly is not a working solution.
  - jack_pp 1y ago
    here's my analogy, it's like you own a museum and you require entrance by "secret" password (your user agent filtering or what not). the problem is the password is the same for everyone so would you be surprised when someone figures it out or gets it from a friend and they visit your museum? Either require a fee (processing power, captcha etc) or make a private password (auth) It is inherently a cat and mouse game that you CHOOSE to play. Either implement throttling for clients that consume too much resources for your server / require auth / captcha / javascript / whatever whenever the client is using too much resources. if the client still chooses to go through the hoops you implemented then I don't see any issue. If u still have an issue then implement more hoops until you're satisfied.
    - jraph 1y ago
      > Either require a fee (processing power, captcha etc) or make a private password (auth) Well, I shouldn't have to work or make things worse for everybody because the LLM bros decided to screw us. > It is inherently a cat and mouse game that you CHOOSE to play No, let's not reverse the roles and blame the victims here. We sysadmins and authors are willing to share our work publicly to the world but never asked for it to be abused.
      - jack_pp 1y ago
        That's like saying you shouldn't have to sanitize your database inputs because you never asked for people to SQL inject your database. This stance is truly mind boggling to me
        jraph 1y ago
        Would you take the defense of attackers using SQL injections? Because it feels like people here, including you, are defending the llm scrapers against sysadmins and authors who dare share their work publicly. Ensuring basic security and robustness of a piece of software is simply not remotely comparable to countering the abuse these llm companies carry on. But it's not even the point. And preventing SQL injections (through healthy programming practices) doesn't make things worse for any legitimate user neither.

by jraph 1y ago

ryandrake 1y ago

The locked door is a ridiculous analogy when it comes to the open web. Pretty much all "door" analogies are flawed, but sure let's imagine your web server has a door. If you want to actually lock the door, you're more than welcome to put an authentication gate around your content. A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way.

jraph 1y ago
Any analogy is flawed and you can kill most analogies very fast. They are meant to illustrate a point hopefully efficiently, not to be mathematically true. They are not to everyone's taste, me included in most cases. They are mostly fine as long as they are not used to make a point, but only to illustrate it. I agree with this criticism of this analogy, I actually had this flaw in mind from the start. There are other flaws I have in mind as well. I have developed more without the analogy in the remaining of the comment. How about we focus on the crux of the matter? > A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way The point is that these scrappers use tricks so that it's difficult not to grant them access. What is unreasonable here is to think that 200 means consent, especially knowing about the tricks. Edit: > you're more than welcome to put an authentication gate around your content. I don't want to. Adding auth so llm providers don't abuse my servers and the work I meant to share publicly is not a working solution.
- jack_pp 1y ago
  here's my analogy, it's like you own a museum and you require entrance by "secret" password (your user agent filtering or what not). the problem is the password is the same for everyone so would you be surprised when someone figures it out or gets it from a friend and they visit your museum? Either require a fee (processing power, captcha etc) or make a private password (auth) It is inherently a cat and mouse game that you CHOOSE to play. Either implement throttling for clients that consume too much resources for your server / require auth / captcha / javascript / whatever whenever the client is using too much resources. if the client still chooses to go through the hoops you implemented then I don't see any issue. If u still have an issue then implement more hoops until you're satisfied.
  - jraph 1y ago
    > Either require a fee (processing power, captcha etc) or make a private password (auth) Well, I shouldn't have to work or make things worse for everybody because the LLM bros decided to screw us. > It is inherently a cat and mouse game that you CHOOSE to play No, let's not reverse the roles and blame the victims here. We sysadmins and authors are willing to share our work publicly to the world but never asked for it to be abused.
    - jack_pp 1y ago
      That's like saying you shouldn't have to sanitize your database inputs because you never asked for people to SQL inject your database. This stance is truly mind boggling to me
      - jraph 1y ago
        Would you take the defense of attackers using SQL injections? Because it feels like people here, including you, are defending the llm scrapers against sysadmins and authors who dare share their work publicly. Ensuring basic security and robustness of a piece of software is simply not remotely comparable to countering the abuse these llm companies carry on. But it's not even the point. And preventing SQL injections (through healthy programming practices) doesn't make things worse for any legitimate user neither.