3 ms·

How can you tell a bot will ignore all your content licenses?

bayindirh 1y ago
Currently all AI companies argue that the content they use falls under fair use, and disregard all licenses. This means any future ones respecting these licenses needs to be whitelisted.
- diggan 1y ago
  How do you know that that bot is part of those AI companies? Maybe it's my personal bot you're blocking, should I also not have (indirectly) access to the content?
  - bayindirh 1y ago
    You can use a honest user string denoting that it's your bot. Some AI companies label their bots transparently, they show up on the logs I keep. While I understand that you may need a personal bot to crawl or mirror a site, I can't guarantee that I'll grant you access. I don't like to be that heavy-handed in the first place, but capitalism is making it harder to trust entities which you can't see and talk face to face.
  - simianparrot 1y ago
    No. Access to my content is a privilege I grant you. I decide how you get to access it, and via a bot that my setup confuses for an AI crawler belonging to an anti-human AI corporation is not a valid way to access it. Get off my virtual lawn.
    - diggan 1y ago
      > No. Access to my content is a privilege I grant you. Right, I thought the conversation was about public websites on the public internet, but I think you're talking about this in the context of a private website now? I understand keeping tighter controls if you're dealing with private content you want accessible via the internet for others but not the public.
      - bayindirh 1y ago
        This interpretation won't take you that far. Crawling-prevention is not new. Many news outlets or biggish websites already was preventing access by non-human agents in various ways for a very long time. Now, non-human agents are improved and started to leech everything they can find, so the methods are evolving, too. News outlets are also public sites on the public internet. Source-available code repositories are also on the public internet, but said agents crawl and use that code, too, backed by fair-use claims.
      - privatelypublic 1y ago
        All websites are private (excepting maybe government sites). In most places the internet infrastructure itself is private. You're conflating a legal concept that applies to areas that are shared, government owned, paid for by taxes, and the government feels like people should be able to access them. The web is closer to a shopping mall. You're on one persons property to access other people's stuff who pay to be there. They set their own rules. If you don't follow those rules you get kicked out, charged with trespassing, and possibly banned from the mall entire. AI bots have been asked to leave. But, since they own the mall too, the store owners are more than a little screwed.
        diggan 1y ago
        > You're on one persons property to access other people's stuff who pay to be there. I see it more like I'm knocking on people's doors (issuing GET requests with my web browser) and people open their door for me (the server responds with something) or not. If you don't wanna open the door, fine you do you, but if you do open the door, I'm gonna assume it was on purpose as I'm not trying to be malicious, I'm just a user with a browser. > AI bots have been asked to leave. But, since they own the mall too, the store owners are more than a little screwed. I don't understand what you mean with this, what is the mall here, are you're saying that people have websites hosted at OpenAI et al? I'm not sure how the "mall owner" and the people running the AI bots are the same owners.
        privatelypublic 1y ago

Currently all AI companies argue that the content they use falls under fair use, and disregard all licenses. This means any future ones respecting these licenses needs to be whitelisted.

diggan 1y ago
How do you know that that bot is part of those AI companies? Maybe it's my personal bot you're blocking, should I also not have (indirectly) access to the content?
- bayindirh 1y ago
  You can use a honest user string denoting that it's your bot. Some AI companies label their bots transparently, they show up on the logs I keep. While I understand that you may need a personal bot to crawl or mirror a site, I can't guarantee that I'll grant you access. I don't like to be that heavy-handed in the first place, but capitalism is making it harder to trust entities which you can't see and talk face to face.
- simianparrot 1y ago
  No. Access to my content is a privilege I grant you. I decide how you get to access it, and via a bot that my setup confuses for an AI crawler belonging to an anti-human AI corporation is not a valid way to access it. Get off my virtual lawn.
  - diggan 1y ago
    > No. Access to my content is a privilege I grant you. Right, I thought the conversation was about public websites on the public internet, but I think you're talking about this in the context of a private website now? I understand keeping tighter controls if you're dealing with private content you want accessible via the internet for others but not the public.
    - bayindirh 1y ago
      This interpretation won't take you that far. Crawling-prevention is not new. Many news outlets or biggish websites already was preventing access by non-human agents in various ways for a very long time. Now, non-human agents are improved and started to leech everything they can find, so the methods are evolving, too. News outlets are also public sites on the public internet. Source-available code repositories are also on the public internet, but said agents crawl and use that code, too, backed by fair-use claims.
    - privatelypublic 1y ago
      All websites are private (excepting maybe government sites). In most places the internet infrastructure itself is private. You're conflating a legal concept that applies to areas that are shared, government owned, paid for by taxes, and the government feels like people should be able to access them. The web is closer to a shopping mall. You're on one persons property to access other people's stuff who pay to be there. They set their own rules. If you don't follow those rules you get kicked out, charged with trespassing, and possibly banned from the mall entire. AI bots have been asked to leave. But, since they own the mall too, the store owners are more than a little screwed.
      - diggan 1y ago
        > You're on one persons property to access other people's stuff who pay to be there. I see it more like I'm knocking on people's doors (issuing GET requests with my web browser) and people open their door for me (the server responds with something) or not. If you don't wanna open the door, fine you do you, but if you do open the door, I'm gonna assume it was on purpose as I'm not trying to be malicious, I'm just a user with a browser. > AI bots have been asked to leave. But, since they own the mall too, the store owners are more than a little screwed. I don't understand what you mean with this, what is the mall here, are you're saying that people have websites hosted at OpenAI et al? I'm not sure how the "mall owner" and the people running the AI bots are the same owners.
        privatelypublic 1y ago