3 ms·

> Blocking bots that collect training data for AIs (and more) > In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to

by tivert 3y ago

> Blocking bots that collect training data for AIs (and more) > In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.) > I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl. Instead of blocking, it would be neater to present and alternative version to the crawlers (like many paywalled sites already to for SEO) that's full of dynamically generated LLM-generated garbage. That'll help the LLMs poison themselves.

JohnFen 3y ago
I was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient). My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.
- sebzim4500 3y ago
  So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation? You have a legal right to do this (assuming the LLM bullshit isn't libel) but I don't see how it could be considered a moral act.
  - JohnFen 3y ago
    > So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation? Yes, although I've been blocking googlebot for years, so nobody will get to my sites through google anyway. > I don't see how it could be considered a moral act. I'm curious about this -- why do you think this is in any way an immoral act? If a naïve human comes across the site, they'll quickly realize that it's not useful and move on. No harm done. How does morality enter into it? Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
    - sebzim4500 3y ago
      >If a naïve human comes across the site, they'll quickly realize that it's not useful and move on I do not share your confidence. >Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"? That would certainly help.
      - JohnFen 3y ago
        > That would certainly help. That seems reasonable enough. If I do this, I'll include such a disclaimer.
amf12 3y ago
> Instead of blocking, it would be neater to present and alternative version to the crawlers IIUC, if a site presents different (view of) content to the crawler than users, the site can get de-indexed.
- JohnFen 3y ago
  If that's a concern, and you're only worried about AI crawlers, then that's not a problem. Only provide the bogus pages to the AI crawlers, not to the search engine crawlers. Assuming there's a difference, anyway. I suspect that with Google and Bing, there isn't.

by tivert 3y ago

JohnFen 3y ago

I was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient). My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.

sebzim4500 3y ago
So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation? You have a legal right to do this (assuming the LLM bullshit isn't libel) but I don't see how it could be considered a moral act.
- JohnFen 3y ago
  > So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation? Yes, although I've been blocking googlebot for years, so nobody will get to my sites through google anyway. > I don't see how it could be considered a moral act. I'm curious about this -- why do you think this is in any way an immoral act? If a naïve human comes across the site, they'll quickly realize that it's not useful and move on. No harm done. How does morality enter into it? Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
  - sebzim4500 3y ago
    >If a naïve human comes across the site, they'll quickly realize that it's not useful and move on I do not share your confidence. >Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"? That would certainly help.
    - JohnFen 3y ago
      > That would certainly help. That seems reasonable enough. If I do this, I'll include such a disclaimer.

amf12 3y ago

> Instead of blocking, it would be neater to present and alternative version to the crawlers IIUC, if a site presents different (view of) content to the crawler than users, the site can get de-indexed.

JohnFen 3y ago
If that's a concern, and you're only worried about AI crawlers, then that's not a problem. Only provide the bogus pages to the AI crawlers, not to the search engine crawlers. Assuming there's a difference, anyway. I suspect that with Google and Bing, there isn't.