4 ms·

LLMs are powered by web scraping. The same way Google and others have been crawling and capturing all your public posts for decades to power their search engin

by sbarre 2y ago

LLMs are powered by web scraping. The same way Google and others have been crawling and capturing all your public posts for decades to power their search engines. Now the data is being used to power LLMs. Were you able to opt out of being part of the search index (and I don't mean at the site level with a robots.txt file)? I think your choice here is "don't post on a publicly accessible website", unfortunately.

diggan 2y ago
> Were you able to opt out of being part of the search index (and I don't mean at the site level with a robots.txt file)? If you're in the EU, then yes, as "Right to be Forgotten" is a thing: https://en.wikipedia.org/wiki/Right_to_be_forgotten#European_Union https://en.wikipedia.org/wiki/Right_to_be_forgotten#European... But in general I agree, the expectation of something remaining "private" and "owned by you" after you publish it on the public internet, should be just about zero. Don't publish stuff you don't want others to read/store/redistribute/archive.
voidUpdate 2y ago
I manually opted in to being on the search index by submitting my website to google. I have never opted in to being part of an LLM dataset
- Daviey 2y ago
  Which search engines are you in that you didn't opt into?
  - voidUpdate 2y ago
    My website currently shows up on bing (which has an opt-out tool, I just haven't bothered to use it), duckduckgo (which scrapes its results from other engines) and yahoo search (which apparently scrapes from bing). I can't check Kagi as I'm not paying for it, and I can't currently think of other search engines off the top of my head
- sbarre 2y ago
  Maybe I misunderstood your original post, I thought you meant your comments here on HN, not a personal website you control. Others have said it already but when you are posting here on a public website, I would argue that you are effectively consenting that your content is now available for consumption by site visitors. "Site visitors" may include people, systems, software, etc.. I think it would be pretty impractical for every visitor to the site to have to seek consent from each poster before making use of the content. That would literally break the Internet.

by sbarre 2y ago

diggan 2y ago

> Were you able to opt out of being part of the search index (and I don't mean at the site level with a robots.txt file)? If you're in the EU, then yes, as "Right to be Forgotten" is a thing: https://en.wikipedia.org/wiki/Right_to_be_forgotten#European_Union https://en.wikipedia.org/wiki/Right_to_be_forgotten#European... But in general I agree, the expectation of something remaining "private" and "owned by you" after you publish it on the public internet, should be just about zero. Don't publish stuff you don't want others to read/store/redistribute/archive.

voidUpdate 2y ago

I manually opted in to being on the search index by submitting my website to google. I have never opted in to being part of an LLM dataset

Daviey 2y ago
Which search engines are you in that you didn't opt into?
- voidUpdate 2y ago
  My website currently shows up on bing (which has an opt-out tool, I just haven't bothered to use it), duckduckgo (which scrapes its results from other engines) and yahoo search (which apparently scrapes from bing). I can't check Kagi as I'm not paying for it, and I can't currently think of other search engines off the top of my head
sbarre 2y ago
Maybe I misunderstood your original post, I thought you meant your comments here on HN, not a personal website you control. Others have said it already but when you are posting here on a public website, I would argue that you are effectively consenting that your content is now available for consumption by site visitors. "Site visitors" may include people, systems, software, etc.. I think it would be pretty impractical for every visitor to the site to have to seek consent from each poster before making use of the content. That would literally break the Internet.