6 ms·

Well you could say same thing about the answers that Google displays on it's pages instead of search results! If you don't want these crawlers to index your con

Well you could say same thing about the answers that Google displays on it's pages instead of search results! If you don't want these crawlers to index your content I am pretty sure you can disable via robots.txt just like Google.

hobs 4y ago
It's the lack of attribution that really hurts, though I think its fairly shady of google to steal the ad revenue from smaller sites.
- Spooky23 4y ago
  You can ask for the source of information.
- theRealMe 4y ago
  You can ask ChatGPT to cite its sources.
  - jefftk 4y ago
    You can ask it to, but it will just make up sources. The connection between its knowledge and the original sources is not represented in the model. (This is an active area of research, though, and version of GPT that could cite its sources is something people widely agree would be valuable.)
altdataseller 4y ago
They still link to the source though. Even when they show a snippet.
2OEH8eoCRo0 4y ago
The idea that a robots.txt will save you is laughable.
- z3c0 4y ago
  Agreed. At best, you can disallow: / and hope they're polite enough to listen. I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they're considering robots.txt at all.
  - 2OEH8eoCRo0 4y ago
    Even if they abide, this is capitalism. Somebody who wants an edge won't. Or OpenAI or Google will get desperate and stop abiding.
- JohnFen 4y ago
  True. Robots.txt is already a very weak thing. I disallow all access using robots.txt, but there are many crawlers who ignore it and I have to maintain an overt blocklist for them.
ssharp 4y ago
The training model data sets have inconsistent respect for robots.txt. Also, I believe most of these models are not continuously crawling websites to update their data like a search engine does. That means if you're crawled once, you may not be crawled again and you'll still be in the datasets. I'd also argue that Google directing traffic to your website is a good alignment of incentives. ChatGPT spitting out answers derived from your work with nothing given back to you in return is not.
- tqwhite 4y ago
  I bet that fully half the time, I read the google answer, click on nothing and go on my way.
  - altdataseller 4y ago
    That's still better than 0%

hobs 4y ago

It's the lack of attribution that really hurts, though I think its fairly shady of google to steal the ad revenue from smaller sites.

Spooky23 4y ago
You can ask for the source of information.
theRealMe 4y ago
You can ask ChatGPT to cite its sources.
- jefftk 4y ago
  You can ask it to, but it will just make up sources. The connection between its knowledge and the original sources is not represented in the model. (This is an active area of research, though, and version of GPT that could cite its sources is something people widely agree would be valuable.)

altdataseller 4y ago

They still link to the source though. Even when they show a snippet.

2OEH8eoCRo0 4y ago

The idea that a robots.txt will save you is laughable.

z3c0 4y ago
Agreed. At best, you can disallow: / and hope they're polite enough to listen. I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they're considering robots.txt at all.
- 2OEH8eoCRo0 4y ago
  Even if they abide, this is capitalism. Somebody who wants an edge won't. Or OpenAI or Google will get desperate and stop abiding.
JohnFen 4y ago
True. Robots.txt is already a very weak thing. I disallow all access using robots.txt, but there are many crawlers who ignore it and I have to maintain an overt blocklist for them.

ssharp 4y ago

The training model data sets have inconsistent respect for robots.txt. Also, I believe most of these models are not continuously crawling websites to update their data like a search engine does. That means if you're crawled once, you may not be crawled again and you'll still be in the datasets. I'd also argue that Google directing traffic to your website is a good alignment of incentives. ChatGPT spitting out answers derived from your work with nothing given back to you in return is not.

tqwhite 4y ago
I bet that fully half the time, I read the google answer, click on nothing and go on my way.
- altdataseller 4y ago
  That's still better than 0%