4 ms·
Agreed. At best, you can disallow: / and hope they're polite enough to listen. I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they'r
by z3c0 4y ago
Agreed. At best, you can disallow: / and hope they're polite enough to listen.
I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they're considering robots.txt at all.
- 2OEH8eoCRo0 4y agoEven if they abide, this is capitalism. Somebody who wants an edge won't. Or OpenAI or Google will get desperate and stop abiding.