4 ms·
The idea that a robots.txt will save you is laughable.
by 2OEH8eoCRo0 4y ago
The idea that a robots.txt will save you is laughable.
- z3c0 4y agoAgreed. At best, you can disallow: / and hope they're polite enough to listen. I can't seem to find anything on OpenAI's crawler agent, so I'm skeptical they're considering robots.txt at all.
- 2OEH8eoCRo0 4y agoEven if they abide, this is capitalism. Somebody who wants an edge won't. Or OpenAI or Google will get desperate and stop abiding.
- JohnFen 4y agoTrue. Robots.txt is already a very weak thing. I disallow all access using robots.txt, but there are many crawlers who ignore it and I have to maintain an overt blocklist for them.