3 ms·
The ones I noticed in my logfiles behave impeccably: retrieve robots.txt every week or so and act on it. (I noticed Claude, OpenAI and a couple of others whose
by Arnt 2y ago
The ones I noticed in my logfiles behave impeccably: retrieve robots.txt every week or so and act on it.
(I noticed Claude, OpenAI and a couple of others whose names were less familiar to me.)