robots.txt being an honor system has always been a bit weird and search engine treatment is odder still. Google will still index your disallowed pages but refu

robots.txt being an honor system has always been a bit weird and search engine treatment is odder still. Google will still index your disallowed pages but refuse to show a description since they are couldn't read the page. The premise being that Google could have discovered your URLs from other pages and they don't consider a robots.txt disallow to indicate that you don't want them in search, just that you don't want Google to directly read the page. To actually disallow indexing of a page a page you need to allow it in robots and then add a noindex to the page itself. No one has really explained how these AI agents will treat robots.txt. Does it just mean that Bard/Gemini won't directly read the page but might somehow incorporate it into a dataset? For a test I asked Gemini "What is the URL of the String api documentation for Java 6?" which is disallowed by robots.txt. The response was: Unfortunately, Oracle does not maintain separate API documentation for older versions of Java like Java 6. The official documentation website (https://docs.oracle.com/javase/7/docs/api/ https://docs.oracle.com/javase/7/docs/api/) only provides documentation for the latest versions. Oracle has archieved some of the older Java API documentation but 6 is still available at https://docs.oracle.com/javase/6/docs/api/index.html https://docs.oracle.com/javase/6/docs/api/index.html. Funny enough the Java 7 API docs which Gemini linked too is also disallowed but maybe that happened after the model was trained.