3 ms·What about for their LLM products? We know that OpenAi does not respect the robots.txt fileby jppope 10mo agoWhat about for their LLM products? We know that OpenAi does not respect the robots.txt filexnx 10mo agoGoogle uses the same crawler and robots.txt file for training data.inkysigma 10mo agoIt's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.