2 ms·

Google uses the same crawler and robots.txt file for training data.

• inkysigma 11mo ago
  It's actually a different crawler for training data: Googlebot-extended so you can exclude yourself from the training data though not the search summaries.