6 ms·

This is what happens if your robot isn't nice > curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com HTTP/2 403

• jamesog 2y ago
  That's not from robots.txt, but their Bot Management feature which blocks things calling themselves Googlebot that don't come from known Google IPs.
  • speedgoose 2y ago
    Are GCP IPs considered Google IPs?
    • crop_rotation 2y ago
      No I am very sure they are not.
    • jgrahamc 2y ago
      No.
    • judge2020 2y ago
      For reference https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...