3 ms·

How "innovative" do you expect robots.txt to be?

• r4vik 15y ago
  Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.