3 ms·

Does anyone have suggestions on what a proper robots.txt would be? How about: User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml

• bragr 2y ago
  That's a valid robots.txt, but "proper" is entirely dependent on what you want to achieve. If you aren't looking to treat different bots differently, and are looking allow all of your site to be indexed, then that is exactly what you want.
• deleted 2y ago
  [deleted]
• akira2501 2y ago
  The recommendation is to use an empty "Disallow:" rule rather than a catch all "Allow:" rule. Otherwise that is the canonical minimal example.
  • tiffanyh 2y ago
    Like this? User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
    • akira2501 2y ago
      Precisely.
      • turnsout 2y ago
        Why not just this: Sitemap: https://example.com/sitemap.xml Won't crawlers crawl by default?