3 ms·

Definitely overcomplicating things there. robots.txt already exists.

• hughesjj 3y ago
  If they haven't set up their landing page I doubt they configured a robots.txt
  • Chaosvex 3y ago
    This is clearly talking about default behaviours, so it'd already be there as standard. It's no different than expecting a different status code for the purpose of requesting crawlers not to index.
• weird-eye-issue 3y ago
  robots.txt deindexing a site by default is not necessarily the right choice. Lots of people are clueless about it and would never even think to change it when going live.