4 ms·
If they haven't set up their landing page I doubt they configured a robots.txt
by hughesjj 3y ago
If they haven't set up their landing page I doubt they configured a robots.txt
- Chaosvex 3y agoThis is clearly talking about default behaviours, so it'd already be there as standard. It's no different than expecting a different status code for the purpose of requesting crawlers not to index.