3 ms·
Unless I’m mistaken, google don’t actually respect robots.txt any more. They recommend the use of some meta tag instead, from memory.
by mxxx 6y ago
Unless I’m mistaken, google don’t actually respect robots.txt any more. They recommend the use of some meta tag instead, from memory.
- yarcob 6y agoTheir docs say they do: https://developers.google.com/search/reference/robots_txt https://developers.google.com/search/reference/robots_txt They do say that they ignore robots.txt for user-initiated actions eg. Google Translate, which makes sense in my opinion.
- mxxx 6y agoAh yes you’re right, my apologies. They don’t support the noindex directive inside a robots.txt any more.