5 ms·
> this would be like a crawler ignoring robots.txt Google ignores the noindex directive in robots.txt now. You're supposed to put it in your HTTP response head
by inportb 5y ago
> this would be like a crawler ignoring robots.txt
Google ignores the noindex directive in robots.txt now. You're supposed to put it in your HTTP response headers or HTML meta tags...
- ldjb 5y ago`noindex` was a Google-specific rule that was never officially documented nor supported. I think they were perfectly entitled to withdraw support for it, especially considering there are alternatives. https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt https://developers.google.com/search/blog/2019/07/a-note-on-...