3 ms·
Google specifically warns that having a Noindex header doesn't work if there is robots.txt disallow (since the crawler never sees the noindex, since it obeys th
by detaro 8y ago
Google specifically warns that having a Noindex header doesn't work if there is robots.txt disallow (since the crawler never sees the noindex, since it obeys the disallow), that's why I asked.
- r1ch 8y agoThere's a non standard Noindex: /URL that Google accepts in robots.txt. Unfortunately it breaks many other crawlers which don't understand it, so you have to rely on user agent sniffing.