3 ms·
That is baffling logic. Sure, they think they know best and want to ignore the wishes of the owner of the web site. Why then respect a NOINDEX meta-tag instead
by cube00 4y ago
That is baffling logic. Sure, they think they know best and want to ignore the wishes of the owner of the web site. Why then respect a NOINDEX meta-tag instead of robots.txt?
- rsbadger 4y agoExactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.