3 ms·

I'm not so sure that even Google respects it. I did some digging about the semantics about robots.txt whilst writing a bot myself, and it seems that Google does

I'm not so sure that even Google respects it. I did some digging about the semantics about robots.txt whilst writing a bot myself, and it seems that Google doesn't follow links that are excluded, but it will visit those pages. Maybe that counts as "paying attention", but I don't think they "respect" it.

butler14 10y ago
they respect it, but because it's so frequently misused or plain broken, it's basically sidelined vs. more optimal methods for preventing indexation, or getting an already-indexed piece of content removed, such as the noindex tag.