3 ms·
By that logic IE could send the entire page of search results back to Microsoft and it wouldn't be a violation of robots.txt. Because hey, it was a user-agent
by haberman 16y ago
By that logic IE could send the entire page of search results back to Microsoft and it wouldn't be a violation of robots.txt. Because hey, it was a user-agent doing it, not a robot! If this is so in the clear, why not have Microsoft do that also?