5 ms·
If they don't want to be crawled, they could just change the robots.txt. But nobody does.
by sek 14y ago
If they don't want to be crawled, they could just change the robots.txt. But nobody does.
- dagw 14y agoAlmost nobody: http://www.guardian.co.uk/media/greenslade/2012/oct/22/google-brazil http://www.guardian.co.uk/media/greenslade/2012/oct/22/googl... I'm looking forward to see in a years time what the result of this little experiment is.