for the simple reason that many sites' robots.txt file is wrong. Which is of course not the real reason. The reason Google doesn't follow the robots.txt proto

for the simple reason that many sites' robots.txt file is wrong. Which is of course not the real reason. The reason Google doesn't follow the robots.txt protocol is (1) they don't want to (2) they can get away with it.