Once upon a time I had to manually disable ROBOTS.txt parsing in one crawler just to stress test a staging machine for a friend. The lesson from this is that R

Once upon a time I had to manually disable ROBOTS.txt parsing in one crawler just to stress test a staging machine for a friend. The lesson from this is that ROBOTS.txt works as long as everyone follows a line set in sand.