Sorry, but this is terrible advice. Yes, you should make sure your site won't break if it's slammed by a large crawl. Yes you should hide destructive actions

Sorry, but this is terrible advice. Yes, you should make sure your site won't break if it's slammed by a large crawl. Yes you should hide destructive actions behind posts, not gets. But, robots.txt is insanely useful. If I didn't have a robots.txt file, google/bing/yahoo would index countless repetitive non-important files and my site would suffer in search engine ranking. In our case, we host GPX/KML files and textual cuesheets for driving and biking. If that stuff is indexed, our sites' relevant keywords are "left", "right" and GPS timestamp fragments like "0z01". So, use it wisely, but don't abandon its usage altogether.