Yes, the robots.txt file is for agents that cooperate, but voluntary cooperation is actually pretty important. Put that line in your robots.txt and disappear fr

Yes, the robots.txt file is for agents that cooperate, but voluntary cooperation is actually pretty important. Put that line in your robots.txt and disappear from Google. (Twitter specifically allows Google earlier in its robots.txt file.)