I was going to make a joke about just using wget --mirror http://en.wikipedia.org/wiki/Main_Page so I decided to try it first. After downloading the robots.

I was going to make a joke about just using wget --mirror http://en.wikipedia.org/wiki/Main_Page so I decided to try it first. After downloading the robots.txt, it stopped. I looked and saw the following: # # Sorry, wget in its recursive mode is a frequent problem. # Please read the man page and use it properly; there is a # --wait option you can use to set the delay between hits, # for instance. # User-agent: wget Disallow: /