It would seem rather pointless to try and block any sort of crawler from crawling your site, if it's already been crawled by someone like the Internet Archive.

It would seem rather pointless to try and block any sort of crawler from crawling your site, if it's already been crawled by someone like the Internet Archive. Because all any future crawler needs to do is crawl through the various archives and plunder their content, rather than being so concerned about what live sites say that they can or cannot do directly.