2 ms·
Are you specifically concerned about the site's past history being available, and/or do you want to block the current version from being archived for the future
by gojomo 15y ago
Are you specifically concerned about the site's past history being available, and/or do you want to block the current version from being archived for the future?
To give a chance to the crawls which feed the Wayback Machine (and drop your de facto veto of showing any prior crawls), you could whitelist 'ia_archiver' and 'archive.org_bot'.
To give a chance to up-and-coming search engines where competition is sorely needed, you could also whitelist 'ScoutJet' (for Blekko.com) and 'DuckDuckBot' (for DuckDuckGo.com).
- user24 15y agoyeah that's a pretty restrictive robots.txt