3 ms·
rsiqueira you're not missing much. Check out the live version :)
by bretthellman 15y ago
rsiqueira you're not missing much. Check out the live version :)
- pyre 15y agoProbably just wondering whether it was being squatted or if there was a legit business on there.
- bretthellman 15y agoIt was a corporation previosly
- gojomo 15y agoAre you specifically concerned about the site's past history being available, and/or do you want to block the current version from being archived for the future? To give a chance to the crawls which feed the Wayback Machine (and drop your de facto veto of showing any prior crawls), you could whitelist 'ia_archiver' and 'archive.org_bot'. To give a chance to up-and-coming search engines where competition is sorely needed, you could also whitelist 'ScoutJet' (for Blekko.com) and 'DuckDuckBot' (for DuckDuckGo.com).
- user24 15y agoyeah that's a pretty restrictive robots.txt