4 ms·

I'd rather let Archive block access to that specific article for a while, but still archiving from the start.

• ajb 5mo ago
  In effect, robots.txt should have an "embargo" directive?
  • Bender 5mo ago
    I dont know if this is still the case but if I told IA via robots.txt not to archive my site, it would still crawl it, archive it but not display it until I shut the site down. Once robots.txt was no longer reachable they would display the archived content. The only way to stop that was to start the site back up making robots.txt reachable and wait for them to crawl it again.
  • Ma8ee 4mo ago
    I like that idea.