3 ms·

Just to be clear, the data doesn't seem to be deleted -- it only becomes inaccessible once the site opts out; all the data can come back if the site changes its

by ent101 7y ago

Just to be clear, the data doesn't seem to be deleted -- it only becomes inaccessible once the site opts out; all the data can come back if the site changes its mind.

pishpash 7y ago
Does it also continue to archive while the 'do not archive' flag is on?
- ent101 7y ago
  I'm not sure, but I do think they follow the standard guidelines, meaning they won't archive if the flag is on. This can be verified by going through the Heritrix crawler's source code.