3 ms·
Does it also continue to archive while the 'do not archive' flag is on?
by pishpash 7y ago
Does it also continue to archive while the 'do not archive' flag is on?
- ent101 7y agoI'm not sure, but I do think they follow the standard guidelines, meaning they won't archive if the flag is on. This can be verified by going through the Heritrix crawler's source code.