3 ms·
IA's spider respects META tags in the HTML right? Quora should prevent archiving of a page using the tags until maybe a couple months after the last person has
by swang 12y ago
IA's spider respects META tags in the HTML right? Quora should prevent archiving of a page using the tags until maybe a couple months after the last person has responded, then allow the page to be archived by removing any META tags related to archiving.
AFAIK, noarchive still allows indexing of the page, which means the robot should realize the page is there and archive it during its next run.