3 ms·
If it's on their own site, that wouldn't be a problem. IIRC, archive.org stops serving pages that later appear in a robots.txt file.
by jimbobthrowawy 2y ago
If it's on their own site, that wouldn't be a problem. IIRC, archive.org stops serving pages that later appear in a robots.txt file.
- codexon 2y agoIt could be on a website they don't control like twitter.