4 ms·
Seems like a sensible move to tell crawlers there isn't a page there so don't bother trying to index it. Probably keeps their logs cleaner and reduces the reque
by cs02rm0 13y ago
Seems like a sensible move to tell crawlers there isn't a page there so don't bother trying to index it. Probably keeps their logs cleaner and reduces the requests they have to serve.
I find it slightly odd that the internet archive then drop historical pages, but the jump some are making to this being Orwellian I find a leap too far.