7 ms·
Archive.org's archiver is a "bad user agent" according to the list it uses, apparently: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-
by lights0123 6y ago
Archive.org's archiver is a "bad user agent" according to the list it uses, apparently: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list https://raw.githubusercontent.com/mitchellkrogza/nginx-ultim...
- andix 6y agoIt can be disabled (BLOCK_USER_AGENT=no), but it is on by default. Why would someone like to block crawlers and also TOR-clients? The malicious ones will be anyway disguised as normal browsers.
- sascha_sl 6y agoConsidering they quite explicitly ignore the nice way to say "don't archive me" (robots), yes, it is a bad user agent.
- samoa42 6y agoconsidering how many sites say "dont archive me" while residing as a public resource on the internet at the same time, i support this practical ingnorance.
- sascha_sl 6y agoI always think it's funny that a community that hates the AGPL for being too restrictive to corporations cause they definitely can't be expected to share their code in exchange for whatever they're taking also wants all takes, good or bad, that anyone has ever puts on the internet to be preserved.