6 ms·
I see old stuff like this and it starts to become clear why the web is in tatters today. It may not be respected, but unless you have a really silly config (I'm
by rglover 1y ago
I see old stuff like this and it starts to become clear why the web is in tatters today. It may not be respected, but unless you have a really silly config (I'm hard-pressed to even guess what you could do short of a weird redirect loop), it won't be doing any harm.
> What this situation does, in fact, is cause many more problems than it solves - catastrophic failures on a website are ensured total destruction with the addition of ROBOTS.TXT.
Of course an archival pedant [1] will tell you it's a bad idea (because it makes their archival process less effective)—but this is one of those "maybe you should think for yourself and not just implement what some rando says on the internet" moments.
If you're using version control, running backups, and not treating your production env like a home computer (i.e., you're aware of the ephemeral nature of a disk on a VPS), you're fine.
[1] Archivists are great (and should be supported), but when you turn it into a crusade, you get foolish, generalized takes like this wiki.
- bigstrat2003 1y agoI really lost a lot of respect for the team when I read this page. No matter how good their intentions are, by deliberately ignoring robots.txt they are behaving just as badly as the various AI companies (and other similar entities) that scrape data against the wishes of the site owner. They are, in other words, directly contributing to the destruction of the commons by abusing trust and ensuring that everyone has to treat each other as a potential bad actor. Dick move, Archive Team.
- akk0 1y agoMind you're reading a 14 year old page. I honestly don't see any value in this being posted on HN.
- rafram 1y agoIt's unambiguously still the official position of the organization. Linked from their FAQ: https://wiki.archiveteam.org/index.php/Frequently_Asked_Questions https://wiki.archiveteam.org/index.php/Frequently_Asked_Ques...
- hyperpape 1y agoRegarding silly configurations: https://danluu.com/googlebot-monopoly/ https://danluu.com/googlebot-monopoly/.