3 ms·

My understanding is that they will disallow access to a webpage if a robots.txt goes up but they will keep the copy of the webpage in case the robots.txt is eve

by af16090 10y ago

My understanding is that they will disallow access to a webpage if a robots.txt goes up but they will keep the copy of the webpage in case the robots.txt is ever changed[1]. So a future administration could change the robots.txt and the webpages would be accessible again. [1]: Seems like that happened here: https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v._Chordiant_Software_Inc https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v.....

hunter2_ 10y ago
I know robots.txt is all about respect in the first place, but is it really expected that obtained-via-robot mirrors will be made inaccessible to humans during periods of time when the upstream prohibits robotic fetching following that original fetch? It seems much more reasonable that the mirror would merely prohibit robots, not humans, from fetching its copy during periods of time when the upstream prohibits robots.
- CM30 10y ago
  I suspect the original thought was that a website owner might later decide they don't want their site publically viewable in the Internet Archive, and would sue/complain if it was still visible there. Like say, if they accidentally allowed robots to archive a 'hidden' directory. Problem is, those use cases are greatly outweighed by ones where either: 1. The site has changed ownership to a different (legitimate) company or organisation, and they don't want their new site archived. 2. A domain squatter/seller has bought the domain, blocked all bots to stop the holding page being indexed in Google and accidentally blocked the archive in the process. 3. A technician or developer has accidentally blocked the archive/all robots due to a personal mistake/copying code from the internet. 4. The site owner doesn't know the archive has a bot, and has blocked all non Google/Bing bots to 'reduce strain' on the server. That last one is depressingly common: http://webmasters.stackexchange.com/questions/75993/only-allow-google-and-bing-bots-to-crawl-a-site http://webmasters.stackexchange.com/questions/75993/only-all...

by af16090 10y ago

hunter2_ 10y ago

I know robots.txt is all about respect in the first place, but is it really expected that obtained-via-robot mirrors will be made inaccessible to humans during periods of time when the upstream prohibits robotic fetching following that original fetch? It seems much more reasonable that the mirror would merely prohibit robots, not humans, from fetching its copy during periods of time when the upstream prohibits robots.

CM30 10y ago
I suspect the original thought was that a website owner might later decide they don't want their site publically viewable in the Internet Archive, and would sue/complain if it was still visible there. Like say, if they accidentally allowed robots to archive a 'hidden' directory. Problem is, those use cases are greatly outweighed by ones where either: 1. The site has changed ownership to a different (legitimate) company or organisation, and they don't want their new site archived. 2. A domain squatter/seller has bought the domain, blocked all bots to stop the holding page being indexed in Google and accidentally blocked the archive in the process. 3. A technician or developer has accidentally blocked the archive/all robots due to a personal mistake/copying code from the internet. 4. The site owner doesn't know the archive has a bot, and has blocked all non Google/Bing bots to 'reduce strain' on the server. That last one is depressingly common: http://webmasters.stackexchange.com/questions/75993/only-allow-google-and-bing-bots-to-crawl-a-site http://webmasters.stackexchange.com/questions/75993/only-all...