19 ms·

Robots.txt meant for search engines don’t work well for web archives

295 points by r721 10y ago

laumars 10y ago
To be honest I see robots.txt as a failed experiment since it relies on trust rather than security or thoughtful design.
- hdhzy 10y ago
  I don't think it's about security. For example I've got a link to do delegated login like /login-with/github. When people click it an oauth flow will start. But it is useless for robots to follow so I disallow it in robots.txt. If they still follow nothing breaks and it's not a security issue but if I can avoid starting unnecessary oauth logins it's an additional benefit.
  - laumars 10y ago
    robots.txt wasn't created for security but it can have security implications if you publish a list of Disallow paths with the intention of hiding sensitive content (sadly I have that seen that happen a lot) where as a better approach would be IP whitelisting and/or user authentication. However I'm not claiming security is the only reason people use (misuse?) robots.txt. For example in your case you could mitigate your need for a robots.txt with a nofollow attribute[1]. Sure bad bots could still crawl your site and find the authentication URL without probing robots.txt so the security implications there is pretty much non-existent. But you've already got a thoughtful design (the other point I raised) that mitigates the need for robots.txt anyway so adding something like "nofollow" maybe enough to remove the robots.txt requirement altogether. [1] https://en.wikipedia.org/wiki/Nofollow https://en.wikipedia.org/wiki/Nofollow
    - dchest 10y ago
      This is crazy, that's not what robots.txt is for. How can you complain about the security of a thing that is not meant to provide security? According to your logic, newspapers are a "failed experiment because they rely on trust rather than security or thoughtful design". I published an article with my treasure map and told people not to go there, but they stole it.
      - laumars 10y ago
        That was an anecdote since the previous poster raised the point about security. I'm definitely not claiming robots.txt should be for security nor was designed for security! I said following proper security and design practices renders obsolete all the edge cases that people might use robots.txt. I'm saying if you design your site properly then you shouldn't really need a robots.txt. That applies for all examples that HN commentators have raised in terms of their robots.txt usage thus far. I would rewrite my OP to make my point clearer but sadly I no longer have the option to edit it.
        dchest 10y ago
        design your site properly then you shouldn't really need a robots.txt But how? For example, if you don't want a page to be indexed by Google, you add this information to robots.txt. Nofollow doesn't work for every case, because any external website can link to it, and Google will discover it.
        laumars 10y ago
        That's a good point. I'm not sure how you'd get around non-HTML documents (eg PDFs) but web pages themselves can be excluded via a meta tag: <meta name="robots" content="noindex"> Source: https://support.google.com/webmasters/answer/93710?hl=en https://support.google.com/webmasters/answer/93710?hl=en Interestingly in that article, there is the following disclaimer about not using robots.txt for your example: "Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it." I must admit even I hadn't realised that could happen and I was critical of the use robots.txt to begin with.
        dchest 10y ago
        Ah, that's true, indeed. The page, though, will appear as a link without any contents, because the bot won't be able to index it.
    - bigbugbag 10y ago
      > robots.txt wasn't created for security but it can have security implications if you publish a list of Disallow paths with the intention of hiding sensitive content Using robots.txt to secure your server from bots is the equivalent of attempting to secure your house from robbery by planting a sign that says "please,don't rob my house". Surprisingly it may works from time to time, by if you're into attempting security by wishful thinking maybe don't be too surprised when it fails about as much as security by chance.
      - laumars 10y ago
        I know. With the greatest of respect your counter argument is literally just reiterating the point I was making. Albeit in the quote you've left off the part of my post where I said it's stupid to use robots.txt in this way.
    - hdhzy 10y ago
      Note that links marked with Nofollow can still be followed by well-behaving bots: https://en.wikipedia.org/wiki/Nofollow#Interpretation_by_the_individual_search_engines https://en.wikipedia.org/wiki/Nofollow#Interpretation_by_the...
- kakarot 10y ago
  I came here to say something about respecting the wishes of others, etc, but you know what? You're absolutely right. We shouldn't even need to have a conversation about trust and respect. It should be non-negotiable if you don't want your personal contents indexed by scrapers and archivers, and it should be enforced by design. It's a broken system.
- nabla9 10y ago
  robots.text is not a security tool. It's communication tool that gives advice. Just like sitemaps. If you need to add security (logins) to protect content you don't need to protect you inconvenience users.
  - laumars 10y ago
    I'd already covered the security point replying to another poster (https://news.ycombinator.com/item?id=14163792 https://news.ycombinator.com/item?id=14163792) but just to be clear, I'm absolutely not claiming robots.txt is a security tool. Quite the opposite, I saying following good security and design practices renders the robots.txt file obsolete. Your point about sitemaps helps illustrate that point of mine because having a decent sitemap mitigates the need for Allow lines in robots.txt. It's another feature of the web where robots.txt isn't well equipped to handle and thus there have been other, better, tools built to highlight pages of interest to search engines.
- bkor 10y ago
  Lots of laws are pretty similar. e.g. technically you could steal loads of things. Practically you don't. Defeating/ignoring mechanisms such as robots.txt (vs maybe some security person in a store) still makes stealing not ok.
  - laumars 10y ago
    The morality of whether bots should obey robots.txt is a separate issue to the point I raised about how you shouldn't trust bots to obey them. To use your example of high street stores: shops have security tags on expensive items / clothing as a method of securing products from theft because you cannot blindly trust everyone not to steal (though wouldn't it be great if that wasn't the case). Equally websites cannot trust that bots will obey robots.txt. Which means any content that doesn't want to be crawled needs to be behind nofollow attributes or (if it's sensitive) user authentication layers and any content that does need to be indexed also needs to be in a sitemap. Once you have all of these extra layers implemented, the robots.txt becomes utterly redundant. Hence why I say it's a failed experiment. The benefits it offers are superseded by better solutions.
- bigbugbag 10y ago
  https://en.wikipedia.org/wiki/Robots_exclusion_standard#Security https://en.wikipedia.org/wiki/Robots_exclusion_standard#Secu... robots.txt was proposed after a badly behaved bot DoSed a web server 20+ years ago, those were different times. With the robots.txt standed now those who wants to play nice can do so without asking anything, for the badly behaved ones it's still up to the admin to put forward the appropriate measures.
  - laumars 10y ago
    Wow has it really been more than 20 years!?? I feel old now... I do get what you're saying but if you have to implement "appropriate measures" anyway then the robots.txt file becomes completely redundant.
mushiake 10y ago
fastastic news. Archive Team's take on this[0] [0]http://www.archiveteam.org/index.php?title=Robots.txt http://www.archiveteam.org/index.php?title=Robots.txt
- rz2k 10y ago
  The logic is sound, and I see that it was mostly written in 2011, but I can also see it being harmful. How about an IETF RFC to clarify? Libraries operate under a lot of unwritten social conventions, perhaps even more than most other institutions. (robots.txt even if largely ignored is a popular convention) Aggressive or confrontational wording, regardless of whether they are "right" doesn't seem in libraries' interests.
- dingaling 10y ago
  It is great news in general, but seems to be done in a clumsy and counterproductive manner that may cause the Internet Archive to be banned from crawling some websites. The problem: when robots.txt for a website is found to have been made more restrictive, the IA retrospectively applies its new restrictions to already-archived pages and hides them from view. This can also cause entire domains to vanish into the deep-archive. No-one outside IA thinks this is sensible. Their solution: ignore robots.txt altogether. What? That will just annoy many website operators. My proposed solution: keep parsing robots.txt on each crawl and obey it progressively, without applying the changes to existing archived material. This is actually less work than what they currently do. If the new robots.txt says to ignore about_iphone.html you just do that and ignore it. Older versions aren't affected. Basically they're switching from being excessively obedient to completely ignoring robots.txt in order to fix a self-made problem. I can only see that antagonising operators.
  - duskwuff 10y ago
    There's some value in allowing site operators to retroactively remove content which was never intended to be public. A common and unfortunate example is backups (like SQL dumps) being stored in web-accessible directories, then subseqently being indexed and archived when a crawler finds the appropriate directory index. What needs to be fixed first is just the really common case mentioned in the blog post, where a domain changes ownership and a restrictive robots.txt is applied to the parking page.
libeclipse 10y ago
On the linked page, I see comments about ignoring the webmasters' wishes et al. All I can say is f*ck that. It's a free and open internet. If you put content up on a public site, anyone has the right to go and look at it. Stop complaining when someone saves it. And sure some people complain that scrapers slow down their site and that's why they use robots.txt, but really? Really? It's 2017 and your site is affected by that. I think you have bigger things to worry about.
- chii 10y ago
  > scrapers slow down their site and that's why they use robots.txt a poorly written scraper may really slow down your site, especially if it wasn't intended to be scrapped repeatedly. There should be something to be said about frequency which scrapers should follow (specified by the website owner via a robots.txt like spec). But website owners cannot demand unreasonable frequencies (such as once a year!), and what constitutes unreasonable is up for debate.
  - davb 10y ago
    Additionally, if excessive scraping became an issue for my site I'd consider rate limiting client.
  - timClicks 10y ago
    The Crawl-delay directive is the de facto standard for this.
  - eknkc 10y ago
    I don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.
  - dingo_bat 10y ago
    > (specified by the website owner via a robots.txt like spec). Nope, if a website wants such a restriction, it must enforce it. Robots.txt is a request. It's worthless.
droithomme 10y ago
I have some sites where I specifically block archiving from some sections for good reason. (Even if I didn't have a good reason though it would still be my choice.) I have a very big problem with them disregarding robots directives. Sure some crawlers ignore them: Hostile net actors up to no good. This decision means they are a hostile net actor. I'll have to take extreme measures such as determining all the ip address ranges they use and totally blocking access. This inconveniences me, which means they are now my enemy. edit- For those interested: Deny from 207.241.224.0/22
- cprecioso 10y ago
  Can I ask what the good reason is?
- omgtehlion 10y ago
  I have easier solution for you: just shut down your site and be done with it. This way no malicious actor will be able to save your precious information.
- andrius4669 10y ago
  Why not just block ia_archiver useragent in your web server for these paths instead? Also, I'm curious, what that good reason could be?
- Asparagirl 10y ago
  Are you under the impression that individual web archivists don't also scrape websites of interest, and submit those WARC's for inclusion into the Wayback Machine, independent of the IA's crawlers? Because believe me, we do...good luck banning every AWS and DO IP range.
  - tomjen3 10y ago
    I actually didn't know that. Do you operate the same crawlers? I have considered putting a single file that is only accessible via no-follow links and perma-ban any ip that access the file, as a way to punish bad robots.
metafunctor 10y ago
It appears that IA applies (or did apply) a new version of robots.txt to pages already in their index, even if they were archived years ago. That's silly, and stopping doing that would probably solve much of this problem.
- LoSboccacc 10y ago
  yeah it's not even hard to solve of a problem, just use the archived version of robots.txt that matches the crawl date too bad they already lost loads of internet content that way
  - tempay 10y ago
    What should happen in the case that a website misconfigures robots.txt and ends up wanting to remove private data? I think I would be tempted to say that the data can't be removed to avoid abuse from future domain owners (or current ones) but I'm not sure if there would be any legal consequences of this attitude.
    - LoSboccacc 10y ago
      Same when an archive crawl illegal/copyrigthed data, a pathway for that needs to exists anyway
    - CM30 10y ago
      Provide a content removal form? It works for DMCA notices, it can work here. Maybe even have a 'reason' textbox to see why someone may want content removed...
      - makomk 10y ago
        Of course, there's the inevitable risk that the Internet Archive's newfound control over who is allowed to make their past disappear into the memory hole and who has it archived forever will be used for political ends, especially since the ability to manually archive pages is already used this way by staff. (Take a look at Jason Scott's Twitter or that of the Archive Team sometime - lots of conspicuous manual archiving of stuff that's embarrassing to a certain US political party.) The issue of curators' views biasing the contents of collections seems to be underappreciated in general in the digital age, for some reason.
sengork 10y ago
Found this[1] via the Wikipedia's Talk page for the robots.txt article. It showcases that early on robots.txt was designated to help maintain bandwidth performance of web servers. Back then it would have been due to bandwidth contention, today it may be bandwidth cost to some operators which robots.txt help mitigate. [1] https://yro.slashdot.org/comments.pl?sid=377285&cid=21554125 https://yro.slashdot.org/comments.pl?sid=377285&cid=21554125
yeukhon 10y ago
I remember writing a dumb parser for robots.txt. I have to agree, robots.txt is simplistic but so non-standard. I wonder why search engines can't just say NO to this. Does search engines today still honor robots.txt? Here's my shameless plug: https://github.com/yeukhon/robots-txt-scanner https://github.com/yeukhon/robots-txt-scanner I still remember writing most of this on Caltrain one morning heading to SF visiting someone I dearly loved.....
afandian 10y ago
I'm not so sure that even Google respects it. I did some digging about the semantics about robots.txt whilst writing a bot myself, and it seems that Google doesn't follow links that are excluded, but it will visit those pages. Maybe that counts as "paying attention", but I don't think they "respect" it.
- butler14 10y ago
  they respect it, but because it's so frequently misused or plain broken, it's basically sidelined vs. more optimal methods for preventing indexation, or getting an already-indexed piece of content removed, such as the noindex tag.
Aissen 10y ago
Finally. A bit late, since a lot of the archive has been removed because of new owner's aggressive (or malicious) new robots.txt
- pjc50 10y ago
  Hidden, rather than removed.
  - Aissen 10y ago
    I hope so.
amelius 10y ago
I think we should write down a legal license in our robots.txt file, as a retribution for all those lengthy EULAs these big companies make us read :)
madshiva 10y ago
Yeah just ignore robots.txt because there's others solution. If the site don't want to be scanned they can adopt a lot of counter measure and robots.txt will not save it from abuse. He remind me the old days when my website wasn't working from US because I just fake that the site was down because there's no reason that somebody goes to my site from US (I know it's kind stupid, but when all your content is in french and you are a kid... :) )
rubatuga 10y ago
One thing that should be considered is the right for an individual to be forgotten.
- freshhawk 10y ago
  Forgotten by whom? Who judges what is "forgotten" and when? How is it enforced? The specifics here matter a great deal, the versions so far are regularly abused by the wealthy and don't apply to any of the data warehouses that the powerful and well connected have access to. Where did this "right" come from? What's the legal and ethical basis for it? It is analogous to censorship or book burning at the basic level, destroying information to hide it from the public. It requires a consistent and strong justification as well as justified limited scope because of that, and it better be obviously beneficial to society even accounting for the inevitable misuse by those in power.
  - beejiu 10y ago
    There is a legal basis in the European Union. https://en.wikipedia.org/wiki/Right_to_be_forgotten https://en.wikipedia.org/wiki/Right_to_be_forgotten
    - merb 10y ago
      and the legal basis is pretty fluffy: - http://www.sueddeutsche.de/digital/bgh-grundsatzurteil-namen-der-sedlmayr-moerder-bleiben-online-1.144087 http://www.sueddeutsche.de/digital/bgh-grundsatzurteil-namen... - http://www.focus.de/digital/internet/bgh-urteil-keine-staendige-kontrolle-von-online-archiven_aid_463464.html http://www.focus.de/digital/internet/bgh-urteil-keine-staend... it's german, but basically it says: "This is not a blank check".
- TeMPOraL 10y ago
  In my (current) opinion, it's this law that should be forgotten. What's on the public Internet is a matter of public interest. All I can see is this law being used by bad people to hide their bad deeds, especially when those bad deeds should be known.
deleted 10y ago
[deleted]
TeMPOraL 10y ago
Archive.org needs to be able to apply to itself. We could then use archive.org to view how archive.org in the past viewed some interesting site, thus avoiding the whole retroactive robots.txt fail. ;).
6d6b73 10y ago
There should be a way to direct archiving bots to a file that has the newest, compressed version of the website for them to download. Wouldn't that be easier for everyone?
- BHSPitMonkey 10y ago
  Seems like it would just get abused with false content
  - 6d6b73 10y ago
    True, but maybe it would be a good option for non-commercial websites that would like to get archived, and make the archiving more efficient for themselves and archive.org
EvanAnderson 10y ago
The policy the Internet Archive applies re: "robots.txt" comes from an archive policy created at U.C. Berkeley in the early 2000's (The Oakland Archive Policy - http://www2.sims.berkeley.edu/research/conferences/aps/removal-policy.html http://www2.sims.berkeley.edu/research/conferences/aps/remov...). Jason Scott (an employee of the Internet Archive) mentioned that the Archive doesn't ever delete anything. He stated that items may be removed from public access because of changes to "robots.txt" but they're not actually deleted. (That's a little comforting, at least.)
dbg31415 10y ago
I think the biggest argument for honoring robots.txt is that sites, especially old sites, can have a lot of really highly resource intensive pages. I don't want someone crawling a page that has 800+ DB calls... for example. Yes, I should optimize the page, or whatever... but really it may be useful for that admin, or 1 out of 10,000 users, who uses the page. It's not ideal to have someone crawl all those pages at once. I think they should honor robots.txt, and the meta tag version on specific pages or links -- given the site publisher went out of their way to give instructions to crawlers it seems reasonable to honor those requests.
cosinetau 10y ago
For whatever it's worth: http://humanstxt.org/ http://humanstxt.org/
c0achmcguirk 10y ago
In the 90s I spent a lot of time on my website and I loved learning how web crawlers worked. I started using a robots.txt file without really understanding it. I ended up blocking everything thinking it would make my site faster for visitors because--crawlers might crawl the site all the time. After I graduated from college I lost access to my website which was hosted on the Computer Science department's web servers. I wish I hadn't used that robots.txt file. I would love to find the pages I made that compared interfold vs. exterfold staple strength, or the site I made with a ranch theme with a cowboy that had humorous advice....I don't have any content in archive.org because it honored the robots.txt file. ...sigh...wish I had backed up my stuff.
deleted 10y ago
[deleted]

295 points by r721 10y ago

To be honest I see robots.txt as a failed experiment since it relies on trust rather than security or thoughtful design.

hdhzy 10y ago
I don't think it's about security. For example I've got a link to do delegated login like /login-with/github. When people click it an oauth flow will start. But it is useless for robots to follow so I disallow it in robots.txt. If they still follow nothing breaks and it's not a security issue but if I can avoid starting unnecessary oauth logins it's an additional benefit.
- laumars 10y ago
  robots.txt wasn't created for security but it can have security implications if you publish a list of Disallow paths with the intention of hiding sensitive content (sadly I have that seen that happen a lot) where as a better approach would be IP whitelisting and/or user authentication. However I'm not claiming security is the only reason people use (misuse?) robots.txt. For example in your case you could mitigate your need for a robots.txt with a nofollow attribute[1]. Sure bad bots could still crawl your site and find the authentication URL without probing robots.txt so the security implications there is pretty much non-existent. But you've already got a thoughtful design (the other point I raised) that mitigates the need for robots.txt anyway so adding something like "nofollow" maybe enough to remove the robots.txt requirement altogether. [1] https://en.wikipedia.org/wiki/Nofollow https://en.wikipedia.org/wiki/Nofollow
  - dchest 10y ago
    This is crazy, that's not what robots.txt is for. How can you complain about the security of a thing that is not meant to provide security? According to your logic, newspapers are a "failed experiment because they rely on trust rather than security or thoughtful design". I published an article with my treasure map and told people not to go there, but they stole it.
    - laumars 10y ago
      That was an anecdote since the previous poster raised the point about security. I'm definitely not claiming robots.txt should be for security nor was designed for security! I said following proper security and design practices renders obsolete all the edge cases that people might use robots.txt. I'm saying if you design your site properly then you shouldn't really need a robots.txt. That applies for all examples that HN commentators have raised in terms of their robots.txt usage thus far. I would rewrite my OP to make my point clearer but sadly I no longer have the option to edit it.
      - dchest 10y ago
        design your site properly then you shouldn't really need a robots.txt But how? For example, if you don't want a page to be indexed by Google, you add this information to robots.txt. Nofollow doesn't work for every case, because any external website can link to it, and Google will discover it.
        laumars 10y ago
        That's a good point. I'm not sure how you'd get around non-HTML documents (eg PDFs) but web pages themselves can be excluded via a meta tag: <meta name="robots" content="noindex"> Source: https://support.google.com/webmasters/answer/93710?hl=en https://support.google.com/webmasters/answer/93710?hl=en Interestingly in that article, there is the following disclaimer about not using robots.txt for your example: "Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it." I must admit even I hadn't realised that could happen and I was critical of the use robots.txt to begin with.
        dchest 10y ago
        Ah, that's true, indeed. The page, though, will appear as a link without any contents, because the bot won't be able to index it.
  - bigbugbag 10y ago
    > robots.txt wasn't created for security but it can have security implications if you publish a list of Disallow paths with the intention of hiding sensitive content Using robots.txt to secure your server from bots is the equivalent of attempting to secure your house from robbery by planting a sign that says "please,don't rob my house". Surprisingly it may works from time to time, by if you're into attempting security by wishful thinking maybe don't be too surprised when it fails about as much as security by chance.
    - laumars 10y ago
      I know. With the greatest of respect your counter argument is literally just reiterating the point I was making. Albeit in the quote you've left off the part of my post where I said it's stupid to use robots.txt in this way.
  - hdhzy 10y ago
    Note that links marked with Nofollow can still be followed by well-behaving bots: https://en.wikipedia.org/wiki/Nofollow#Interpretation_by_the_individual_search_engines https://en.wikipedia.org/wiki/Nofollow#Interpretation_by_the...
kakarot 10y ago
I came here to say something about respecting the wishes of others, etc, but you know what? You're absolutely right. We shouldn't even need to have a conversation about trust and respect. It should be non-negotiable if you don't want your personal contents indexed by scrapers and archivers, and it should be enforced by design. It's a broken system.
nabla9 10y ago
robots.text is not a security tool. It's communication tool that gives advice. Just like sitemaps. If you need to add security (logins) to protect content you don't need to protect you inconvenience users.
- laumars 10y ago
  I'd already covered the security point replying to another poster (https://news.ycombinator.com/item?id=14163792 https://news.ycombinator.com/item?id=14163792) but just to be clear, I'm absolutely not claiming robots.txt is a security tool. Quite the opposite, I saying following good security and design practices renders the robots.txt file obsolete. Your point about sitemaps helps illustrate that point of mine because having a decent sitemap mitigates the need for Allow lines in robots.txt. It's another feature of the web where robots.txt isn't well equipped to handle and thus there have been other, better, tools built to highlight pages of interest to search engines.
bkor 10y ago
Lots of laws are pretty similar. e.g. technically you could steal loads of things. Practically you don't. Defeating/ignoring mechanisms such as robots.txt (vs maybe some security person in a store) still makes stealing not ok.
- laumars 10y ago
  The morality of whether bots should obey robots.txt is a separate issue to the point I raised about how you shouldn't trust bots to obey them. To use your example of high street stores: shops have security tags on expensive items / clothing as a method of securing products from theft because you cannot blindly trust everyone not to steal (though wouldn't it be great if that wasn't the case). Equally websites cannot trust that bots will obey robots.txt. Which means any content that doesn't want to be crawled needs to be behind nofollow attributes or (if it's sensitive) user authentication layers and any content that does need to be indexed also needs to be in a sitemap. Once you have all of these extra layers implemented, the robots.txt becomes utterly redundant. Hence why I say it's a failed experiment. The benefits it offers are superseded by better solutions.
bigbugbag 10y ago
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Security https://en.wikipedia.org/wiki/Robots_exclusion_standard#Secu... robots.txt was proposed after a badly behaved bot DoSed a web server 20+ years ago, those were different times. With the robots.txt standed now those who wants to play nice can do so without asking anything, for the badly behaved ones it's still up to the admin to put forward the appropriate measures.
- laumars 10y ago
  Wow has it really been more than 20 years!?? I feel old now... I do get what you're saying but if you have to implement "appropriate measures" anyway then the robots.txt file becomes completely redundant.

mushiake 10y ago

fastastic news. Archive Team's take on this[0] [0]http://www.archiveteam.org/index.php?title=Robots.txt http://www.archiveteam.org/index.php?title=Robots.txt

rz2k 10y ago
The logic is sound, and I see that it was mostly written in 2011, but I can also see it being harmful. How about an IETF RFC to clarify? Libraries operate under a lot of unwritten social conventions, perhaps even more than most other institutions. (robots.txt even if largely ignored is a popular convention) Aggressive or confrontational wording, regardless of whether they are "right" doesn't seem in libraries' interests.
dingaling 10y ago
It is great news in general, but seems to be done in a clumsy and counterproductive manner that may cause the Internet Archive to be banned from crawling some websites. The problem: when robots.txt for a website is found to have been made more restrictive, the IA retrospectively applies its new restrictions to already-archived pages and hides them from view. This can also cause entire domains to vanish into the deep-archive. No-one outside IA thinks this is sensible. Their solution: ignore robots.txt altogether. What? That will just annoy many website operators. My proposed solution: keep parsing robots.txt on each crawl and obey it progressively, without applying the changes to existing archived material. This is actually less work than what they currently do. If the new robots.txt says to ignore about_iphone.html you just do that and ignore it. Older versions aren't affected. Basically they're switching from being excessively obedient to completely ignoring robots.txt in order to fix a self-made problem. I can only see that antagonising operators.
- duskwuff 10y ago
  There's some value in allowing site operators to retroactively remove content which was never intended to be public. A common and unfortunate example is backups (like SQL dumps) being stored in web-accessible directories, then subseqently being indexed and archived when a crawler finds the appropriate directory index. What needs to be fixed first is just the really common case mentioned in the blog post, where a domain changes ownership and a restrictive robots.txt is applied to the parking page.

libeclipse 10y ago

On the linked page, I see comments about ignoring the webmasters' wishes et al. All I can say is f*ck that. It's a free and open internet. If you put content up on a public site, anyone has the right to go and look at it. Stop complaining when someone saves it. And sure some people complain that scrapers slow down their site and that's why they use robots.txt, but really? Really? It's 2017 and your site is affected by that. I think you have bigger things to worry about.

chii 10y ago
> scrapers slow down their site and that's why they use robots.txt a poorly written scraper may really slow down your site, especially if it wasn't intended to be scrapped repeatedly. There should be something to be said about frequency which scrapers should follow (specified by the website owner via a robots.txt like spec). But website owners cannot demand unreasonable frequencies (such as once a year!), and what constitutes unreasonable is up for debate.
- davb 10y ago
  Additionally, if excessive scraping became an issue for my site I'd consider rate limiting client.
- timClicks 10y ago
  The Crawl-delay directive is the de facto standard for this.
- eknkc 10y ago
  I don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.
- dingo_bat 10y ago
  > (specified by the website owner via a robots.txt like spec). Nope, if a website wants such a restriction, it must enforce it. Robots.txt is a request. It's worthless.

droithomme 10y ago

I have some sites where I specifically block archiving from some sections for good reason. (Even if I didn't have a good reason though it would still be my choice.) I have a very big problem with them disregarding robots directives. Sure some crawlers ignore them: Hostile net actors up to no good. This decision means they are a hostile net actor. I'll have to take extreme measures such as determining all the ip address ranges they use and totally blocking access. This inconveniences me, which means they are now my enemy. edit- For those interested: Deny from 207.241.224.0/22

cprecioso 10y ago
Can I ask what the good reason is?
omgtehlion 10y ago
I have easier solution for you: just shut down your site and be done with it. This way no malicious actor will be able to save your precious information.
andrius4669 10y ago
Why not just block ia_archiver useragent in your web server for these paths instead? Also, I'm curious, what that good reason could be?
Asparagirl 10y ago
Are you under the impression that individual web archivists don't also scrape websites of interest, and submit those WARC's for inclusion into the Wayback Machine, independent of the IA's crawlers? Because believe me, we do...good luck banning every AWS and DO IP range.
- tomjen3 10y ago
  I actually didn't know that. Do you operate the same crawlers? I have considered putting a single file that is only accessible via no-follow links and perma-ban any ip that access the file, as a way to punish bad robots.

metafunctor 10y ago

It appears that IA applies (or did apply) a new version of robots.txt to pages already in their index, even if they were archived years ago. That's silly, and stopping doing that would probably solve much of this problem.

LoSboccacc 10y ago
yeah it's not even hard to solve of a problem, just use the archived version of robots.txt that matches the crawl date too bad they already lost loads of internet content that way
- tempay 10y ago
  What should happen in the case that a website misconfigures robots.txt and ends up wanting to remove private data? I think I would be tempted to say that the data can't be removed to avoid abuse from future domain owners (or current ones) but I'm not sure if there would be any legal consequences of this attitude.
  - LoSboccacc 10y ago
    Same when an archive crawl illegal/copyrigthed data, a pathway for that needs to exists anyway
  - CM30 10y ago
    Provide a content removal form? It works for DMCA notices, it can work here. Maybe even have a 'reason' textbox to see why someone may want content removed...
    - makomk 10y ago
      Of course, there's the inevitable risk that the Internet Archive's newfound control over who is allowed to make their past disappear into the memory hole and who has it archived forever will be used for political ends, especially since the ability to manually archive pages is already used this way by staff. (Take a look at Jason Scott's Twitter or that of the Archive Team sometime - lots of conspicuous manual archiving of stuff that's embarrassing to a certain US political party.) The issue of curators' views biasing the contents of collections seems to be underappreciated in general in the digital age, for some reason.

sengork 10y ago

Found this[1] via the Wikipedia's Talk page for the robots.txt article. It showcases that early on robots.txt was designated to help maintain bandwidth performance of web servers. Back then it would have been due to bandwidth contention, today it may be bandwidth cost to some operators which robots.txt help mitigate. [1] https://yro.slashdot.org/comments.pl?sid=377285&cid=21554125 https://yro.slashdot.org/comments.pl?sid=377285&cid=21554125

yeukhon 10y ago

I remember writing a dumb parser for robots.txt. I have to agree, robots.txt is simplistic but so non-standard. I wonder why search engines can't just say NO to this. Does search engines today still honor robots.txt? Here's my shameless plug: https://github.com/yeukhon/robots-txt-scanner https://github.com/yeukhon/robots-txt-scanner I still remember writing most of this on Caltrain one morning heading to SF visiting someone I dearly loved.....

afandian 10y ago

I'm not so sure that even Google respects it. I did some digging about the semantics about robots.txt whilst writing a bot myself, and it seems that Google doesn't follow links that are excluded, but it will visit those pages. Maybe that counts as "paying attention", but I don't think they "respect" it.

butler14 10y ago
they respect it, but because it's so frequently misused or plain broken, it's basically sidelined vs. more optimal methods for preventing indexation, or getting an already-indexed piece of content removed, such as the noindex tag.

Aissen 10y ago

Finally. A bit late, since a lot of the archive has been removed because of new owner's aggressive (or malicious) new robots.txt

pjc50 10y ago
Hidden, rather than removed.
- Aissen 10y ago
  I hope so.

amelius 10y ago

I think we should write down a legal license in our robots.txt file, as a retribution for all those lengthy EULAs these big companies make us read :)

madshiva 10y ago

Yeah just ignore robots.txt because there's others solution. If the site don't want to be scanned they can adopt a lot of counter measure and robots.txt will not save it from abuse. He remind me the old days when my website wasn't working from US because I just fake that the site was down because there's no reason that somebody goes to my site from US (I know it's kind stupid, but when all your content is in french and you are a kid... :) )

rubatuga 10y ago

One thing that should be considered is the right for an individual to be forgotten.

freshhawk 10y ago
Forgotten by whom? Who judges what is "forgotten" and when? How is it enforced? The specifics here matter a great deal, the versions so far are regularly abused by the wealthy and don't apply to any of the data warehouses that the powerful and well connected have access to. Where did this "right" come from? What's the legal and ethical basis for it? It is analogous to censorship or book burning at the basic level, destroying information to hide it from the public. It requires a consistent and strong justification as well as justified limited scope because of that, and it better be obviously beneficial to society even accounting for the inevitable misuse by those in power.
- beejiu 10y ago
  There is a legal basis in the European Union. https://en.wikipedia.org/wiki/Right_to_be_forgotten https://en.wikipedia.org/wiki/Right_to_be_forgotten
  - merb 10y ago
    and the legal basis is pretty fluffy: - http://www.sueddeutsche.de/digital/bgh-grundsatzurteil-namen-der-sedlmayr-moerder-bleiben-online-1.144087 http://www.sueddeutsche.de/digital/bgh-grundsatzurteil-namen... - http://www.focus.de/digital/internet/bgh-urteil-keine-staendige-kontrolle-von-online-archiven_aid_463464.html http://www.focus.de/digital/internet/bgh-urteil-keine-staend... it's german, but basically it says: "This is not a blank check".
TeMPOraL 10y ago
In my (current) opinion, it's this law that should be forgotten. What's on the public Internet is a matter of public interest. All I can see is this law being used by bad people to hide their bad deeds, especially when those bad deeds should be known.

deleted 10y ago

[deleted]

TeMPOraL 10y ago

Archive.org needs to be able to apply to itself. We could then use archive.org to view how archive.org in the past viewed some interesting site, thus avoiding the whole retroactive robots.txt fail. ;).

6d6b73 10y ago

There should be a way to direct archiving bots to a file that has the newest, compressed version of the website for them to download. Wouldn't that be easier for everyone?

BHSPitMonkey 10y ago
Seems like it would just get abused with false content
- 6d6b73 10y ago
  True, but maybe it would be a good option for non-commercial websites that would like to get archived, and make the archiving more efficient for themselves and archive.org

EvanAnderson 10y ago

The policy the Internet Archive applies re: "robots.txt" comes from an archive policy created at U.C. Berkeley in the early 2000's (The Oakland Archive Policy - http://www2.sims.berkeley.edu/research/conferences/aps/removal-policy.html http://www2.sims.berkeley.edu/research/conferences/aps/remov...). Jason Scott (an employee of the Internet Archive) mentioned that the Archive doesn't ever delete anything. He stated that items may be removed from public access because of changes to "robots.txt" but they're not actually deleted. (That's a little comforting, at least.)

dbg31415 10y ago

I think the biggest argument for honoring robots.txt is that sites, especially old sites, can have a lot of really highly resource intensive pages. I don't want someone crawling a page that has 800+ DB calls... for example. Yes, I should optimize the page, or whatever... but really it may be useful for that admin, or 1 out of 10,000 users, who uses the page. It's not ideal to have someone crawl all those pages at once. I think they should honor robots.txt, and the meta tag version on specific pages or links -- given the site publisher went out of their way to give instructions to crawlers it seems reasonable to honor those requests.

cosinetau 10y ago

For whatever it's worth: http://humanstxt.org/ http://humanstxt.org/

c0achmcguirk 10y ago

In the 90s I spent a lot of time on my website and I loved learning how web crawlers worked. I started using a robots.txt file without really understanding it. I ended up blocking everything thinking it would make my site faster for visitors because--crawlers might crawl the site all the time. After I graduated from college I lost access to my website which was hosted on the Computer Science department's web servers. I wish I hadn't used that robots.txt file. I would love to find the pages I made that compared interfold vs. exterfold staple strength, or the site I made with a ranch theme with a cowboy that had humorous advice....I don't have any content in archive.org because it honored the robots.txt file. ...sigh...wish I had backed up my stuff.

deleted 10y ago

[deleted]