4 ms·

One point to note here, is that the Wayback Machine obeys robots.txt retroactively, so sites can hide evidence by changing their robots.txt to disallow indexing

One point to note here, is that the Wayback Machine obeys robots.txt retroactively, so sites can hide evidence by changing their robots.txt to disallow indexing of specific content. The data won't be purged from the servers, but it will not be displayed through the site. This is quite disappointing (though probably wise from a legal standpoint). It makes it less useful if a party in a lawsuit can retroactively hide evidence of wrongdoing, and thereby deny access to evidence. It's also a problem if a domain name changes ownership and the new owner suddenly adds a restrictive robots.txt - the old content will no longer be accessible even though the current owner has no claim to it.

huherto 10y ago
Very interesting. At least they cannot add content. And it is easy to know if they may have hidden something. Since the robots.txt should be public. I hope. May be a judge can make them unhide. I don't know.
epicaricacy 10y ago
That goes to weight more than admissibility. The only thing that the judge ruled in this case is that it CAN be used.
pjc50 10y ago
deny access to evidence Subpoena?
bonyt 10y ago
In 2009, there was an interesting case where a federal district court ordered a plaintiff to disable its robots.txt to allow the Wayback Machine to disclose an old version of the plaintiff's website.[1] Someone at the Internet Archive provided a declaration stating that it would place a significant burden on them[2] to respond to a subpoena themselves, whereas all the plaintiff had to do was modify its robots.txt. The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. [1]: http://www.american-justice.org/upload/page/123/69/docket-187-order-on-IA-motion.pdf http://www.american-justice.org/upload/page/123/69/docket-18... [2]: https://tonybox.net/tmp/ia_decl_pacer.pdf https://tonybox.net/tmp/ia_decl_pacer.pdf (downloaded from PACER)
- greglindahl 10y ago
  Thanks for making the PACER doc available -- is it in RECAP?
- nightcracker 10y ago
  > The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. How is this not indirect self-incrimination?
  - mey 10y ago
    It's available in the wayback machine, it was already made publicly available at a previous point in time. This is primarily reducing the courts burden.
  - mentat 10y ago
    Does self-incrimination apply to companies?
  - snuxoll 10y ago
    A subpeona was already issued and this information could have still been extracted from the Internet Archive directly, just at a higher technological hurdle. If the information was still on the plaintiff's servers they could have just extracted it from there as well (or any backups that they might have had laying around), since it was deleted they needed other ways to recover it. robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).
  - LeifCarrotson 10y ago
    The government has the right to use force in an investigation. Things you have in your mind or - in some cases - in an extension of your mind (you're secretly actually guilty, you know where incriminating evidence is, or a safe combination or password that you haven't forgotten) cannot be compelled. Things you possess (the contents of your home or gun safe, the thumb drive clenched in your fist, the robots.txt file on your server) can be compelled with a search warrant or subpoena. They are your things, and they could incriminate you, but that's not what the 5th Amendment is for. Laconic.net is a resource I have enjoyed. Here is the 5th amendment passage: http://lawcomic.net/guide/?p=2470 http://lawcomic.net/guide/?p=2470 or go back for a history lesson.

huherto 10y ago

Very interesting. At least they cannot add content. And it is easy to know if they may have hidden something. Since the robots.txt should be public. I hope. May be a judge can make them unhide. I don't know.

epicaricacy 10y ago

That goes to weight more than admissibility. The only thing that the judge ruled in this case is that it CAN be used.

pjc50 10y ago

deny access to evidence Subpoena?

bonyt 10y ago

In 2009, there was an interesting case where a federal district court ordered a plaintiff to disable its robots.txt to allow the Wayback Machine to disclose an old version of the plaintiff's website.[1] Someone at the Internet Archive provided a declaration stating that it would place a significant burden on them[2] to respond to a subpoena themselves, whereas all the plaintiff had to do was modify its robots.txt. The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. [1]: http://www.american-justice.org/upload/page/123/69/docket-187-order-on-IA-motion.pdf http://www.american-justice.org/upload/page/123/69/docket-18... [2]: https://tonybox.net/tmp/ia_decl_pacer.pdf https://tonybox.net/tmp/ia_decl_pacer.pdf (downloaded from PACER)

greglindahl 10y ago
Thanks for making the PACER doc available -- is it in RECAP?
nightcracker 10y ago
> The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. How is this not indirect self-incrimination?
- mey 10y ago
  It's available in the wayback machine, it was already made publicly available at a previous point in time. This is primarily reducing the courts burden.
- mentat 10y ago
  Does self-incrimination apply to companies?
- snuxoll 10y ago
  A subpeona was already issued and this information could have still been extracted from the Internet Archive directly, just at a higher technological hurdle. If the information was still on the plaintiff's servers they could have just extracted it from there as well (or any backups that they might have had laying around), since it was deleted they needed other ways to recover it. robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).
- LeifCarrotson 10y ago
  The government has the right to use force in an investigation. Things you have in your mind or - in some cases - in an extension of your mind (you're secretly actually guilty, you know where incriminating evidence is, or a safe combination or password that you haven't forgotten) cannot be compelled. Things you possess (the contents of your home or gun safe, the thumb drive clenched in your fist, the robots.txt file on your server) can be compelled with a search warrant or subpoena. They are your things, and they could incriminate you, but that's not what the 5th Amendment is for. Laconic.net is a resource I have enjoyed. Here is the 5th amendment passage: http://lawcomic.net/guide/?p=2470 http://lawcomic.net/guide/?p=2470 or go back for a history lesson.