3 ms·

In 2009, there was an interesting case where a federal district court ordered a plaintiff to disable its robots.txt to allow the Wayback Machine to disclose an

by bonyt 10y ago

In 2009, there was an interesting case where a federal district court ordered a plaintiff to disable its robots.txt to allow the Wayback Machine to disclose an old version of the plaintiff's website.[1] Someone at the Internet Archive provided a declaration stating that it would place a significant burden on them[2] to respond to a subpoena themselves, whereas all the plaintiff had to do was modify its robots.txt. The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. [1]: http://www.american-justice.org/upload/page/123/69/docket-187-order-on-IA-motion.pdf http://www.american-justice.org/upload/page/123/69/docket-18... [2]: https://tonybox.net/tmp/ia_decl_pacer.pdf https://tonybox.net/tmp/ia_decl_pacer.pdf (downloaded from PACER)

greglindahl 10y ago
Thanks for making the PACER doc available -- is it in RECAP?
nightcracker 10y ago
> The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. How is this not indirect self-incrimination?
- mey 10y ago
  It's available in the wayback machine, it was already made publicly available at a previous point in time. This is primarily reducing the courts burden.
- mentat 10y ago
  Does self-incrimination apply to companies?
- snuxoll 10y ago
  A subpeona was already issued and this information could have still been extracted from the Internet Archive directly, just at a higher technological hurdle. If the information was still on the plaintiff's servers they could have just extracted it from there as well (or any backups that they might have had laying around), since it was deleted they needed other ways to recover it. robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).
- LeifCarrotson 10y ago
  The government has the right to use force in an investigation. Things you have in your mind or - in some cases - in an extension of your mind (you're secretly actually guilty, you know where incriminating evidence is, or a safe combination or password that you haven't forgotten) cannot be compelled. Things you possess (the contents of your home or gun safe, the thumb drive clenched in your fist, the robots.txt file on your server) can be compelled with a search warrant or subpoena. They are your things, and they could incriminate you, but that's not what the 5th Amendment is for. Laconic.net is a resource I have enjoyed. Here is the 5th amendment passage: http://lawcomic.net/guide/?p=2470 http://lawcomic.net/guide/?p=2470 or go back for a history lesson.
  - LeifCarrotson 10y ago
    Whoops, that should be "lawcomic.net". The modern Miranda passage starts here: http://lawcomic.net/guide/?p=2589 http://lawcomic.net/guide/?p=2589
- anigbrowl 10y ago
  For one, losing in a civil trial isn't the same as being subject to criminal penalties, so you can't self-incriminate unless what you're hiding would itself have been a crime - many contractual disputes have no criminal dimension whatsoever and so your liberty isn't at risk. But more to the point, archive.org already has a copy of what you previously made public, and there's no reason to burden them with the expense of manually retrieving it as they're not party to the case, and it would be wasteful of the court's resources and archive.org's time to make them sue you for the expense involved. Here's a more detail breakdown of how the 5th applies in a civil litigation context: http://www.litigationandtrial.com/2013/04/articles/attorney/pleading-the-fifth-adverse-inferences/ http://www.litigationandtrial.com/2013/04/articles/attorney/... The bottom line is that the 5th can help you stay out of prison, but you can't use it just to avoid losing a dispute.
  - Lawtonfogle 10y ago
    Sounds like saying we already have clear evidence but it'll take time to dig through it so you are now compelled to show us where you hid the body.
CJefferson 10y ago
Thanks for answering a question I had been worried about for ages -- did changes to robot.txt make the internet archive delete it's history, or just hide it.

by bonyt 10y ago

greglindahl 10y ago

Thanks for making the PACER doc available -- is it in RECAP?

nightcracker 10y ago

> The court reasoned that since the plaintiff had the technical ability to un-block access, they could be compelled to do so, and made them disable their robots.txt. How is this not indirect self-incrimination?

mey 10y ago
It's available in the wayback machine, it was already made publicly available at a previous point in time. This is primarily reducing the courts burden.
mentat 10y ago
Does self-incrimination apply to companies?
snuxoll 10y ago
A subpeona was already issued and this information could have still been extracted from the Internet Archive directly, just at a higher technological hurdle. If the information was still on the plaintiff's servers they could have just extracted it from there as well (or any backups that they might have had laying around), since it was deleted they needed other ways to recover it. robots.txt is assumed to only be used to block automated access to a website by search engines and other crawlers, not to hide it from human view as well (in fact, it makes sense to NOT use it to hide things from humans as it announces the presence of content that you are trying to hide). Removing a file that is only to be used to prevent crawlers from indexing certain parts of your site (because they are irrelevant for search results) is hardly self-incrimination (again, because you are advertising the fact that content exists at this location and you don't want it indexed).
LeifCarrotson 10y ago
The government has the right to use force in an investigation. Things you have in your mind or - in some cases - in an extension of your mind (you're secretly actually guilty, you know where incriminating evidence is, or a safe combination or password that you haven't forgotten) cannot be compelled. Things you possess (the contents of your home or gun safe, the thumb drive clenched in your fist, the robots.txt file on your server) can be compelled with a search warrant or subpoena. They are your things, and they could incriminate you, but that's not what the 5th Amendment is for. Laconic.net is a resource I have enjoyed. Here is the 5th amendment passage: http://lawcomic.net/guide/?p=2470 http://lawcomic.net/guide/?p=2470 or go back for a history lesson.
- LeifCarrotson 10y ago
  Whoops, that should be "lawcomic.net". The modern Miranda passage starts here: http://lawcomic.net/guide/?p=2589 http://lawcomic.net/guide/?p=2589
anigbrowl 10y ago
For one, losing in a civil trial isn't the same as being subject to criminal penalties, so you can't self-incriminate unless what you're hiding would itself have been a crime - many contractual disputes have no criminal dimension whatsoever and so your liberty isn't at risk. But more to the point, archive.org already has a copy of what you previously made public, and there's no reason to burden them with the expense of manually retrieving it as they're not party to the case, and it would be wasteful of the court's resources and archive.org's time to make them sue you for the expense involved. Here's a more detail breakdown of how the 5th applies in a civil litigation context: http://www.litigationandtrial.com/2013/04/articles/attorney/pleading-the-fifth-adverse-inferences/ http://www.litigationandtrial.com/2013/04/articles/attorney/... The bottom line is that the 5th can help you stay out of prison, but you can't use it just to avoid losing a dispute.
- Lawtonfogle 10y ago
  Sounds like saying we already have clear evidence but it'll take time to dig through it so you are now compelled to show us where you hid the body.

CJefferson 10y ago

Thanks for answering a question I had been worried about for ages -- did changes to robot.txt make the internet archive delete it's history, or just hide it.