6 ms·

Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of sys

by kodah 5y ago

Personally, I think eternally archiving everything and infinitely available public data has been not-so-great. If this was an "archive with consent" sort of system, then sure. My response may be better summarized as, "Does IA support robots.txt, and if not why?"

1MachineElf 5y ago
There is some public discussion about why IA does not strictly adhere to robots.txt: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- kodah 5y ago
  They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.
  - BenjiWiebe 5y ago
    When those people die and quit paying for hosting, the information on their website doesn't magically become useless. Perhaps other people still have a need for it. Thank goodness the IA doesn't blindly obey robots.txt.
rectang 5y ago
Throughout human history, records have been forgotten, rewritten, changed, mutated, degraded, eroded away to nothingness. "The internet is forever" has always struck me as inhumane. Make a mistake or expose a weakness on the internet and it will always accompany you. It turns out that the internet is not always forever. I find that comforting.
- symlinkk 5y ago
  Maybe humans should become more accommodating of past mistakes.
  - _jal 5y ago
    Suggest that to them. I'm sure they'll get right on it.
- treesknees 5y ago
  Depends on how you think of "forever". If you post an embarrassing video and someone saves and reposts it with your name attached, odds are that video isn't going to be around in 200 years. But what about the next 10, 20 or 40 years? In the context of your overall professional adult life, that's a long time. "The Internet is forever" isn't some natural law by which all content abides as though it can never disappear. It's a warning that you don't control the content once it's accessible on the Internet.
- oconnor663 5y ago
  I think there's inhumanity of a kind on both sides of this question. "Everything you've done will be forgotten, and no one will remember your name" is the sort of thing the bad guys say in movies. But that's what happens to most of us in the end. I think it's natural not to want that.
garaetjjte 5y ago
You're consenting by posting it on public internet in the first place.
- ghaff 5y ago
  Posting something on the public internet is not consent for you to scrape it and post it on your own site forever. And requiring an explicit opt-in would basically mean no IA. To be clear, the IA is a positive, maybe even a great one. But it skirts by because most people don't care. (They did as you say post whatever on the public internet.) Add the facts that they're a non-profit, aren't trying to monetize their hosting, and will generally take things if the owner asks. Libraries and other archives have some very limited special rights (which mostly relate to making physical backups of physical books). But invoking "library" isn't some general get out of jail free card with respect to copyright.
  - blackearl 5y ago
    People have recorded others without their consent for millennia. Whether it's telling a story about what someone said, a photo, or now a screenshot of a twitter post, that is reality. You'll never be able to stop someone from telling another that you said X.
  - pessimizer 5y ago
    It can be. The reason libraries and other archives have special rights is because they fought for them against the express wishes of people who sold paper. There are no arguments made against archive.org that weren't also made against libraries.
    - Mezzie 5y ago
      Thank you. These rights are also under constant attack: It's normal to charge libraries exorbitant prices for digital materials compared to their analogue counterparts, for example.
  - _jal 5y ago
    > Posting something on the public internet is not consent for you to scrape it and post it on your own site forever. It effectively is. Your consent is not required, and people are doing far worse than just keeping it available (Clearview; there are also reports of people hoovering up encrypted data to crack in the coming decades when we're post-quantum). This is no different than demanding people not keep track of anything else, and attacking archive.org might make you feel better, but that won't make anyone else stop.
FrostKiwi 5y ago
Public data is... public. No one should be stopped from saving a public page and nothing should stop Internet Archive, be it robot or human. There is if course a need for removal of archived content infringing on someone's rights, whatever that might be, but "archive with consent" will fail for the goal of preserving culture. I think it's worrying that some online newspapers enacted archive blockers or IA needing DMCA excemptions, just so companies can't DMCA anything with their name on it. To preserve journalistic integreity and to save culture, even if it collides intellectual property rights, "archive with consent" won't cut it.
- orev 5y ago
  Content posted on a web site is NOT “public” (domain), it is (in the US) automatically copyrighted to the author, unless they specifically waive those rights. Just because you can see it through a browser doesn’t in any way mean you can make it yours and do what you want with it.
  - ImprobableTruth 5y ago
    We store books in public libraries even if they aren't public domain and authors can do nothing to prevent them from doing so.
    - rat9988 5y ago
      If you buy the book, you have an authorization to do so. But you can't put copies of it.
      - Mezzie 5y ago
        You do if you're an archive, actually.
    - ghaff 5y ago
      First sale doctrine in the US. If I buy a physical book, I can give it, loan it, throw it in the trash, etc. This doesn't apply to making electronic copies--see what happened with Google Books for example.
Ericson2314 5y ago
There are clearly some things we, and the author, wants to persist, and yet the internet fails to do so. Trying to muddle in privacy concerns with the accumulation of public knowledge undermines the whole concept of a shared society, or there being even the potential of accumulating "progress" in the first place. Also, historically speaking, people with means used to save their letters for posterity, which proved to be a very valuable resource for future academics, so the idea of, what, deleting all your proton mails and signal messages as encouraged is arguably overshooting the return to some pre-internet norm.
- kodah 5y ago
  > Also, historically speaking, people with means used to save their letters for posterity, which proved to be a very valuable resource for future academics Your example is an example of choice or consent. They also had the option to burn their hand written books and scrolls down periodically. Systems like IA take that choice away.
  - Ericson2314 5y ago
    There is no reason to do person communication with staticish websites. Don't foist the problems of social media onto the Internet Archive.
    - kodah 5y ago
      I'm not sure I understand. robots.txt was a matter of consent and was standardized well over twenty years ago. The IA willfully ignores it because they believe it interferes with their mission. This was long before social media, when static websites were more dominant than dynamic ones. Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
      - Ericson2314 5y ago
        This is a fair philosophical quibble, but the tiny amount of harm done in in real world makes this a waste of time discussing. You made it public on the damn web, people doing what they wanted with it is fair game per the original ethos of the web. If you wanted it to be private, you should have some auth or encryption. The actual real world harm is in actually-common-used means of personal communication were network effects preclude on carrying out they business with technology appropriate to desired privacy levels. Chatting on my custom website was always niche, and therefore the network effects argument doesn't carry water. We should refrain from worrying about robots.txt minute until the elephant in the room is put to rset.
jrwr 5y ago
They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offline. Their documentation about it is rather crap right now, but its in several of their FAQs about it
- pseudalopex 5y ago
  Respecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.

by kodah 5y ago

1MachineElf 5y ago

There is some public discussion about why IA does not strictly adhere to robots.txt: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

kodah 5y ago
They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.
- BenjiWiebe 5y ago
  When those people die and quit paying for hosting, the information on their website doesn't magically become useless. Perhaps other people still have a need for it. Thank goodness the IA doesn't blindly obey robots.txt.

rectang 5y ago

Throughout human history, records have been forgotten, rewritten, changed, mutated, degraded, eroded away to nothingness. "The internet is forever" has always struck me as inhumane. Make a mistake or expose a weakness on the internet and it will always accompany you. It turns out that the internet is not always forever. I find that comforting.

symlinkk 5y ago
Maybe humans should become more accommodating of past mistakes.
- _jal 5y ago
  Suggest that to them. I'm sure they'll get right on it.
treesknees 5y ago
Depends on how you think of "forever". If you post an embarrassing video and someone saves and reposts it with your name attached, odds are that video isn't going to be around in 200 years. But what about the next 10, 20 or 40 years? In the context of your overall professional adult life, that's a long time. "The Internet is forever" isn't some natural law by which all content abides as though it can never disappear. It's a warning that you don't control the content once it's accessible on the Internet.
oconnor663 5y ago
I think there's inhumanity of a kind on both sides of this question. "Everything you've done will be forgotten, and no one will remember your name" is the sort of thing the bad guys say in movies. But that's what happens to most of us in the end. I think it's natural not to want that.

garaetjjte 5y ago

You're consenting by posting it on public internet in the first place.

ghaff 5y ago
Posting something on the public internet is not consent for you to scrape it and post it on your own site forever. And requiring an explicit opt-in would basically mean no IA. To be clear, the IA is a positive, maybe even a great one. But it skirts by because most people don't care. (They did as you say post whatever on the public internet.) Add the facts that they're a non-profit, aren't trying to monetize their hosting, and will generally take things if the owner asks. Libraries and other archives have some very limited special rights (which mostly relate to making physical backups of physical books). But invoking "library" isn't some general get out of jail free card with respect to copyright.
- blackearl 5y ago
  People have recorded others without their consent for millennia. Whether it's telling a story about what someone said, a photo, or now a screenshot of a twitter post, that is reality. You'll never be able to stop someone from telling another that you said X.
- pessimizer 5y ago
  It can be. The reason libraries and other archives have special rights is because they fought for them against the express wishes of people who sold paper. There are no arguments made against archive.org that weren't also made against libraries.
  - Mezzie 5y ago
    Thank you. These rights are also under constant attack: It's normal to charge libraries exorbitant prices for digital materials compared to their analogue counterparts, for example.
- _jal 5y ago
  > Posting something on the public internet is not consent for you to scrape it and post it on your own site forever. It effectively is. Your consent is not required, and people are doing far worse than just keeping it available (Clearview; there are also reports of people hoovering up encrypted data to crack in the coming decades when we're post-quantum). This is no different than demanding people not keep track of anything else, and attacking archive.org might make you feel better, but that won't make anyone else stop.

FrostKiwi 5y ago

Public data is... public. No one should be stopped from saving a public page and nothing should stop Internet Archive, be it robot or human. There is if course a need for removal of archived content infringing on someone's rights, whatever that might be, but "archive with consent" will fail for the goal of preserving culture. I think it's worrying that some online newspapers enacted archive blockers or IA needing DMCA excemptions, just so companies can't DMCA anything with their name on it. To preserve journalistic integreity and to save culture, even if it collides intellectual property rights, "archive with consent" won't cut it.

orev 5y ago
Content posted on a web site is NOT “public” (domain), it is (in the US) automatically copyrighted to the author, unless they specifically waive those rights. Just because you can see it through a browser doesn’t in any way mean you can make it yours and do what you want with it.
- ImprobableTruth 5y ago
  We store books in public libraries even if they aren't public domain and authors can do nothing to prevent them from doing so.
  - rat9988 5y ago
    If you buy the book, you have an authorization to do so. But you can't put copies of it.
    - Mezzie 5y ago
      You do if you're an archive, actually.
  - ghaff 5y ago
    First sale doctrine in the US. If I buy a physical book, I can give it, loan it, throw it in the trash, etc. This doesn't apply to making electronic copies--see what happened with Google Books for example.

Ericson2314 5y ago

There are clearly some things we, and the author, wants to persist, and yet the internet fails to do so. Trying to muddle in privacy concerns with the accumulation of public knowledge undermines the whole concept of a shared society, or there being even the potential of accumulating "progress" in the first place. Also, historically speaking, people with means used to save their letters for posterity, which proved to be a very valuable resource for future academics, so the idea of, what, deleting all your proton mails and signal messages as encouraged is arguably overshooting the return to some pre-internet norm.

kodah 5y ago
> Also, historically speaking, people with means used to save their letters for posterity, which proved to be a very valuable resource for future academics Your example is an example of choice or consent. They also had the option to burn their hand written books and scrolls down periodically. Systems like IA take that choice away.
- Ericson2314 5y ago
  There is no reason to do person communication with staticish websites. Don't foist the problems of social media onto the Internet Archive.
  - kodah 5y ago
    I'm not sure I understand. robots.txt was a matter of consent and was standardized well over twenty years ago. The IA willfully ignores it because they believe it interferes with their mission. This was long before social media, when static websites were more dominant than dynamic ones. Source: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
    - Ericson2314 5y ago
      This is a fair philosophical quibble, but the tiny amount of harm done in in real world makes this a waste of time discussing. You made it public on the damn web, people doing what they wanted with it is fair game per the original ethos of the web. If you wanted it to be private, you should have some auth or encryption. The actual real world harm is in actually-common-used means of personal communication were network effects preclude on carrying out they business with technology appropriate to desired privacy levels. Chatting on my custom website was always niche, and therefore the network effects argument doesn't carry water. We should refrain from worrying about robots.txt minute until the elephant in the room is put to rset.

jrwr 5y ago

They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offline. Their documentation about it is rather crap right now, but its in several of their FAQs about it

pseudalopex 5y ago
Respecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.