4 ms·

If the destruction of all this unique user-generated content pisses you off -- and it should -- then feel free to join ArchiveTeam and help save this (and other

If the destruction of all this unique user-generated content pisses you off -- and it should -- then feel free to join ArchiveTeam and help save this (and other) content for the Internet Archive. Check out the ArchiveTeam wiki at archiveteam.org, or come on over to #archiveteam and #archiveteam-bs on efnet throughout this week to watch a self-organizing data rescue mission happen in real time. (There will probably be a new dedicated channel spun off for this project later today or tomorrow.) In the meantime, programs like youtube-dl and livestreamer are your friends. http://rg3.github.io/youtube-dl/documentation.html http://rg3.github.io/youtube-dl/documentation.html http://livestreamer.readthedocs.org/en/latest/ http://livestreamer.readthedocs.org/en/latest/

IvyMike 12y ago
To elaborate, the ArchiveTeam distributes a very simple-to-use virtual machine image that you can run. When you do so, your machine will help download and collect from disappearing websites. I don't think they've got scripts for justin.tv yet, but it sounds like they are going to, so now would be a good time to get your VM ready. It's easy, and the more people who run the machine, the more power the ArchiveTeam has at their disposal. I encourage everyone to give it a look. You can get the virtual machine here: http://tracker.archiveteam.org/ http://tracker.archiveteam.org/
- g_lined 12y ago
  I downloaded the VM today and spent a fair while trying to diagnose a CheckIP error. It turns out, after going on the IRC channel that none of the projects on the VM are up to date and hence the VM can't be used to help the project. Hence the error. If I see the website update to mention justin.tv I'll certainly fire it up. But it seems a shame that there is nothing I can do to help with their project as a whole; I couldn't even see any way to donate - I guess I should just donate to archive.org.
- Asparagirl 12y ago
  Right, the idea is that justin.tv could be formatted as a new project for the ArchiveTeam Warrior -- which you can run on your laptop or a VM or a cloud server or a few Amazon instances, or whatever. But with only a week's notice before the shutdown, I don't know whether they can get the customized scripts up and running in time. They're working on extracting a list of user and video id's right now.
- ddorian43 12y ago
  it has stopped working for me, what about you, it still works ?
  - IvyMike 12y ago
    Everything seems ok for me. What is going wrong?
- MichaelStubbs 12y ago
  It's worth noting that you don't have to use a virtual machine. If you're familiar with compiling basic utilities and installing Python packages, then you can run it without the overhead of a virtual machine. It's generally all on the wiki/Github.
Asparagirl 12y ago
As expected, the dedicated channel for the data rescue has now been set up: join #justouttv on efnet. Right now, coders with Python, Lua, and/or Redis knowledge would probably be most in demand, given the existing toolset. Hopefully very soon there will be a new Warrior project formally set up, and then anyone who can run a VM or an AMI or install the tools from the command line would be able to join in to distribute the downloading. During the Hyves.nl rescue in late 2013, ArchiveTeam did accept donations to cover Amazon instances earmarked for that project, so it's possible they may do something like that again this time. In case people are curious, the Warrior code and seesaw code are here: https://github.com/archiveteam/warrior-code2 https://github.com/archiveteam/warrior-code2 https://github.com/ArchiveTeam/seesaw-kit https://github.com/ArchiveTeam/seesaw-kit
Smerity 12y ago
It honestly disturbs me that Myspace, Justin.tv, and these other companies don't seem to ever consider shipping their soon-to-be-deceased data to the Internet Archive. Why not help preservation a little? It'd likely be simpler to handle than the oncoming denial of preservation defense that the ArchiveTeam launches. Have there been any examples of companies doing this? You might think that this is all useless information, but I guarantee we'll look back in likely only a few years and think in horror at all that we've destroyed. The best art blog about a half decade including a kid who's dead now just disappeared. When Myspace deleted all of their content, I lost all the content of my dead high school friend (turns out you need to back up everyone and thing you care about, not just yourself[1]) and @maxfenton mentioned "The best art blog about a half decade including a kid who's dead now just disappeared.". It's not an isolated problem. I almost feel like we need a name-and-shame registry and/or public awareness campaign. I'd love to see this written into the user rights of a website when it launches. "If the website is to be shut down, all publicly accessible content will be sent to the Internet Archive. If you would like to opt out, tick this box." [1]: http://smerity.com/articles/2013/deleted_digital_tombstones.html http://smerity.com/articles/2013/deleted_digital_tombstones....
mantrax5 12y ago
"If the destruction of all this unique user-generated content pisses you off -- and it should" I'm not sure why it should. When did we get this strange idea that when you make some piece of content, no matter if it's quality or not, it deserves to pollute the cyberspace forever. I realize this is an unpopular opinion, but one of the reasons nature "works" is that everything old is allowed to dissolve and become the base ground for the new that will come after it. People naturally copy content that they find interesting. And content which isn't that interesting naturally gets lost. Maybe that lost content stuck around enough to inspire someone to do something in that spirit later or (which also counts), but it doesn't survive itself. Imagine if everyone gets Glasses one day and starts doing non-stop 1080p video stream of their lives. Are we going to be "pissed off" at that being deleted too? If you do the stats you'll notice that over 99.99% of the content in archive.org is never accessed. Nobody cares. And that's how it should be. We should really allow content to die. I have the firm belief that information doesn't just "want" to be free. It also wants to be allowed to die gracefully at some point.
- ivank 12y ago
  People sometimes make copies of things they find interesting, but those copies are rarely accessible to the public. It can be pretty hard to get your hands on something after it disappears from the web. For example, forums.truecrypt.org went down three days ago, it's of interest to some people, and you're going to have a hell of a time finding some guy who happened to launch HTTrack on it before it went down.
  - mantrax5 12y ago
    Valid points, but you need some strategy for pruning the archive. We can't just keep everything forever. First it was text files, then html files, then html files with images, and now videos. Without pruning, the task of archive.org will become impossible, so the question of what's important and what isn't will have to be decided at some point.
- voltagex_ 12y ago
  I'm currently running through a Peter Norton assembly tutorial from the 1980s. It's already helped me understand the code better than I have before. Should that content have died, just because it's old?

IvyMike 12y ago

To elaborate, the ArchiveTeam distributes a very simple-to-use virtual machine image that you can run. When you do so, your machine will help download and collect from disappearing websites. I don't think they've got scripts for justin.tv yet, but it sounds like they are going to, so now would be a good time to get your VM ready. It's easy, and the more people who run the machine, the more power the ArchiveTeam has at their disposal. I encourage everyone to give it a look. You can get the virtual machine here: http://tracker.archiveteam.org/ http://tracker.archiveteam.org/

g_lined 12y ago
I downloaded the VM today and spent a fair while trying to diagnose a CheckIP error. It turns out, after going on the IRC channel that none of the projects on the VM are up to date and hence the VM can't be used to help the project. Hence the error. If I see the website update to mention justin.tv I'll certainly fire it up. But it seems a shame that there is nothing I can do to help with their project as a whole; I couldn't even see any way to donate - I guess I should just donate to archive.org.
Asparagirl 12y ago
Right, the idea is that justin.tv could be formatted as a new project for the ArchiveTeam Warrior -- which you can run on your laptop or a VM or a cloud server or a few Amazon instances, or whatever. But with only a week's notice before the shutdown, I don't know whether they can get the customized scripts up and running in time. They're working on extracting a list of user and video id's right now.
ddorian43 12y ago
it has stopped working for me, what about you, it still works ?
- IvyMike 12y ago
  Everything seems ok for me. What is going wrong?
MichaelStubbs 12y ago
It's worth noting that you don't have to use a virtual machine. If you're familiar with compiling basic utilities and installing Python packages, then you can run it without the overhead of a virtual machine. It's generally all on the wiki/Github.

Asparagirl 12y ago

As expected, the dedicated channel for the data rescue has now been set up: join #justouttv on efnet. Right now, coders with Python, Lua, and/or Redis knowledge would probably be most in demand, given the existing toolset. Hopefully very soon there will be a new Warrior project formally set up, and then anyone who can run a VM or an AMI or install the tools from the command line would be able to join in to distribute the downloading. During the Hyves.nl rescue in late 2013, ArchiveTeam did accept donations to cover Amazon instances earmarked for that project, so it's possible they may do something like that again this time. In case people are curious, the Warrior code and seesaw code are here: https://github.com/archiveteam/warrior-code2 https://github.com/archiveteam/warrior-code2 https://github.com/ArchiveTeam/seesaw-kit https://github.com/ArchiveTeam/seesaw-kit

Smerity 12y ago

It honestly disturbs me that Myspace, Justin.tv, and these other companies don't seem to ever consider shipping their soon-to-be-deceased data to the Internet Archive. Why not help preservation a little? It'd likely be simpler to handle than the oncoming denial of preservation defense that the ArchiveTeam launches. Have there been any examples of companies doing this? You might think that this is all useless information, but I guarantee we'll look back in likely only a few years and think in horror at all that we've destroyed. The best art blog about a half decade including a kid who's dead now just disappeared. When Myspace deleted all of their content, I lost all the content of my dead high school friend (turns out you need to back up everyone and thing you care about, not just yourself[1]) and @maxfenton mentioned "The best art blog about a half decade including a kid who's dead now just disappeared.". It's not an isolated problem. I almost feel like we need a name-and-shame registry and/or public awareness campaign. I'd love to see this written into the user rights of a website when it launches. "If the website is to be shut down, all publicly accessible content will be sent to the Internet Archive. If you would like to opt out, tick this box." [1]: http://smerity.com/articles/2013/deleted_digital_tombstones.html http://smerity.com/articles/2013/deleted_digital_tombstones....

mantrax5 12y ago

"If the destruction of all this unique user-generated content pisses you off -- and it should" I'm not sure why it should. When did we get this strange idea that when you make some piece of content, no matter if it's quality or not, it deserves to pollute the cyberspace forever. I realize this is an unpopular opinion, but one of the reasons nature "works" is that everything old is allowed to dissolve and become the base ground for the new that will come after it. People naturally copy content that they find interesting. And content which isn't that interesting naturally gets lost. Maybe that lost content stuck around enough to inspire someone to do something in that spirit later or (which also counts), but it doesn't survive itself. Imagine if everyone gets Glasses one day and starts doing non-stop 1080p video stream of their lives. Are we going to be "pissed off" at that being deleted too? If you do the stats you'll notice that over 99.99% of the content in archive.org is never accessed. Nobody cares. And that's how it should be. We should really allow content to die. I have the firm belief that information doesn't just "want" to be free. It also wants to be allowed to die gracefully at some point.

ivank 12y ago
People sometimes make copies of things they find interesting, but those copies are rarely accessible to the public. It can be pretty hard to get your hands on something after it disappears from the web. For example, forums.truecrypt.org went down three days ago, it's of interest to some people, and you're going to have a hell of a time finding some guy who happened to launch HTTrack on it before it went down.
- mantrax5 12y ago
  Valid points, but you need some strategy for pruning the archive. We can't just keep everything forever. First it was text files, then html files, then html files with images, and now videos. Without pruning, the task of archive.org will become impossible, so the question of what's important and what isn't will have to be decided at some point.
voltagex_ 12y ago
I'm currently running through a Peter Norton assembly tutorial from the 1980s. It's already helped me understand the code better than I have before. Should that content have died, just because it's old?