5 ms·

How exactly does this work? Is this farming out the work of wget-ing URLs, followed by making those contents available on archive.org Wayback Machine? It seems

How exactly does this work? Is this farming out the work of wget-ing URLs, followed by making those contents available on archive.org Wayback Machine? It seems trivial to poison the uploaded content. And there's no way to validate that the content has not been tampered with in flight, unless a redundant fetch were performed in a trusted environment, at which point farming out the work serves no purpose. Secondly, this doesn't solve the discoverability problem. Sure, after Blogger shuts down, a particular webpage on a blog might be preserved. But you will only be able to view its content 10 years in the future if you happen to already know the exact URL where it used to exist.

pests 3y ago
This is the same process ArchiveTeam has used for all their previous work as well and there have been no issues. Isn't a potentially-tampered copy better than none at all?
- lopkeny12ko 3y ago
  "This vulnerability has always existed and it's never been an issue in the past" is not a very robust approach to security.
  - Larrikin 3y ago
    What is your solution that can begin preserving content from today? Tearing into a working solution is easy.
  - pests 3y ago
    I hate to be this guy but, go donate some code or time and make it happen?
- kelnos 3y ago
  > Isn't a potentially-tampered copy better than none at all? I guess it depends on the specific nature of the tampering, but I can think of quite a few situations where no copy is indeed better.
- progval 3y ago
  It was actually an issue with the Reddit archival, but it wasn't malicious. Some people made changes to the code they thought would make it faster, but actually corrupted results.
bradknowles 3y ago
ArchiveTeam has their own "Warrior" tool they run, but it does effectively farm out the equivalent of wget to a bunch of client machines. Their tool is very careful to preserve the content exactly as it was and in an archival format. The problem I had with ArchiveTeam is that they are very insular, and don't accept help quickly or easily from someone who is not already inside their inner circle. I'm guessing they've been burned more than a few times. There are publicly available versions of their tools, but those aren't the versions that actually get deployed. You have to know or be told the private locations for the tools that actually get deployed. What's important to them is not how much resources you could help bring to bear to solving this problem that you care about right now, but are you still going to be here in a year to help them with the 99th problem that you don't care about? I can't say that I blame them for being insular in the face of whatever their past experiences were. I just wish that they had let me join and help them, because I do think they have a good mission and they deserve to be helped. But if they won't let you help, then all you can really do is sit on the sidelines and watch. Or not, as you choose. I chose not to stick around.
- jacquesm 3y ago
  I sympathize with both you and AT on this. I spent a ton of time archiving geocities when it went down and that got a lot of attention, some of it seemed to be more to social engineer the other party into being able to access what I'd already crawled rather than to help out as they pretended. But when you genuinely are trying to help it can be pretty frustrating to find a party skeptical or non-responsive. Eventually I turned the project over to someone else. It's still alive ( https://geocities.restorativland.org/ https://geocities.restorativland.org/ ) and the reocities.com domain still redirects there so at least it wasn't for nothing.
  - pembrook 3y ago
    > It's still alive ( https://geocities.restorativland.org/ https://geocities.restorativland.org/ ) and the reocities.com domain still redirects there so at least it wasn't for nothing. This is amazing, thanks for sharing!
supriyo-biswas 3y ago
As I understand it, Archiveteam doesn't do anything to mitigate the issue, which is why they used to discourage users with a "censored" internet to avoid running the warrior. They could use some form of quorum, where the archived page is distributed to multiple users, preferably in different countries, and only accepted if the results mostly agree. (Given dynamic pages from some of the archived services, it is an interesting problem as to how you would compare if two pages are almost the same.)
- rakoo 3y ago
  Ah, I had the same thought you had and didn't think of dynamic content. Maybe using variable length blocks split with content-defined chunking, and if 99% of the blocks are the same then assume the content is the same, and store the 2 versions ?
init2null 3y ago
The risk of poisoned content puts this squarely in the domain of our ancestor's storytelling history. This won't be admissible in a court of law, but it'll be a piece of our pre-LLM past occasionally flavored by the storytellers. I still think that's worth it.
ghaff 3y ago
Discoverability is an issue with the IA in general. A lot of content is preserved that no one will ever stumble across.
- xeromal 3y ago
  Luckily that's a problem that can be solved in the future.
  - h4ckerle 3y ago
    I don't think it will. Discovery in Archives has been a problem forever and with more content made today than ever before it is becoming ever more difficult.

pests 3y ago

This is the same process ArchiveTeam has used for all their previous work as well and there have been no issues. Isn't a potentially-tampered copy better than none at all?

lopkeny12ko 3y ago
"This vulnerability has always existed and it's never been an issue in the past" is not a very robust approach to security.
- Larrikin 3y ago
  What is your solution that can begin preserving content from today? Tearing into a working solution is easy.
- pests 3y ago
  I hate to be this guy but, go donate some code or time and make it happen?
kelnos 3y ago
> Isn't a potentially-tampered copy better than none at all? I guess it depends on the specific nature of the tampering, but I can think of quite a few situations where no copy is indeed better.
progval 3y ago
It was actually an issue with the Reddit archival, but it wasn't malicious. Some people made changes to the code they thought would make it faster, but actually corrupted results.

bradknowles 3y ago

ArchiveTeam has their own "Warrior" tool they run, but it does effectively farm out the equivalent of wget to a bunch of client machines. Their tool is very careful to preserve the content exactly as it was and in an archival format. The problem I had with ArchiveTeam is that they are very insular, and don't accept help quickly or easily from someone who is not already inside their inner circle. I'm guessing they've been burned more than a few times. There are publicly available versions of their tools, but those aren't the versions that actually get deployed. You have to know or be told the private locations for the tools that actually get deployed. What's important to them is not how much resources you could help bring to bear to solving this problem that you care about right now, but are you still going to be here in a year to help them with the 99th problem that you don't care about? I can't say that I blame them for being insular in the face of whatever their past experiences were. I just wish that they had let me join and help them, because I do think they have a good mission and they deserve to be helped. But if they won't let you help, then all you can really do is sit on the sidelines and watch. Or not, as you choose. I chose not to stick around.

jacquesm 3y ago
I sympathize with both you and AT on this. I spent a ton of time archiving geocities when it went down and that got a lot of attention, some of it seemed to be more to social engineer the other party into being able to access what I'd already crawled rather than to help out as they pretended. But when you genuinely are trying to help it can be pretty frustrating to find a party skeptical or non-responsive. Eventually I turned the project over to someone else. It's still alive ( https://geocities.restorativland.org/ https://geocities.restorativland.org/ ) and the reocities.com domain still redirects there so at least it wasn't for nothing.
- pembrook 3y ago
  > It's still alive ( https://geocities.restorativland.org/ https://geocities.restorativland.org/ ) and the reocities.com domain still redirects there so at least it wasn't for nothing. This is amazing, thanks for sharing!

supriyo-biswas 3y ago

As I understand it, Archiveteam doesn't do anything to mitigate the issue, which is why they used to discourage users with a "censored" internet to avoid running the warrior. They could use some form of quorum, where the archived page is distributed to multiple users, preferably in different countries, and only accepted if the results mostly agree. (Given dynamic pages from some of the archived services, it is an interesting problem as to how you would compare if two pages are almost the same.)

rakoo 3y ago
Ah, I had the same thought you had and didn't think of dynamic content. Maybe using variable length blocks split with content-defined chunking, and if 99% of the blocks are the same then assume the content is the same, and store the 2 versions ?

init2null 3y ago

The risk of poisoned content puts this squarely in the domain of our ancestor's storytelling history. This won't be admissible in a court of law, but it'll be a piece of our pre-LLM past occasionally flavored by the storytellers. I still think that's worth it.

ghaff 3y ago

Discoverability is an issue with the IA in general. A lot of content is preserved that no one will ever stumble across.

xeromal 3y ago
Luckily that's a problem that can be solved in the future.
- h4ckerle 3y ago
  I don't think it will. Discovery in Archives has been a problem forever and with more content made today than ever before it is becoming ever more difficult.