5 ms·

This sounds a lot like torrents, and I would think that it'd be less work to solve the problem of newer snapshots of torrents (or put all the new content into n

This sounds a lot like torrents, and I would think that it'd be less work to solve the problem of newer snapshots of torrents (or put all the new content into new "chunks" and thus new torrents) vs. re-inventing the entire thing.

martindale 12y ago
This is already solved with BEP 39 [1]. It wouldn't be difficult for the Internet Archive to implement, either. [1]: http://www.bittorrent.org/beps/bep_0039.html http://www.bittorrent.org/beps/bep_0039.html
sp332 12y ago
The tracker would have to manage millions of torrents, and each client would have to figure out which subset of torrents it wanted to download. And there needs to be a mechanism for the server to force a client to recompute hashes from time to time. So I don't see how it's very torrent-like.
- db48x 12y ago
  Torrents would work perfectly well, and are in fact one of the proposed solutions. The problems you mention are real though. One idea is to use a custom BitTorrent client which the user can install (a bit like the ArchiveTeam Warrior VM). The user would tell it how much disk space they want to donate, and the client would join swarms over time to fill up that donated space. The user could even join specific swarms representing content that they enjoy and particularly want to save, but wouldn't have to go to the hassle of picking every torrent manually. Also, IA already provides a torrent for every single item in the archive with IA as a web seed, so this could be pretty simple to do from their perspective.
- e12e 12y ago
  If one uses one torrent per chunk (500 MB), that's just 42K torrents -- as mentioned here[1]: "Update: Here’s a copy of 17 million torrents from Bitsnoop.com, pretty much the same format but nicely categorized. It’s only 535 MB." So, that's 17M [magnet]s, which means the archive could grow by some orders of magnitude from it's current need of 42K [magnet]s, and still doling out a subset to clients seems to be quite possible to manage. No needs for trackers any more, [magnet]s work fine, and are regularly used to distribute ~1GB torrents (eg: hd tv episodes). Whatever one might think of distributing unlicensed copies of media -- they show that distributing vast amounts of data is quite easy with technology that is readily available and well tested. [1] https://torrentfreak.com/download-a-copy-of-the-pirate-bay-its-only-90-mb-120209/ https://torrentfreak.com/download-a-copy-of-the-pirate-bay-i...
  - TheLoneWolfling 12y ago
    What we need now is a torrent client that can prioritize. In other words: here's this absurd number of torrent chunks, here's the space I have, now what are the best chunks to archive for the health of the swarm?
    - renata 12y ago
      Most torrent clients already prefer rare chunks, assuming the user doesn't turn on "Prefer chunks in streaming order" or whatever similar option the client has.
      - TheLoneWolfling 12y ago
        I don't know of a torrent client that will allow me say "download at least these files of the torrent, and the 100mb of anything else that's the most likely to help the swarm". Downloading the chunks that are the rarest first is different than downloading only rare chunks - although it's good that some of the infrastructure is already in place.
  - rakoo 12y ago
    As they say themselves (http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/torrents_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/t...), that would be 42k * 500 Gb. Can you imagine a torrent with a single chunk of 500Gb ?
    - sp332 12y ago
      e12e said one torrent per 500GB chunk, not one chunk per 500GB.

martindale 12y ago

This is already solved with BEP 39 [1]. It wouldn't be difficult for the Internet Archive to implement, either. [1]: http://www.bittorrent.org/beps/bep_0039.html http://www.bittorrent.org/beps/bep_0039.html

sp332 12y ago

The tracker would have to manage millions of torrents, and each client would have to figure out which subset of torrents it wanted to download. And there needs to be a mechanism for the server to force a client to recompute hashes from time to time. So I don't see how it's very torrent-like.

db48x 12y ago
Torrents would work perfectly well, and are in fact one of the proposed solutions. The problems you mention are real though. One idea is to use a custom BitTorrent client which the user can install (a bit like the ArchiveTeam Warrior VM). The user would tell it how much disk space they want to donate, and the client would join swarms over time to fill up that donated space. The user could even join specific swarms representing content that they enjoy and particularly want to save, but wouldn't have to go to the hassle of picking every torrent manually. Also, IA already provides a torrent for every single item in the archive with IA as a web seed, so this could be pretty simple to do from their perspective.
e12e 12y ago
If one uses one torrent per chunk (500 MB), that's just 42K torrents -- as mentioned here[1]: "Update: Here’s a copy of 17 million torrents from Bitsnoop.com, pretty much the same format but nicely categorized. It’s only 535 MB." So, that's 17M [magnet]s, which means the archive could grow by some orders of magnitude from it's current need of 42K [magnet]s, and still doling out a subset to clients seems to be quite possible to manage. No needs for trackers any more, [magnet]s work fine, and are regularly used to distribute ~1GB torrents (eg: hd tv episodes). Whatever one might think of distributing unlicensed copies of media -- they show that distributing vast amounts of data is quite easy with technology that is readily available and well tested. [1] https://torrentfreak.com/download-a-copy-of-the-pirate-bay-its-only-90-mb-120209/ https://torrentfreak.com/download-a-copy-of-the-pirate-bay-i...
- TheLoneWolfling 12y ago
  What we need now is a torrent client that can prioritize. In other words: here's this absurd number of torrent chunks, here's the space I have, now what are the best chunks to archive for the health of the swarm?
  - renata 12y ago
    Most torrent clients already prefer rare chunks, assuming the user doesn't turn on "Prefer chunks in streaming order" or whatever similar option the client has.
    - TheLoneWolfling 12y ago
      I don't know of a torrent client that will allow me say "download at least these files of the torrent, and the 100mb of anything else that's the most likely to help the swarm". Downloading the chunks that are the rarest first is different than downloading only rare chunks - although it's good that some of the infrastructure is already in place.
- rakoo 12y ago
  As they say themselves (http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/torrents_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/t...), that would be 42k * 500 Gb. Can you imagine a torrent with a single chunk of 500Gb ?
  - sp332 12y ago
    e12e said one torrent per 500GB chunk, not one chunk per 500GB.