6 ms·
Does archive.org let you download dumps of all of their data? If they make a weekly torrent containing their whole data-set it would be easily immortalised...
by fdsary 12y ago
Does archive.org let you download dumps of all of their data? If they make a weekly torrent containing their whole data-set it would be easily immortalised...
- pronoiac 12y agoThey hit 10 petabytes two years ago* - which is 10000 terabytes. Torrents don't really cut it at that scale. * http://blog.archive.org/2012/10/26/10000000000000000-bytes-archived/ http://blog.archive.org/2012/10/26/10000000000000000-bytes-a...
- kanzure 12y ago> They hit 10 petabytes two years ago* - which is 10000 terabytes. Torrents don't really cut it at that scale. I see no reason why torrents couldn't handle this. Data transport through bittorrent still works even with large numbers of bytes.
- pronoiac 12y agoYou could use floppies to transfer terabytes of data, but that doesn't make it a good idea.
- pronoiac 12y agoAt these scales, bandwidth to disks isn't trivial - at a gigabyte a second, it would still take over 130 days to build the checksums for the torrent. I don't think Bittorrent fundamentally allows multithreaded torrent generation. I think they bundle their data into WARCs of a few (dozen?) megabytes. Copying those around is much more straightforward, with much lower latency.
- zamalek 12y agoI think he could have meant that some DHT could have been used, it needn't be specifically torrents. You could then volunteer your machine to redundantly store, say, 5GB of the archive. If you are willing to spend time copying files around you should donate: your pay-per-hour probably far exceeds the time it would take you to dilly dally with billions of WARCs. Even if you flip burgers for a living. There are far better ways to contribute than by typing 'cp'.
- razster 12y agoA good portion of the images/video are spam - take that into account.