4 ms·
What if Archive.today published torrents of its datasets?
On Friday there was a thread about "who's behind Archive.today?" [0], and it linked to a post by the developer saying [1]:
Archive.today is doomed to eventually die.
It's estimated the quantity of data under management is 1 petabyte. Given they've been going for around 8-10 years, we can extrapolate this is only 100-150 terabytes per year.
While not completely trivial, this is only 5 or 6 hard drives at today's maximum capacity of 28TB.
What if the developer published the data to a torrent on a monthly or other periodic basis? While not completely legal, it would enable their legacy to live on indefinitely and ensure future access to a lot of information.
[0] Archive.today: on the trail of mysterious guerrilla archivists of the Internet - https://news.ycombinator.com/item?id=37009598 - Aug 2023 (58 comments)
[1] "archive.today is doomed to die" https://blog.archive.today/post/657822115776610304/not-respecting-peoples-privacy-copyright-laws
- metadat 3y agoClickable links: [0] Archive.today: on the trail of mysterious guerrilla archivists of the Internet - https://news.ycombinator.com/item?id=37009598 https://news.ycombinator.com/item?id=37009598 - Aug 2023 (58 comments) [1] "archive.today is doomed to die" - https://blog.archive.today/post/657822115776610304/not-respecting-peoples-privacy-copyright-laws https://blog.archive.today/post/657822115776610304/not-respe...
- h2odragon 3y agoPublishing even just the URL table would be a much smaller quantity of data and yet of immense archival value.
- justsomehnguy 3y ago> this is only 5 or 6 hard drives at today's maximum capacity of 28TB And 500 IOPS. It's fine for the personal usage, it's another thing to seed this amount
- metadat 3y agoIf you released a monthly snapshot it'd be ~10TB, which is still a lot but definitely feasible. If it gets packaged as a series of daily tar archives the IOPS won't be a significant issue. Hard disks perform well in sequential reads.
- 22c 3y agoThe beauty of torrents is that as soon as someone else starts to download, provided they're a good swarm member, the overall seeder capacity increases. You don't need to have 100% of the data to start seeding, you only need a single chunk. It might take several weeks for that first peer to hit 100% but it should be quickly followed by a whole lot more as those peers are all seeding each other up to 99% amid the process.
- justsomehnguy 3y agoThanks, my first torrent was published on TPB in 2005, I know how that works. > It might take several weeks for that first peer to hit 100% And this is a problem here. More leechers -> more IO for the random blocks -> less overall IOPS. If you find a torrent client what can advertise the blocks sequentially then it would be much faster.
- 22c 3y agoI think the problem you're talking about is solved by super-seeding. https://en.wikipedia.org/wiki/Super-seeding https://en.wikipedia.org/wiki/Super-seeding I believe libtorrent supports it, among others.
- KomoD 3y ago> What if the developer published the data to a torrent on a monthly or other periodic basis? While not completely legal, it would enable their legacy to live on indefinitely and ensure future access to a lot of information No, it isn't a solution in my opinion, torrents need to be seeded, (at least) initially by Archive.Today, seeding a shit load of multi terabyte torrents is a lot of bandwidth which will probably up costs Torrents die, and I would especially expect massive torrents like that to die out fast, there's not a lot of people who can spare that much storage and/or bandwidth, it's not just "go buy a few drives, put in and seed", especially not for that duration of time.
- metadat 3y agoI download and upload more than 50TB per month, every month, just for my personal stuff. Bandwidth is dirt cheap these days.
- sgjohnson 3y ago> Bandwidth is dirt cheap these days Not at scale. 50TB per month is nothing. That’s 151mbps. Most transit providers will be okay with a 10% commitment, so this is a gigabit port which is like $60-300/mo. Assuming 50 people would download the 1PB archive from the provider, that would require at least 151gbps of bandwidth, rounded up to 200gbps, at the cheapest transit provider (HE) this is still at least $12k/mo for transit alone. Is that dirt-cheap for a non-profit? I’m not sure.
- metadat 3y agoTorrents make distribution even more dirt cheap. See sibling comment: https://news.ycombinator.com/item?id=37028576 https://news.ycombinator.com/item?id=37028576
- sgjohnson 3y agoIn theory yes, but we’re talking about such a massive dataset that archive themselves would have to seed it forever anyway. But yes, in theory, if there would be a lot of seeders for it, this would be a non-issue, however I highly doubt that there are enough people interested in seeding a petabyte worth of archives. It’s certainly less than people who are willing and able to keep a local copy. And if you did that with a consumer ISP, they would terminate you as a customer very quickly. At which point you’ll have to pay cloud provider for traffic, or a T1 network for IP transit (alongside colocation fees and actually putting a server in a datacenter and all that).
- mcdonje 3y ago>It's estimated the quantity of data under management is 1 petabyte. Given they've been going for around 8-10 years, we can extrapolate this is only 100-150 terabytes per year. You're assuming the volume has been flat for the past decade. Not exponential or even linear growth. That doesn't pass the sniff test.