4 ms·
If I understand correctly, 2 is a problem of how already existing torrents were created. But if we, say, somehow could convert all torrents to [{pathname, hashe
by wruza 7y ago
If I understand correctly, 2 is a problem
of how already existing torrents were created. But if we, say, somehow could convert all torrents to [{pathname, hashes:[...]}] of 64k 0-padded blocks? (Not zero-pad blocks on a wire, only hash this way.) If such tech were seen as useful, it seems that clients only need to rehash contents once, save a map to original torrent and let data go, sort of a backwards-compatible upgrade step.
Ed: seen your edit, will read later, thanks for your thoughts on this.
- derefr 7y agoYeah, adds overhead, but works somewhat in the “easy” case where it’s the torrent client itself producing the packfile. But most files we might want to find on the network aren’t actually left “unpacked” on disk by their seeders and only virtually packed at torrent creation time; but rather are sitting in existing archive files that the seeder wants to redistribute unchanged (usually because the archive is cryptographically signed, or is a functioning self-extracting installer, or has some other property it would lose if the extraction of the archive’s contents were given instead.) In such cases, a naive index wouldn’t work; but you could just slap libarchive into torrent clients, allowing them to pull out the leaf-node hashes from files packed into an archive file at any depth (and also the branch-node hashes of any intermediate archives.) In a sense, we’re talking about the same thing that virus scanners do, except the “signature database” is a produced and shared artifact of the network rather than a manually-curated set! This requires every torrent file in existence to be re-created from the source data, though, which is probably unlikely at this point, unless someone is planning to write the equivalent of Archive.org’s Wayback Machine to spider all torrent trackers, download everything they’ve got, recreate the torrents, and seed them in the new format, which is hopefully backward-compatible with the old format in such a way that this seed shares peers with the existing seeds.
- wruza 7y agoYeah, I see that this raises more complex questions. On one hand, it could be solved by signing an entire torrent instead of original data, and/or builtin zipping instead of pre-packaging. On the other hand, existing torrents and packaging habits would be likely incompatible. I also thought of a thing similar to libarchive - a format analyzer which could e.g. separate a movie from its subtitles and audio tracks for specific formats, enabling world-wide availability for heavy streams. Or sharing jpegs/mp3s with metadata changed to ones needs, but with main parts intact and thus still seedable. It’s not about really splitting a file in parts, but about detecting better block beginnings, sort of network-wide struct member alignment.