3 ms·
Hmm, you can just create a torrent out of it. Either as a single file (impractical, but you can just avoid downloading it fully), or chunk it into multiple file
by MayeulC 4y ago
Hmm, you can just create a torrent out of it. Either as a single file (impractical, but you can just avoid downloading it fully), or chunk it into multiple files.
You don't even need to store it all at once on your computer: stream it and generate checksums on the fly. Then distribute the torrent, and seed sections at a time. It can also be distributed on IPFS.
I've seen a lot of torrents being used for distributing neural network (mostly stable diffusion forks).
- liuliu 4y agoGood point. Let me think through this. I have about 15T free space so this can be seeded separately in ~12 batches.
- MayeulC 4y agoYou can also distribute different torrents, it's easier. Or multiple IPFS CIDs. I think you can have a "directory" (CID) that contains multiple CIDs, and only need the content hashes to build it. You can also publish multiple CIDs and ask people to seed random ones; that's how Libgen does it (and is similar to the multiple torrents concept). The same file can be used to seed both torrents and IPFS.