4 ms·
Yeah, need to write the program and distributed in r/datahoarder This should be next on my list since my current project depends on SD model and having data ba
by liuliu 4y ago
Yeah, need to write the program and distributed in r/datahoarder
This should be next on my list since my current project depends on SD model and having data backed up gives me confidence that I can get rid of all their stuff if needed.
- MayeulC 4y agoHmm, you can just create a torrent out of it. Either as a single file (impractical, but you can just avoid downloading it fully), or chunk it into multiple files. You don't even need to store it all at once on your computer: stream it and generate checksums on the fly. Then distribute the torrent, and seed sections at a time. It can also be distributed on IPFS. I've seen a lot of torrents being used for distributing neural network (mostly stable diffusion forks).
- liuliu 4y agoGood point. Let me think through this. I have about 15T free space so this can be seeded separately in ~12 batches.
- MayeulC 4y agoYou can also distribute different torrents, it's easier. Or multiple IPFS CIDs. I think you can have a "directory" (CID) that contains multiple CIDs, and only need the content hashes to build it. You can also publish multiple CIDs and ask people to seed random ones; that's how Libgen does it (and is similar to the multiple torrents concept). The same file can be used to seed both torrents and IPFS.