4 ms·
I'm wondering if torrents as such are actually useful for this. I'd figure some kind of virtual file system (perhaps based on BitTorrent) would be very useful.
by jakub_h 10y ago
I'm wondering if torrents as such are actually useful for this. I'd figure some kind of virtual file system (perhaps based on BitTorrent) would be very useful. You'd simply pass a file path to an open() routine in your scientific code and data would get opened transparently. You currently have this with URLs and HTTP but there's no useful caching or data distribution.
- rakoo 10y agoIf you have the torrent file, you can use one of the things that mount them as virtual file systems, like btfs (https://github.com/johang/btfs https://github.com/johang/btfs).
- seanp2k2 10y agoYou might like https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/about.rst https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/ab... , https://urbit.org/docs/using/filesystem/ https://urbit.org/docs/using/filesystem/ , or https://ipfs.io https://ipfs.io BTSync and SyncThing are also tools to do this, and I'm sure there are FUSE things to work with BT and block chains ("bittorrent fuse" google results look promising).
- zielmicha 10y agoIPFS (https://ipfs.io https://ipfs.io) does that.
- Eridrus 10y agoFilesystems have distribution problems. You don't always have permissions to install a new filesystem.
- oscargrouch 10y agoMagnet links from torrents already do this. You just need to have the SHA1 reference of the torrent, working effectively as a url, to load the metadata, and then the files described by the metadata payload. The P2P nature of the network then help the descentralization of sources, populating several clones of the dataset.