7 ms·
Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty
by livueta 6y ago
Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints.
Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice.
Unlike HTTP, you get reasonable retry behavior on network hiccups.
Also, more robust data integrity guarantees, though a manual hash test is probably a good idea either way.
At least among people I'm throwing TBs of data around with, torrent infra is common and it's nice to not have to deal with some special-purpose tool that, in practice, is probably a pain in the ass to get compatible versions deployed across a range of OSes. Basically every platform known to man can run a torrent client of some sort.
And obviously, no dependency on an intermediary. This is good if you're trying to avoid Google et al. That does, however, bring a potential con: if my side is fast and your side is slow, I'm seeing until you're done. If I'm uploading something to gdrive or whatever, I can disconnect one r the upload is done. If you control an intermediary like a seedbox, that's less of a problem.
In general, though, torrents are pretty great for this sort of thing. Just encrypt your stuff beforehand.
- mderazon 6y agoCan you recommend easy software to do the creation? Don't you need a tracker as well?
- deleted 6y ago[deleted]
- jedimastert 6y ago+1
- parliament32 6y agoAlmost all BT clients can create torrents. You can use a public tracker if you want, or you can use a magnet link (DHT magic will find the peer eventually), or worst-case scenario you can manually add the peer IP on either end.
- tick_tock_tick 6y agoqBittorrent has a built in tracker for private torrents https://www.ghacks.net/2019/09/20/how-to-create-a-private-torrent-using-qbittorrent/ https://www.ghacks.net/2019/09/20/how-to-create-a-private-to... this method will also not leak information into the DHT though 99% of the time you don't care about that.
- slim 6y agotransmission is pretty complete and easy to use
- pessimizer 6y agoWith transmission-create you can create a torrent from the commandline. Load it into your bittorrent client, find its hash in the UI, and make it into a magnet link, like: magnet:?xt=urn:btih:[this is where the hash goes] Example: magnet:?xt=urn:btih:a008663c4fc7e710bd59200e5ef99ba04f83f319
- smabie 6y agoThere's tons of available trackers. Just use whatever one you want. Also most clients can create torrents.
- libertine 6y agoSometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers). I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something right.
- aargh_aargh 6y ago> people need dedicated clients, instead of it working for example in browsers "Version 9.0 [...], released on June 20, 2006" "Opera introduced [...] a built-in BitTorrent client" [1] That's history now, of course, courtesy of today's browser monoculture. [1] https://en.wikipedia.org/wiki/History_of_the_Opera_web_browser#Version_9 https://en.wikipedia.org/wiki/History_of_the_Opera_web_brows...
- ogisan 6y agoWebTorrent [1] can be used to run BitTorrent on the web. It's actually integrated into the Brave browser which is a nice feature. Other browsers might catch on soon. [1] https://webtorrent.io/ https://webtorrent.io/
- mc10 6y agoWebTorrent is pretty limited though: > ... a browser-based WebTorrent client or "web peer" can only connect to other clients that support WebTorrent/WebRTC. (from https://webtorrent.io/faq https://webtorrent.io/faq) This essentially splinters the world into the bittorrent and webtorrent halves, which makes it far less useful.
- myself248 6y ago> Unlike HTTP, you get reasonable retry behavior on network hiccups Seriously. Zmodem/resume was doing better in 1986.
- arsome 6y agowget -c, it ain't hard.
- oasisbob 6y agoThe problem isn't the clients. The HTTP server needs to provide optional support for range requests on that resource. (RFC 7233) Not all do.
- arsome 6y agoEvery major webserver supports resume.
- asveikau 6y agoYou need to notice the connection died and do that manually, or think ahead and have your script do it in a loop. A bt client already has the reconnect logic. And it will hash the contents to make sure it worked correctly.
- StavrosK 6y agoBitTorrent is the best for sending large files. I wish there existed a tool to make torrent creation easier, and I also wish my computer could seed. Every time I create a torrent, nobody can discover my server, even though the port is open and even if I add the server IP as a peer directly onto the client. I have no trouble seeding in swarms, only when I create torrents. Other than that, BT is ideal.
- arsome 6y agoJust be careful - unless you mark that torrent private it'll get posted to the DHT and crawlers like BtDig will pick it up and list it publicly. For this reason I prefer using something like Syncthing which is designed more with this purpose in mind.
- Polylactic_acid 6y agoYou could also gpg encrypt the file first which is a little more work but bulletproof. Maybe someone should make a tool that creates a torrent and encrypts it for you and presents a password box on the other end.
- voltagex_ 6y agoHow are BtDig crawling the DHT? I thought this was made impossible or at least more difficult a while ago.
- chid 6y agoI don't see how it was made impossible. It's definitely possible, you can do it yourself there's many examples on github (even btdig has an old version on it)
- 3np 6y agoIts trivial to disable PEX and DHT in any reasonable client
- orionblastar 6y ago
- hoorayimhelping 6y ago>Just encrypt your stuff beforehand. This is such a facepalm moment. "This service is awesome, it does everything you need, it's reliable, it's safe, it's practical, and all you have to do is encrypt your files, share encryption keys, validate identities, and establish trust with the person you're sharing with. Also if you screw it up in any way, your file is public forever. It's so rad!"
- yjftsjthsd-h 6y agoNo, "here's a magnet link, and here's the matching 50-char password; don't share the password with anyone you don't want have file". Just because you can make security more complicated doesn't mean you're obligated to.
- ekianjo 6y ago> Also if you screw it up in any way, your file is public forever. You can literally say that for ANYTHING out there. Not sure what point you are trying to make. Let me guess, "Encryption is hard?". Well there's no free lunch.
- nikisweeting 6y agoWell there was free lunch when Mozilla's Send thing was running. Sharing the password in the URL hash was brilliant. But then it ran into all the same Mega.nz problems where people immediately use it to host malware. Encryption is hard, which is why it's sad to see that when someone gets it right and nails the UX, it still gets ruined by the real world somehow.
- deleted 6y ago[deleted]
- efreak 6y agoI had to transfer 2tb of data over a wireless connection that kept dropping packets. BitTorrent was the only method I found that worked; everything else either broke down when it disconnected, or had too low of a retry limit to be useful, or wouldn't resume properly.
- deleted 6y ago[deleted]
- cryo 6y agoI've written a cross platform gui tool to transfer arbitrary large files via p2p with end-to-end encryption. https://cryonet.io https://cryonet.io - Fully p2p, even the signalling server part which is done via Tor hidden services (later on there will be an alternative using a DHT). - So unless the Internet or Tor goes offline it can never be down. - Works with proxies and in corporate environments when UDP might be blocked.