6 ms·
Ask HN: Best modern file transfer/synchronization protocol?
Hi HN,
I'm looking to build something to transfer files between two devices on a network (one client and one server, both of which are under my control).
Obviously I could write something bespoke with raw TCP sockets, but rather than reinventing the wheel I'm curious about what existing options people recommend. I assume there are some better technologies than FTP nowadays?
Ideally some kind of built-in fault tolerance would be great, because my plan is to use this on a phone/in an environment where the connection could be interrupted.
Edit: just to clarify - this is something I want to build into an application I am writing, ideally with support across iOS (client), Windows, and mac (server).
One way transfer is all I need, and I mostly plan on photos/videos (so multiple files ~3-20MB in size).
Thanks!
- smackeyacky 3y agoGrab an S3 bucket on amazon. Do a 3 way sync with the s3 command line tool. That way, you have a neat cloud backup as well. Wouldn't take any more than 20 minutes total to set up.
- jftuga 3y agoRespectfully disagree. You get charged 9 cents per GB after the first 100 GB xfer each month. This doesn't fit in well with the OP's environment, which is a LAN.
- 2color 3y agohttps://iroh.computer/sendme https://iroh.computer/sendme https://iroh.computer/ https://iroh.computer/ Iroh is what you’re looking for. You can embed into your code, and it handled wire encryption, verification, and NAT hole punching. All over QUIC.
- b_fiive 3y agoiroh team member here. OP we’d be happy to help! https://iroh.computer/discord https://iroh.computer/discord if you’d like to chat
- eternityforest 3y agoWhat about Jami? It runs almost everywhere and having an embeddable Jami library would be absolutely amazing, although a fair amount of work.
- arun-mani-j 3y agoOn a similar note, can someone tell me what's the fastest (wireless) way to transfer files between two laptops on same network (i.e. hotspot)? scp, rsync, wormhole give me only 2-3 mb/s. For the context, I'm trying to transfer about 50-70 GB files. What's causing the bottleneck here and what am I missing? Thanks in advance! https://github.com/magic-wormhole/magic-wormhole https://github.com/magic-wormhole/magic-wormhole
- Fire-Dragon-DoL 3y agoFilezilla server + client is very fast and has a resume option
- neurostimulant 3y agoAre you transferring a large amount of small files?
- arun-mani-j 3y agoNo, it is a large zip file.
- neurostimulant 3y agoAre speed tests (e.g. speed.cloudflare.com) on both laptop also giving similar number? If speed test result is much faster, the bottleneck could be the CPU incapable of encrypting/decrypting data fast enough using the default encryption method used by SCP/rsync. In that case, try unencrypted file transfer instead. Maybe just serve the file temporarily with `python -m http.server`
- arun-mani-j 3y agoI shall check the speed on Cloudflare but the speed is not much improved on Python HTTP server too. :(
- 3y ago
- Helmut10001 3y agorsync. If you are looking for a long term solution: zfs on both sides with zfs send.
- tripleo1 3y ago1. warpinator, syncthing 2. rclone?? -- 3. self hosted ipfs in tailscale or something? (that would be cool)
- jayknight 3y agoI would probably start with rsync.
- daveidol 3y agoThanks - is there a reference implementation for rsync that works on windows and iOS? My impression was it was more of a CLI tool for Linux/macOS.
- necovek 3y agoIf you are looking to integrate into another app, maybe check https://librsync.github.io/ https://librsync.github.io/ out.
- daveguy 3y agoUnfortunately one of the caveats about what librsync is not (from the link): librsync also does not include any network functions for talking to SSH or any other server. To access a remote filesystem, you need to provide your own code or make use of some other virtual filesystem layer. Having this seems to be one of the primary requirements.
- necovek 3y agoI did not read the OP that way: cross platform support, fault-tolerance and ability to integrate into a bespoke client/server solution (iow, there is already a network client and a server). Even with that, librsync highlights that rsync is really a protocol rather than the CLI tool GP post referred to.
- teddyh 3y agoFor Windows: <https://community.chocolatey.org/packages/rsync https://community.chocolatey.org/packages/rsync>
- sneak 3y agorsync over ssh for one-shots. syncthing for continuous use.
- thedaly 3y agoI've starting using rclone over rsync for this application. Rclone can do segmented transfers and handles large numbers of files better, at least in my experience.
- sneak 3y agoWhat are segmented transfers?
- daveguy 3y agoSend multiple pieces of the file and reconstruct the file on the other end. It's more reliable to send smaller chunks and the transfer can be hash validated on individual chunks and after reconstruction. Multiple download streams typically performs better too (from single or multiple servers). e.g. BitTorrent for the multiple servers case.
- xnx 3y agorclone is great! So useful as an alternative/superior client for cloud storage too (Google Drive, OneDrive, etc.)
- bhaney 3y agoYeah I usually just use rsync for this. In a loop if the network is unreliable.
- whalesalad 3y agoseconding rsync and syncthing. the server could expose an smb or nfs share, the client could mount it, and then sync to that mount. rsync over ssh also works, if you do not want to run smb/nfs. this is also a cool tool https://rclone.org/ https://rclone.org/
- orbz 3y agoNo need to get fancy, scp or rsync are the tried and true options here.
- brudgers 3y agoOr on Windows, Xcopy.
- krylon 3y agorobocopy is nice.
- PopAlongKid 3y agoOr robocopy.
- jftuga 3y agoYes. It has many options - one of them being the capability of copying multiple files concurrently with the /MT switch.
- ryukoposting 3y agoI use syncthing for this. It's a little fiddly to set up, and transfer speeds aren't great. However, it's very reliable once configured, and it barely uses any resources after an initial scan of the folder you want to sync.
- BrandoElFollito 3y agoI ended up with Syncthing after trying everything else (that O can self-host). You can ease the eureka moment by remembering that each node is completely independent and decided what comes in, and suggests what goes out. You can say that a folder is two way on a node and read only on another - and this is great because tout make decisions locally that allow you to build nice things. The real issue is that you have to check on both sides of the pipe what is allowed. It is very robust and good for LAN and remote sync (the traffic is outgoing to a relay server. Note that many of these relay servers are also TOR nudes so they can be flagged by your severity systems. You can always use your own)
- toomim 3y agoUse HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/ https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...
- Flimm 3y agoI've had terrible experiences downloading large files over HTTP. I'm not sure why, range requests don't seem to be reliable or well supported. Something like BitTorrent is much better for large files: it divides the large file into chunks and hashes each chunk, and by default the chunks are downloaded in random order. BitTorrent seems much more reliable than range requests.
- otherme123 3y agoSame here. I downloaded some ~100Gb files, and everytime the connection broke, after relaunching, the shasum never matched. Even when the file size was exactly the same that the server reported.
- mynegation 3y agoWhat are your latency and bandwidth requirements? How big are the files? If you are already looking past obvious TCP-based choices like HTTP and FTP, you might be interested in FASP/Aspera https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol Edit: I’ll leave it here just in case it is useful for others but it may or may not be embeddable into your app, especially on the phone.
- oschrenk 3y agoIf it’s one way (that wasn’t quite clear from the requirements to me). take a look at https://tus.io/ https://tus.io/
- daveidol 3y agoThis is awesome! I am indeed looking for something one way - so this looks great. Thanks for sharing.
- rmorey 3y agoI cannot recommend rclone enough. Been using it to transfer petabyte-scale datasets flawlessly. Available as librclone as well
- tetris11 3y agoit even has a decent Android client in F-droid, RCX. I just wish it supported SSHFS
- gazby 3y agoRCX is approaching abandonware status (last release 2y ago). Round Sync is worth a look.
- tetris11 3y agoThanks for the recommendation! I searched F-droid but found nothing, and then I realised it was probably.on Izzy
- neurostimulant 3y agoOr just use rclone from inside Termux.
- tetris11 3y agoSure, but then it's not visible across the Android filesystem
- ksjskskskkk 3y agorclone is garbage if you need performance or just copy between two points you control. it shines when you need to sprinkle your data over many "clouds"
- esafak 3y agoSo what do you use?
- rrix2 3y agoI built something on top of the Syncthing API this week after using it on its own for years. A local instance of Syncthing can behave as a robust sync tool + inotify API for applications consuming the files: https://docs.syncthing.net/rest/events-get.html#get-rest-events-disk https://docs.syncthing.net/rest/events-get.html#get-rest-eve... i believe there's an embeddable golang library, but if you want something easy to use on android check in on syncthing-fork which lets you define more granular sync conditions including "just turn on 5 minutes every hour" https://github.com/Catfriend1/syncthing-android https://github.com/Catfriend1/syncthing-android
- pluto_modadic 3y agocontinuous sync - mutagen.io (maybe you could extract some of the libraries) depends on if it's large or small files.
- pvtmert 3y agoI highly recommend Unison (https://github.com/bcpierce00/unison https://github.com/bcpierce00/unison) It allows you to sync between 2 machines (bi-directional) over TCP or SSH. Note that TCP way is not encrypted, you may use wireguard as transport layer encryption for that purpose... You can use an external application to copy if file size is larger than an arbitrary number. (Eg: use rsync for files > 1gb)
- q0uaur 3y agoanother vote for Unison, been using it for about a year for practically everything and it's great. takes a moment to grasp the concept, but it's 100% worth it. that said, i haven't managed to set it up for my android phone yet - it's not available in termux and i have NO idea where to start if i'd like to package it myself. probably has to be done by someone who knows ocaml, since termux' environment is so different from normal linux. unless... maybe i should give proot a shot (chroot in termux, lets you run something closer to linux). but it's another layer of complexity on top of everything....
- yetanother12345 3y agoeh... wget or curl ? or not modern enough for you?