6 ms·
Ask HN: How do you handle transferring large files over the internet?
I've often heard that the best way to send large amounts of data is to actually send it via snail mail. But if that is not an option what is the best way to send, say 500GB, over the internet? Past Experience preferred but creative solutions welcome!
- CarolineW 10y agorsync
- Piskvorrr 10y agoIndeed. I've been trying all sorts of weird stuff, but this takes the cake. Ubiquitous, rock-solid, sane. Plus, no worrying "is it done yet? Do I have the latest version?" Just let it run (again) - this makes it rather foolproof.
- kraemate 10y agoDoes rsync auto-resume after failed connections?
- CarolineW 10y agoI've written a script that every minute tests to see if the appropriate rsync command is running. If not, it simply runs it again. In that way it effectively restarts and gracefully resumes. This Google search: https://www.google.co.uk/search?q=rsync+auto-restart+failed+connection https://www.google.co.uk/search?q=rsync+auto-restart+failed+... returns this link: http://superuser.com/questions/302842/resume-rsync-over-ssh-after-broken-connection http://superuser.com/questions/302842/resume-rsync-over-ssh-... which contains this script: #!/bin/bash while [ 1 ] do rsync -avz --partial source dest if [ "$?" = "0" ] ; then echo "rsync completed normally" exit else echo "Rsync failure. Backing off and retrying..." sleep 180 fi done The comment says: When the connection dies, rsync will quit with a non-zero exit code. This script simply keeps re-running rsync, letting it continue until the synchronisation completes normally. That's pretty much what I've done.
- kraemate 10y agoWow, thanks for the script. Surprising in its simplicity. I would have thought this use-case was popular enough to warrant specialized tools etc. Especially in the scientific community where they transfer large files.
- CarolineW 10y agoIt's simple enough that it's the sort of thing I type out in 30 seconds and there it is. No need for specialist tools - finding the tool, remembering how to use it, working out the right parameters ... Easier, faster, and more flexible just to write the script. It's what I do.
- Piskvorrr 10y agoYou mean, if the connection fails before completing? I don't think so (I believe that's a feature). For automated transfers ("retry until done"), I use the lsyncd wrapper (which also watches the source files, so you don't need to poll for changes: it wakes up by inotify).
- atsaloli 10y agoSure, just use --partial switch: --partial keep partially transferred files
- Piskvorrr 10y agoThat is incorrect. This will keep partial transfers in destination (and when invoked again, will continue tranferring from that point onwards), but will not restart an aborted transfer (e.g. for a broken connection). In other words, this feature is a prerequisite for auto-resume, not auto-resume itself (which can be scripted in ~10 lines, as shown in a sibling thread).
- atsaloli 10y agoThanks! But no need to script, you can just use --append append data onto shorter files to resume interrupted transfers.
- atsaloli 10y agoYes, it it does, if you use: --partial keep partially transferred files --append append data onto shorter files
- wmf 10y agohttp://moo.nac.uci.edu/~hjm/HOWTO_move_data.html http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html
- Piskvorrr 10y agoIf, for some reason, you are averse to rsync (there are plenty of Windows clients), then bittorrent is second-best: it works just as well for transferring private data, just turn off all the metadata broadcast options and make sure that encryption is required. (Not so easy for incremental transfers, though)
- lucasch 10y agoWe considered rsync but were wondering if there were more specialized tools available. We figured that those who work in the scientific community would have a way to transfer their large data sets between institutions.
- ColinWright 10y agoWe transfer large files containing raw radar data, and moderate sized files contains databases of target movements and track information. We use rsync. When I worked for the local university we had to transfer data between machines to run experimental parallel programs on so-called "big data." We used rsync.
- lucasch 10y agoAhh ok thats good to hear. Have you considered using multipath transport protocols with something like rsync? I am curious if it could benefit this situation. MPTCP sounds like an interesting protocol if you control both hosts. https://www.multipath-tcp.org/ https://www.multipath-tcp.org/
- ColinWright 10y agoWe were/are always restricted by intermediate limits on throughput, so it's never been useful or interesting to consider alternatives. YMMV, but if you want to improve throughput, consider carefully where your data has to go through. But rsync is rock-solid, well-understood, mature, and just does exactly what it is intended to do.
- 10y ago
- niftich 10y agoI assume you're transferring between two hosts you both control. Some sane options are: - rsync - SFTP-over-SSH, or SFTP over a dedicated VPN - private, encrypted torrents
- Piskvorrr 10y agoSFTP gets tricky on resume; rsync and BT have builtin data integrity checks.
- deleted 10y ago[deleted]
- teddyc 10y agoI would use scp with the -C option to compress the data for the first attempt to transfer the file. Any subsequent attempts to transfer the file should use rsync (this includes any updates to the file). Another option is to use a cloud data storage provider (Dropbox, Box, Google Drive, etc) and install their software that keeps files in sync. Then you can just put the file in a local folder and let their software sync it to the cloud. After sync is complete, you email a link to share the file. Of course, their software is probably just a GUI for rsync.
- Piskvorrr 10y agoWhy not use rsync for the first transfer as well? There's `-z` for the same effect... As for commercial providers - I've seen rate limits (never did it saturate the link) and size limits (0.5 TB will cost you extra). Moreover, the data will go through an intermediate hop, which is slow (Dropbox starts downloading only when upload completes, doubling your transfer time), plus your data is now...somewhere (which could be a regulatory issue, depending on the data).
- Piskvorrr 10y agobtw not relevant in your case, but a friend working for a large video company regularly drives a truckload of 10 TB tapes around the continent - bandwidth is still an issue at that scale.
- lucasch 10y agoAmazing. If only there was away to get around the limitations at that level.
- Piskvorrr 10y agoWell...two decades ago, 500 GB of data would have been moved as freight, too. Data expands to fill any available capacity - I don't think there is any way around that.
- atsaloli 10y agoAbout 9 years ago, while working as a system administration consultant, I had a gig to fly a portable hard drive with about 360 GB from LA to St. Louis as part of a migration of a web application. It was faster than the network connections available to my client at the time. I remember calculating the throughput... I asked why don't you just FedEx it? It's too important, the client said, and we know and trust you. It was funny, I had it in a laptop bag, and didn't let go of it except to go through the security scanner... the only thing missing was the handcuff connecting it to my wrist. :)
- Piskvorrr 10y agoWell...you are probably not going to throw a hard drive at a client's door and run. A delivery guy might, as it's just another cardboard package, not priceless data (might be easier now with SSDs). Indeed, getting a trustworthy courier service is so hard that actually sending an in-house employee is worthwhile, even though their hourly rates make this extremely expensive: you are removing tens of abstraction layers, while preserving high degree of control ("Oh, we might have run it over with a truck. And accidentally put it on a plane to New Zealand on hop #3. And they can't seem to find it there.")
- lastofus 10y agoAspera is often used in the film industry to move large video files around. My understanding is their software takes over some layers of the network stack to make sure the pipe stays saturated. http://asperasoft.com/ http://asperasoft.com/
- atsaloli 10y agoYep, when I worked in the film industry, we used Aspera or GridFTP (open source).
- kennell 10y agoMore details please. * Are we talking a few giant files or thousand of small-ish files? * Who is on the receiving end? Technical people from who we can expect that they are able to run some command-line stuff or your average joe? * What type of scenarios must the solution work? What OSes? Is it acceptable to install extra software or must it work out of the box? etc. etc. etc.
- soulbadguy 10y agoDo you need to transmit 500GB every time or just a diff from a previous dataset ? If it's the later case, using the send/receive functionality of a file system with snapshot and incremental backup (ZFS,BTRFS etc..) can be significantly faster than using pure rsync. Rsync would needs to scan the complete 500GB of data to find the blocks to send, while send/receive can compute the diff much faster
- Piskvorrr 10y agoNegligible W/R/T transfer time. While using native capabilities of ZFS is awesome, you're now locked into a particular FS at both sides of the transfer (this may or may not be an issue). (Also, there's a patch for rsync that allows you to force computing the checksum in batch, not on each invocation; but that's getting into hairy territory that's rarely needed - I used it in exactly one case so far)