5 ms·
Ask HN: Fastest way to tranfer large files?
It's 2019 and I have fiber in two locations with computers that I need to transfer large files between quickly. What's the fastest way to do that?
G-Drive has always capped me around 15mb/s and then it still has to be downloaded. Bittorrent Sync / Resilio doesn't want to go above 500kbps.
- golem14 8y agoYou can copy chunks of the large files in parallel. Entire doable but needs thorough testing.
- golem14 8y agoActually, https://www.sanger.ac.uk/science/tools/pcp-parallel-copy https://www.sanger.ac.uk/science/tools/pcp-parallel-copy seems to have an implementation- have not tried it myself.
- shpx 8y agorsync -P -e ssh <username>@<hostname>:/file/you/want/to/copy.txt ~/wherever/you/want/to/put/it "-P" is to report progress and to not delete files that haven't been transferred completely (in case the connection drops out) and "-e ssh" is to transfer files over SSH.
- mindcrime 8y agoPossibly GridFTP[1], UDT[2], or Tsunami[3]? Also consider the old adage: "Nothing beats the bandwidth of a station wagon full of tapes hurtling down the freeway." [1]: https://en.wikipedia.org/wiki/GridFTP https://en.wikipedia.org/wiki/GridFTP [2]: https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protoc... [3]: https://en.wikipedia.org/wiki/Tsunami_UDP_Protocol https://en.wikipedia.org/wiki/Tsunami_UDP_Protocol
- imagetic 8y agoAnd today we drove hard drives to save 6 hours!
- imagetic 8y agotranfer? I would have a typo in the title.
- rolfeb 7y agoPretend it is 1990... FTP.
- _shadi 7y agoif you can open any port on the receiver computer then I would go with nc. Receiver: nc -q 1 -l -p $PORT | tar xv Sender: tar cv . | nc -q 1 $IP_ADDRESS $PORT
- lastofus 7y agoIf you are serious about saturating your pipe as a business with some cash to spend, there is Aspera. My understanding is that they build their own protocol on top of UDP to avoid pesky behavior of TCP.
- imagetic 7y agoAspera I have used for bigger TV / production work. The bill breaks the budgets for my freelance work.
- Adamantcheese 7y agoWouldn't you be speed limited by the lowest bandwidth equipment in the mix? Unless it's fiber (or equivalent bandwidth) from end to end you're not going to be able to go faster than what the slowest machine will process. If your outbound bandwidth is limited, you may want to compress files locally first, to save on transfer time. That being said, 64-bit scp will probably work fine? If there's no restrictions, making it a torrent would probably work as well, if you need to transfer them more passively.
- imagetic 7y agoIn this situation both points are on consumer fiber (AT&T). A 28 GB file takes roughly 1 hour to upload to G-Drive and 1-2 hours to download on the other side. Speeds caps out at 14 MB/s using most services like G-Drive, Dropbox, etc. Multipart/chunking with S3 / Google Storage may yield faster results. Bittorrent/Resilio Sync is extremely slow, sub 1MB/s.
- elamje 7y agohttps://send.firefox.com https://send.firefox.com seems good
- imagetic 7y agoWe're trying to send 28GB files though.
- Nextgrid 7y agoSFTP, although you might be limited by the encryption’s performance. If authentication & confidentiality aren’t needed you can downgrade to a quicker (& less secure) cipher such as arcfour.