4 ms·

There are actually special file transfer protocols for high-bandwidth high-latency situations where you want to transfer gigabytes/terabytes across the world.

by slyall 14y ago

There are actually special file transfer protocols for high-bandwidth high-latency situations where you want to transfer gigabytes/terabytes across the world. The main example I've heard is special effects companies like Weta Digital in New Zealand wanting to deliver video to the US across a 200ms latency link. There are a few open protocols in various stages of development or you can pay for a product like FileCatalyst http://www.filecatalyst.com/ http://www.filecatalyst.com/ ( some interesting case studies on their website ).

smithian 14y ago
We use Filecatalyst to transfer large files internally and to/from customers. +1 works as advertised. What open protocols were you thinking of?
Karunamon 14y ago
Aspera (http://asperasoft.com http://asperasoft.com) is another one (but my god are they expensive.. the most basic setup possible will run you about $25K). I'm definitely interested if there are open alternatives.
mevodig 14y ago
Yes, you're likely thinking about the Aspera range of file transfer software which is quite popular within the film industry. It's actually quite elegant; a control connection over SSH plus a custom UDP protocol (FASP) for data. It does work as advertised and has some nice features that are handy in production, but licensing is very expensive ($20K for point-to-point at 1gb/s).
efuquen 14y ago
"There are a few open protocols in various stages of development" Can you, or somebody else please name some? It always irks me a little when someone just mentions that "these projects" exist, and then fail to mention any of them.
- EvanAnderson 14y ago
  We had a discussion about this over on Server Fault (http://serverfault.com/q/111813/7200 http://serverfault.com/q/111813/7200) and UFTP (http://www.tcnj.edu/~bush/uftp.html http://www.tcnj.edu/~bush/uftp.html), UDT (http://udt.sourceforge.net/ http://udt.sourceforge.net/), and Tsunami-UDP (http://tsunami-udp.sourceforge.net/ http://tsunami-udp.sourceforge.net/) all came up as viable open source offerings (along with FileCatalyst and Aspera in the commercial space).
- Terretta 14y ago
  No need to over engineer... We offer a custom protocol file transfer as a service (uses UDP instead of TCP), but you can do a home grown solution yourself very easily. If you have a big file of compressed content (video), do the following. If you have a lot of small files, tar and gzip them, then do the following. md5 your file split your file into chunks (eg parts _a - _z, or _aa - _zz) use ncftp or similar capable of parallel transfers optimize between 8 and 30 parallel depending on latency to saturate link concat the file at the other end check the md5 You can handily saturate a link from EU to Australia or India to US this way. That said, for Aussie customers sending us 1000s of movies, we FedEx or DHL pelican cases full of drives.

by slyall 14y ago

smithian 14y ago

We use Filecatalyst to transfer large files internally and to/from customers. +1 works as advertised. What open protocols were you thinking of?

Karunamon 14y ago

Aspera (http://asperasoft.com http://asperasoft.com) is another one (but my god are they expensive.. the most basic setup possible will run you about $25K). I'm definitely interested if there are open alternatives.

mevodig 14y ago

Yes, you're likely thinking about the Aspera range of file transfer software which is quite popular within the film industry. It's actually quite elegant; a control connection over SSH plus a custom UDP protocol (FASP) for data. It does work as advertised and has some nice features that are handy in production, but licensing is very expensive ($20K for point-to-point at 1gb/s).

efuquen 14y ago

"There are a few open protocols in various stages of development" Can you, or somebody else please name some? It always irks me a little when someone just mentions that "these projects" exist, and then fail to mention any of them.

EvanAnderson 14y ago
We had a discussion about this over on Server Fault (http://serverfault.com/q/111813/7200 http://serverfault.com/q/111813/7200) and UFTP (http://www.tcnj.edu/~bush/uftp.html http://www.tcnj.edu/~bush/uftp.html), UDT (http://udt.sourceforge.net/ http://udt.sourceforge.net/), and Tsunami-UDP (http://tsunami-udp.sourceforge.net/ http://tsunami-udp.sourceforge.net/) all came up as viable open source offerings (along with FileCatalyst and Aspera in the commercial space).
Terretta 14y ago
No need to over engineer... We offer a custom protocol file transfer as a service (uses UDP instead of TCP), but you can do a home grown solution yourself very easily. If you have a big file of compressed content (video), do the following. If you have a lot of small files, tar and gzip them, then do the following. md5 your file split your file into chunks (eg parts _a - _z, or _aa - _zz) use ncftp or similar capable of parallel transfers optimize between 8 and 30 parallel depending on latency to saturate link concat the file at the other end check the md5 You can handily saturate a link from EU to Australia or India to US this way. That said, for Aussie customers sending us 1000s of movies, we FedEx or DHL pelican cases full of drives.