3 ms·

UDP for fast file transfer. Working in the television industry at the time and a few companies were making tools based on UDP, for sending large files across p

UDP for fast file transfer. Working in the television industry at the time and a few companies were making tools based on UDP, for sending large files across public networks. Smartjog, Aspera, Signiant. The companies are still around, but only operating in the B2B space as far as I know. I thought this technology was going to replace FTP. People wanting to punt bigger and bigger files around, they needed better tools. Smartjog even open sourced an early version of their tool. Anyway, it didn't happen. I'm not in the televisual industry anymore and find myself entirely satisfied with the current file transfer tools commonly available (FTP!). I guess there was just never consumer demand and I was looking at things from the perspective of a very narrow niche.

nikanj 9y ago
The #1 most used file transfer tool seems to be Dropbox. I put stuff into mine, give you a link and everything magically works.
- phillc73 9y ago
  Yes, that's a nice workflow for non-technical users. However, if you're transfering 50GB files, this takes some time. UDP behind the scenes could fix that. I guess consumers just aren't punting 50GB of data around all that much, so there's not enough wait time pain to justify it.
  - stephen_g 9y ago
    The biggest problem with UDP is NAT, since UDP is connectionless. Still, it is extensively used in all sorts of applications. The reason it's not used to much in file transfer is because you basically need to re-implement everything you get for free with TCP for reliable transfer. Secondly, one of the big things that made TCP slow for large file transfer was that the way most TCP congestion control algorithms worked meant that transfer rates would drop quickly as latency increased. Google have come up with an algorithm that drastically improves this [1], and contributed it to the Linux kernel. I expect their strategy will be adopted by most operating systems. I believe Google is already using it, and I read something on Netflix's tech blog that they are at least trialing it. For small files, TCP slow-start is an issue but protocols like HTTP/2 can work around this by multiplexing multiple downloads through one connection. So I don't think there will be much advantage to UDP based file transfer for things that you want reliability for. 1. http://queue.acm.org/detail.cfm?id=3022184 http://queue.acm.org/detail.cfm?id=3022184
  - nikanj 9y ago
    I just realized I have no idea if Dropbox uses UDP behind the scenes. And I love it. From a larger "tragedy of the commons" point of view, using UDP to force maximum packets down the shared pipes is not fair towards everyone doing TCP. Congestion detection/prevention tries to make traffic more fair to everyone.
swyx 9y ago
that's interesting. Isn't UDP still in use today? (honestly i'm not too clear in what context, i just know that its used in some webapps)
- phillc73 9y ago
  Maybe it is and I'm just not aware of it. I left TV about four years ago, so not sure if the technology has made it to consumer file transfer tools, but as far as I know clients for services like Dropbox (mentioned below) don't use UDP. If they do, and I just don't know about it, then the future is here!
- inDigiNeous 9y ago
  Many games (if not most) use UDP for multiplayer functionality, as it's more bandwith efficient. UDP has no error recovery, packets are sent without any idea whether the receiver got them or not, and there is no handshaking the connection like with TCP, resulting in smaller latencies.
ben1040 9y ago
I used to work in biotech and we were using Aspera to transfer 200-300GB genomics files up to the NIH's central repository, because FTP stopped cutting it at that scale. http://asperasoft.com/ http://asperasoft.com/

nikanj 9y ago

The #1 most used file transfer tool seems to be Dropbox. I put stuff into mine, give you a link and everything magically works.

phillc73 9y ago
Yes, that's a nice workflow for non-technical users. However, if you're transfering 50GB files, this takes some time. UDP behind the scenes could fix that. I guess consumers just aren't punting 50GB of data around all that much, so there's not enough wait time pain to justify it.
- stephen_g 9y ago
  The biggest problem with UDP is NAT, since UDP is connectionless. Still, it is extensively used in all sorts of applications. The reason it's not used to much in file transfer is because you basically need to re-implement everything you get for free with TCP for reliable transfer. Secondly, one of the big things that made TCP slow for large file transfer was that the way most TCP congestion control algorithms worked meant that transfer rates would drop quickly as latency increased. Google have come up with an algorithm that drastically improves this [1], and contributed it to the Linux kernel. I expect their strategy will be adopted by most operating systems. I believe Google is already using it, and I read something on Netflix's tech blog that they are at least trialing it. For small files, TCP slow-start is an issue but protocols like HTTP/2 can work around this by multiplexing multiple downloads through one connection. So I don't think there will be much advantage to UDP based file transfer for things that you want reliability for. 1. http://queue.acm.org/detail.cfm?id=3022184 http://queue.acm.org/detail.cfm?id=3022184
- nikanj 9y ago
  I just realized I have no idea if Dropbox uses UDP behind the scenes. And I love it. From a larger "tragedy of the commons" point of view, using UDP to force maximum packets down the shared pipes is not fair towards everyone doing TCP. Congestion detection/prevention tries to make traffic more fair to everyone.

swyx 9y ago

that's interesting. Isn't UDP still in use today? (honestly i'm not too clear in what context, i just know that its used in some webapps)

phillc73 9y ago
Maybe it is and I'm just not aware of it. I left TV about four years ago, so not sure if the technology has made it to consumer file transfer tools, but as far as I know clients for services like Dropbox (mentioned below) don't use UDP. If they do, and I just don't know about it, then the future is here!
inDigiNeous 9y ago
Many games (if not most) use UDP for multiplayer functionality, as it's more bandwith efficient. UDP has no error recovery, packets are sent without any idea whether the receiver got them or not, and there is no handshaking the connection like with TCP, resulting in smaller latencies.

ben1040 9y ago

I used to work in biotech and we were using Aspera to transfer 200-300GB genomics files up to the NIH's central repository, because FTP stopped cutting it at that scale. http://asperasoft.com/ http://asperasoft.com/