6 ms·

I always wondered why nobody used UDP for bulk data transfer: Ship all the chunks over with a sequential number identifying each, then when you've reached the e

I always wondered why nobody used UDP for bulk data transfer: Ship all the chunks over with a sequential number identifying each, then when you've reached the end, have the client request any lost packets, then repeat the cycle until all are transferred. After all, the client doesn't need everything in sequential order if it knows the size, and it can deal with holes as long as they're eventually repaired. This gives you (in theory, I've never tried it) better throughout, and you get the equivalent of TCP's retransmission without the window semantics, since you move all the retransmissions to the end. On the other hand, it's probably bad for everyone else, since there's no congestion control.

lend000 10y ago
That sounds like a TCP congestion control mechanism, except worse! TCP leaves much to be desired in terms of performance, largely because their congestion control fails miserably on lossy links (TCP RENO). There are better backoff mechanisms that can be added on many distros, like Westwood. And then there are altogether more efficient protocols like QUIC. But designing one for general purposes from UDP is easier said than done.
orasis 10y ago
Back in the day of hard drives, the random access I/O would be nasty and you'd have to be careful about creating sparse files, so you'd end up double writing the missing packets. you'd also still need congestion control so might as well just use TCP to retransmit.
vvanders 10y ago
You've kinda answered your own question there, congestion control becomes a big issue(also MTU discovery to a lesser degree). Games sit in this happy space where the data rate is ~10-20kb at most so they don't need to worry about congestion control as much.
- user5994461 10y ago
  Additional information: The source engine (valve) has been limited to 30 kb/s for more than a decade.
acomjean 10y ago
UDP messages are limited to 64K. It can be a pain to reassemble. lots of ugly corner cases. We had need to send one type of message split due to size with udp. We were on the same subnet sending and receiving so it worked almost always. We were also using multicast. Its very very hard to figure out whats going on when things don't work ( You don't know if the control messages are getting through, when you ask again or if the message just isn't being sent). At some point you are reinventing tcp/ip. I always liked the way udp messages are received though, all or nothing. Not the byte stream that is tcp which needs to be broken into messages manually.
- lobster_johnson 10y ago
  Picking a smallish, fixed chunk size should work, and it could even be made to self-regulate. Reassembly is trivial -- allocate the file first, write chunks to their right location. And for the control stuff (i.e. requesting chunks) you just use a TCP connection. UDP for the data only.
  - jerf 10y ago
    If you do the obvious work to convert UDP to a reliable streaming mechanism, you will generally more-or-less reinvent TCP, only probably more poorly, and would probably be better off just using TCP. UDP is a good choice when you have options or requirements that TCP can't accommodate. For instance, the classic case of streaming audio has both; it both requires timely delivery and has some very sophisticated techniques for dealing with missing packets with a lot of local intelligence about human perception and such built into the codecs. If you can drop packets without issue, if future packets completely supercede past packets (FPS network gaming for instance), there's a lot of possibilities, but if you're just going to recreate reliable streaming you probably be net losing on account of the effort spent doing that instead of something useful, and bugs in the implementation.
    - theWatcher37 10y ago
      Why are so many comments here ignoring the purpose of the article? The purpose of this is for games, not web traffic or torrent2.0. In real time games where you're syncing physics states or player locations data >200ms isn't just useless it's actually negative. This article is 2+ years old, and is the foundation I suspect for libyojimbo which hybridizes UDP low latency communication for large player real time games and reliable messaging (basically custom TCP?) for less time critical things.
blackflame7000 10y ago
What it really comes down to is file block size granularity. TCP is simply using a much finer block size (~1500 bytes) which means a higher likelihood of arriving than for a large UDP packet. Furthermore, you'd still have to implement your own checksumming algorithm to ensure data integrity as UDP doesn't require a packet checksum for IPv4. I suppose you could offload file-block integrity validation to another thread so you can receive and verify in parallel, but at some point you have to checksum every block anyways which means the full file must be processed to figure out where corruption might be before sending retry requests for the missing packets. I see what you're getting at by trying to avoid processing the packets in order if it wastes time, but ultimately I think its a micro-op that introduces other bottlenecks. That's why bit-torrent still typically uses TCP even though they could potentially receive file parts in any order.
- SomeStupidPoint 10y ago
  But bittorrent breaks the files in to chunks it can receive out of order and sends those over TCP, so TCP only preserves ordering within a chunk. It might even get chunks from several different people. The real answer is that chunking over TCP is efficient enough that little is gained by chunking over UDP.
  - blackflame7000 10y ago
    Yes what I'm saying is that bit torrent uses TCP to segment fileblocks instead of using UDP to send the file block as one large datagram
ra1n85 10y ago
Bad for everyone else, and it's up to the implementation to handle flow-control. Congestion-control handles network related loss/latency/load. Flow-control helps with receiver related loss/latency/load.
scurvy 10y ago
NFS on Linux defaulted to UDP for what seems like a really long time. There were perf issues with TCP (can't remember what they were).
- mullen 10y ago
  NFS used to default to UDP for the same reasons that games still would use UDP, it just needed the data as quickly as possible since it was blocking IO. Now a days, we don't need UDP for NFS because networks are a lot faster and the advantages of TCP start to over rule UDP.
  - fulafel 10y ago
    The original reason was that using UDP was much faster CPU-wise back in the 80's. There was a requirement to go fast because it was used for transferring data in bulk, not events like games. When Linux came around it just followed suit. TCP support came to NFS only later with other bells and whistles in later NFS versions, and Linux stuck with basic NFSv2 for a good while. NFS was designed for local-area networks, so it didn't need to worry about working over the Internet. In addition to the network being much faster relative to CPUs back then, the TCP stacks hadn't seen the tuning they have now. Making TCP fast came later.
- tedunangst 10y ago
  NFS (was) stateless and idempotent. Lose a packet? Send it again! Which is mostly what you want, since there can be dozens of processes performing IO and they're all independent. Last thing you want is head of line blocking.
ak217 10y ago
There are many protocols that use this approach, Aspera FASP is possibly the most advanced one. It uses a TCP control channel to manage the rate on the UDP data channel.
justin66 10y ago
> I always wondered why nobody used UDP for bulk data transfer That would be odd. It's not true. There's TFTP all the way up to UDT.
- gregschlom 10y ago
  Although to be fair, given that with TFTP each packet must be acknowledged before the next one is sent, you're actually achieving a way worse throughput than TCP. Not a very good example on how to use UDP efficiently IMO.
  - justin66 10y ago
    UDT is a better example. http://udt.sourceforge.net/ http://udt.sourceforge.net/
kevhito 10y ago
> ship all the chunks over with a sequential number Except that if you ship them too fast, they will at best simply build up in a buffer somewhere (e.g. near a bottleneck link), or more likely they will overflow causing many of your packets to be lost. So not only did the recipient not get them (and you have to figure this out and resend them later), but you also wasted a bunch of resources sending packets only to have them dropped. And if you send them too slow, you are leaving some potential capacity on the table. So you need to send them fast, but not too fast. So you add some additional logic to implement flow control and congestion control. And that's TCP, pretty much.
- ajross 10y ago
  Exactly. The core feature provided by TCP is not reliability, that is trivially easy. What TCP gives us is robust flow control that can detect and adapt to current network capacity.
  - Dylan16807 10y ago
    A robust flow control that's tuned for bulk downloads, and badly adapted for small real-time streams.
    - ajross 10y ago
      Sure. And not even the best known flow control mechanism. But nonetheless that is the important innovation of TCP, not reliability. The post by lobster_johnson upthread implied that you could use UDP for "bulk data transfer", which basically doesn't work without reinventing something akin to TCP.
  - snupples 10y ago
    CNN has a video transfer tool for b2b subscribers that uses multiple UDP streams to increase transfer speeds. Instead it saturated and brought down our company firewalls, which were older and our main bottleneck. All for just a potentially few more megs of speed on TB sized videos. Gotta be careful with UDP.
morgante 10y ago
In practice, I think you'd end up having to reimplement most of TCP on top of UDP anyways.
chaosphere2112 10y ago
On LAN, this is basically how Symantec Ghost works; it multicasts the image over UDP, and each client figures out what they missed. Those chunks get retransmitted at the end.
ajosh 10y ago
There is actually such a protocol - UFTP. It was designed for high-latency connections such as satellite links. It can use multicast to allow multiple recipients over such a link. If memory serves, it was designed by Gannet to help transfer news content nationally for USA Today. I've tested it on "long fat pipes" when evaluating it for a project a number of years ago. The numbers don't sound that impressive now. I was able to saturate the 100Mbps NIC of the slower server in the transfer. The transfer was from Northern VA to LA over the Internet. This was from data center to data center and the network capacity was such that I knew that the transfer rate would not cause any serious packet loss. Using simple TCP over the same link, same computers was limited to around 40Mbps. More modern TCP implementations have a larger window and are able to better saturate high-throughput, high-latency links.
deleted 10y ago
[deleted]
user5994461 10y ago
> I always wondered why nobody used UDP for bulk data transfer 1) Bulk data transfer doesn't work over UDP. There is no flow control whatsoever with UDP and no guarantee, it sends stuff without distinction between a 56k uplink and a 10Gigabit LAN. The packet loss and lack of reliability are atrocious when you send non negligible amount of data. 2) You'd need to create a protocol on top of UDP to handle the basics (controlflow/error/retransmission), that's equivalent to re-inventing TCP. Don't re-invent your own TCP, just use TCP ;)
- cthalupa 10y ago
  >1) Bulk data transfer doesn't work over UDP. Tell that to all of the people that use TsunamiUDP to do bulk data transfer. It uses TCP as a control protocol, but the data transfer is all UDP.
  - paulddraper 10y ago
    It sounds like they would agree. Bulk data transfer doesn't work over UDP, but it can work over UDP and TCP.
    - cthalupa 10y ago
      The data transfer itself is happening over UDP.
astrobe_ 10y ago
UDP is usable for this when used to transport a file transfer protocol. I've seen for instance Kermit over UDP. Basically, if you transport data using a protocol that already takes care of integrity, using TCP is more or less redundant and just adds annoyances (connection management).
Franciscouzo 10y ago
A lot of the internet's bulk data transfer uses UDP, torrents.
- lobster_johnson 10y ago
  BitTorrent is mostly TCP these days, though, as pointed out elsewhere in this thread.

lend000 10y ago

That sounds like a TCP congestion control mechanism, except worse! TCP leaves much to be desired in terms of performance, largely because their congestion control fails miserably on lossy links (TCP RENO). There are better backoff mechanisms that can be added on many distros, like Westwood. And then there are altogether more efficient protocols like QUIC. But designing one for general purposes from UDP is easier said than done.

orasis 10y ago

Back in the day of hard drives, the random access I/O would be nasty and you'd have to be careful about creating sparse files, so you'd end up double writing the missing packets. you'd also still need congestion control so might as well just use TCP to retransmit.

vvanders 10y ago

You've kinda answered your own question there, congestion control becomes a big issue(also MTU discovery to a lesser degree). Games sit in this happy space where the data rate is ~10-20kb at most so they don't need to worry about congestion control as much.

user5994461 10y ago
Additional information: The source engine (valve) has been limited to 30 kb/s for more than a decade.

acomjean 10y ago

UDP messages are limited to 64K. It can be a pain to reassemble. lots of ugly corner cases. We had need to send one type of message split due to size with udp. We were on the same subnet sending and receiving so it worked almost always. We were also using multicast. Its very very hard to figure out whats going on when things don't work ( You don't know if the control messages are getting through, when you ask again or if the message just isn't being sent). At some point you are reinventing tcp/ip. I always liked the way udp messages are received though, all or nothing. Not the byte stream that is tcp which needs to be broken into messages manually.

lobster_johnson 10y ago
Picking a smallish, fixed chunk size should work, and it could even be made to self-regulate. Reassembly is trivial -- allocate the file first, write chunks to their right location. And for the control stuff (i.e. requesting chunks) you just use a TCP connection. UDP for the data only.
- jerf 10y ago
  If you do the obvious work to convert UDP to a reliable streaming mechanism, you will generally more-or-less reinvent TCP, only probably more poorly, and would probably be better off just using TCP. UDP is a good choice when you have options or requirements that TCP can't accommodate. For instance, the classic case of streaming audio has both; it both requires timely delivery and has some very sophisticated techniques for dealing with missing packets with a lot of local intelligence about human perception and such built into the codecs. If you can drop packets without issue, if future packets completely supercede past packets (FPS network gaming for instance), there's a lot of possibilities, but if you're just going to recreate reliable streaming you probably be net losing on account of the effort spent doing that instead of something useful, and bugs in the implementation.
  - theWatcher37 10y ago
    Why are so many comments here ignoring the purpose of the article? The purpose of this is for games, not web traffic or torrent2.0. In real time games where you're syncing physics states or player locations data >200ms isn't just useless it's actually negative. This article is 2+ years old, and is the foundation I suspect for libyojimbo which hybridizes UDP low latency communication for large player real time games and reliable messaging (basically custom TCP?) for less time critical things.

blackflame7000 10y ago

What it really comes down to is file block size granularity. TCP is simply using a much finer block size (~1500 bytes) which means a higher likelihood of arriving than for a large UDP packet. Furthermore, you'd still have to implement your own checksumming algorithm to ensure data integrity as UDP doesn't require a packet checksum for IPv4. I suppose you could offload file-block integrity validation to another thread so you can receive and verify in parallel, but at some point you have to checksum every block anyways which means the full file must be processed to figure out where corruption might be before sending retry requests for the missing packets. I see what you're getting at by trying to avoid processing the packets in order if it wastes time, but ultimately I think its a micro-op that introduces other bottlenecks. That's why bit-torrent still typically uses TCP even though they could potentially receive file parts in any order.

SomeStupidPoint 10y ago
But bittorrent breaks the files in to chunks it can receive out of order and sends those over TCP, so TCP only preserves ordering within a chunk. It might even get chunks from several different people. The real answer is that chunking over TCP is efficient enough that little is gained by chunking over UDP.
- blackflame7000 10y ago
  Yes what I'm saying is that bit torrent uses TCP to segment fileblocks instead of using UDP to send the file block as one large datagram

ra1n85 10y ago

Bad for everyone else, and it's up to the implementation to handle flow-control. Congestion-control handles network related loss/latency/load. Flow-control helps with receiver related loss/latency/load.

scurvy 10y ago

NFS on Linux defaulted to UDP for what seems like a really long time. There were perf issues with TCP (can't remember what they were).

mullen 10y ago
NFS used to default to UDP for the same reasons that games still would use UDP, it just needed the data as quickly as possible since it was blocking IO. Now a days, we don't need UDP for NFS because networks are a lot faster and the advantages of TCP start to over rule UDP.
- fulafel 10y ago
  The original reason was that using UDP was much faster CPU-wise back in the 80's. There was a requirement to go fast because it was used for transferring data in bulk, not events like games. When Linux came around it just followed suit. TCP support came to NFS only later with other bells and whistles in later NFS versions, and Linux stuck with basic NFSv2 for a good while. NFS was designed for local-area networks, so it didn't need to worry about working over the Internet. In addition to the network being much faster relative to CPUs back then, the TCP stacks hadn't seen the tuning they have now. Making TCP fast came later.
tedunangst 10y ago
NFS (was) stateless and idempotent. Lose a packet? Send it again! Which is mostly what you want, since there can be dozens of processes performing IO and they're all independent. Last thing you want is head of line blocking.

ak217 10y ago

There are many protocols that use this approach, Aspera FASP is possibly the most advanced one. It uses a TCP control channel to manage the rate on the UDP data channel.

justin66 10y ago

> I always wondered why nobody used UDP for bulk data transfer That would be odd. It's not true. There's TFTP all the way up to UDT.

gregschlom 10y ago
Although to be fair, given that with TFTP each packet must be acknowledged before the next one is sent, you're actually achieving a way worse throughput than TCP. Not a very good example on how to use UDP efficiently IMO.
- justin66 10y ago
  UDT is a better example. http://udt.sourceforge.net/ http://udt.sourceforge.net/

kevhito 10y ago

> ship all the chunks over with a sequential number Except that if you ship them too fast, they will at best simply build up in a buffer somewhere (e.g. near a bottleneck link), or more likely they will overflow causing many of your packets to be lost. So not only did the recipient not get them (and you have to figure this out and resend them later), but you also wasted a bunch of resources sending packets only to have them dropped. And if you send them too slow, you are leaving some potential capacity on the table. So you need to send them fast, but not too fast. So you add some additional logic to implement flow control and congestion control. And that's TCP, pretty much.

ajross 10y ago
Exactly. The core feature provided by TCP is not reliability, that is trivially easy. What TCP gives us is robust flow control that can detect and adapt to current network capacity.
- Dylan16807 10y ago
  A robust flow control that's tuned for bulk downloads, and badly adapted for small real-time streams.
  - ajross 10y ago
    Sure. And not even the best known flow control mechanism. But nonetheless that is the important innovation of TCP, not reliability. The post by lobster_johnson upthread implied that you could use UDP for "bulk data transfer", which basically doesn't work without reinventing something akin to TCP.
- snupples 10y ago
  CNN has a video transfer tool for b2b subscribers that uses multiple UDP streams to increase transfer speeds. Instead it saturated and brought down our company firewalls, which were older and our main bottleneck. All for just a potentially few more megs of speed on TB sized videos. Gotta be careful with UDP.

morgante 10y ago

In practice, I think you'd end up having to reimplement most of TCP on top of UDP anyways.

chaosphere2112 10y ago

On LAN, this is basically how Symantec Ghost works; it multicasts the image over UDP, and each client figures out what they missed. Those chunks get retransmitted at the end.

ajosh 10y ago

There is actually such a protocol - UFTP. It was designed for high-latency connections such as satellite links. It can use multicast to allow multiple recipients over such a link. If memory serves, it was designed by Gannet to help transfer news content nationally for USA Today. I've tested it on "long fat pipes" when evaluating it for a project a number of years ago. The numbers don't sound that impressive now. I was able to saturate the 100Mbps NIC of the slower server in the transfer. The transfer was from Northern VA to LA over the Internet. This was from data center to data center and the network capacity was such that I knew that the transfer rate would not cause any serious packet loss. Using simple TCP over the same link, same computers was limited to around 40Mbps. More modern TCP implementations have a larger window and are able to better saturate high-throughput, high-latency links.

deleted 10y ago

[deleted]

user5994461 10y ago

> I always wondered why nobody used UDP for bulk data transfer 1) Bulk data transfer doesn't work over UDP. There is no flow control whatsoever with UDP and no guarantee, it sends stuff without distinction between a 56k uplink and a 10Gigabit LAN. The packet loss and lack of reliability are atrocious when you send non negligible amount of data. 2) You'd need to create a protocol on top of UDP to handle the basics (controlflow/error/retransmission), that's equivalent to re-inventing TCP. Don't re-invent your own TCP, just use TCP ;)

cthalupa 10y ago
>1) Bulk data transfer doesn't work over UDP. Tell that to all of the people that use TsunamiUDP to do bulk data transfer. It uses TCP as a control protocol, but the data transfer is all UDP.
- paulddraper 10y ago
  It sounds like they would agree. Bulk data transfer doesn't work over UDP, but it can work over UDP and TCP.
  - cthalupa 10y ago
    The data transfer itself is happening over UDP.

astrobe_ 10y ago

UDP is usable for this when used to transport a file transfer protocol. I've seen for instance Kermit over UDP. Basically, if you transport data using a protocol that already takes care of integrity, using TCP is more or less redundant and just adds annoyances (connection management).

Franciscouzo 10y ago

A lot of the internet's bulk data transfer uses UDP, torrents.

lobster_johnson 10y ago
BitTorrent is mostly TCP these days, though, as pointed out elsewhere in this thread.