6 ms·

Optimizing UDP transmission over internet is an interesting topic. I remember reading a paper a while ago that showed that if you send two consecutive UDP pack

Optimizing UDP transmission over internet is an interesting topic. I remember reading a paper a while ago that showed that if you send two consecutive UDP packets with exact same data over the internet, at least 1 of them will arrive to the destination at pretty high success rate (something like 99.99%) I wonder if this still works with current internet infrastructure, and if this trick is still used in real-time streaming protocols.

wmf 7y ago
So basically rate 1/2 FEC.
- deleted 7y ago
  [deleted]
zamadatix 7y ago
99.99% for two tries would be a 1% drop chance which I'd say is pretty lenient - we average better than that on our sites running off 4G (jitter is horrible though and that will kill any real-time protocols without huge delays added). Generally you'd just implement a more generic FEC algorithm though unless you had 2 separate paths you wanted to try (e.g. race a cable modem and 4G with every packet and if one side drops it hope the other side still finishes the race) as there are FEC options that allow non integer redundancy levels and can reduce header overhead compared to sending multiple copies of small packets.
- syrrim 7y ago
  >99.99% for two tries would be a 1% drop chance Not per se. The drop chance for consecutive packets is likely correlated, such that if you know the first one was dropped you should increase your prior that the second one will also be dropped.
  - zamadatix 7y ago
    Depends on the cause and root question. For instance in the most common scenario of congestion routers do intelligent random drops with increasing probability as the buffer gets more full https://en.wikipedia.org/wiki/Random_early_detection https://en.wikipedia.org/wiki/Random_early_detection. The internet actually relies on this random low drop chance to make things work smoothly rather than waiting til things are failing apart to signal to streams to slow down all at once while it catches up. Same randomness with transmission bit errors which will cause drops but the randomness is not by design as much as by the way noise is what is causing those. On the other hand if the root question is if there is an outage style issue then yeah if the path to the destination is having a hard down style issue no number of packets are going to help because they are all going to drop. Likewise if the question is "on a short enough time scale is reliability of delivering a single packet somewhere on the internet ever less than 99%" then yeah somewhere there is a failure scenario and if you look at a short enough time scale any failure scenario can be made to say there is 0% reliability.
mcguire 7y ago
Odds are, at least one of the links between the source and destination will be shared. If so, sending two packets is an expensive attempt at reliability; it will cut the bandwidth in half. Further, one data packet will arrive with a highish success rate.
- snisarenko 7y ago
  Depends on the use-case. If you are streaming a live sports event you don't want the video frames to fall behind from the real clock too far, but you also don't want the video to stutter when there is packet loss. I believe you can use the UDP trick I mentioned above to amortize loss of high quality frame data, with "double sent" low quality frame data.
  - mcguire 7y ago
    As others have mentioned, you're looking for forward error correction, and there are better ways than sending duplicate packets, although they all do add redundancy in various ways.
ignoramous 7y ago
u/noselasd: > Also keep in mind this note: http://technet.microsoft.com/en-us/library/cc940021.aspx http://technet.microsoft.com/en-us/library/cc940021.aspx > Basically, if you send() 2 or more UDP datagrams in quick succession, and the OS has to resolve the destination with ARP, all but the 1 packet is dropped until you get an ARP reply (this behavior isn't entirely unique to windows, btw). https://news.ycombinator.com/item?id=8468313 https://news.ycombinator.com/item?id=8468313
- zamadatix 7y ago
  Note this only applies to things on the same subnet as you. Once you're off your local subnet routing takes over so you only need to know the ARP of the local gateway (which the OS handles as part of bringing network up).
tomohawk 7y ago
It depends on what the characteristics of the transmission line are. If it is purely random, that is one thing, but often if one packet is dropped or smashed, there is a higher probability of following ones to meet the same fate. For example, if the transmission is over a microwave link, it is easy to see how something could cause a few thousand packets in a row to go missing.
- snisarenko 7y ago
  > if one packet is dropped or smashed, there is a higher probability of following ones to meet the same fate. Not necessarily. I am not an expert on this topic, but I believe physical device failure is one of the least common reasons a packet is dropped (at least on physical cable transmission). I think link saturation, priority routing (for certain types of packets, or contractual reasons) might be more common. There are thousands of routers out there with a variety of different configurations that decide which packets go through under which conditions.
amelius 7y ago
> Optimizing UDP transmission over internet is an interesting topic. Can someone summarize the problem in abstract terms? What are the parameters that the algorithm can play with? Packet size and rate of transmission, anything else?
- snisarenko 7y ago
  Those two are pretty much it. I believe the ordering of the data transmission can matter as well (i.e. you are about to send 3 packets of 3 different sizes. the ordering might change deliverability behavior.) I guess I meant it was an interesting topic to me when I read the paper :) It might not be that exciting overall.
readmodifywrite 7y ago
Do you have a link to the paper? I'd love to read it.

wmf 7y ago

So basically rate 1/2 FEC.

deleted 7y ago
[deleted]

zamadatix 7y ago

99.99% for two tries would be a 1% drop chance which I'd say is pretty lenient - we average better than that on our sites running off 4G (jitter is horrible though and that will kill any real-time protocols without huge delays added). Generally you'd just implement a more generic FEC algorithm though unless you had 2 separate paths you wanted to try (e.g. race a cable modem and 4G with every packet and if one side drops it hope the other side still finishes the race) as there are FEC options that allow non integer redundancy levels and can reduce header overhead compared to sending multiple copies of small packets.

syrrim 7y ago
>99.99% for two tries would be a 1% drop chance Not per se. The drop chance for consecutive packets is likely correlated, such that if you know the first one was dropped you should increase your prior that the second one will also be dropped.
- zamadatix 7y ago
  Depends on the cause and root question. For instance in the most common scenario of congestion routers do intelligent random drops with increasing probability as the buffer gets more full https://en.wikipedia.org/wiki/Random_early_detection https://en.wikipedia.org/wiki/Random_early_detection. The internet actually relies on this random low drop chance to make things work smoothly rather than waiting til things are failing apart to signal to streams to slow down all at once while it catches up. Same randomness with transmission bit errors which will cause drops but the randomness is not by design as much as by the way noise is what is causing those. On the other hand if the root question is if there is an outage style issue then yeah if the path to the destination is having a hard down style issue no number of packets are going to help because they are all going to drop. Likewise if the question is "on a short enough time scale is reliability of delivering a single packet somewhere on the internet ever less than 99%" then yeah somewhere there is a failure scenario and if you look at a short enough time scale any failure scenario can be made to say there is 0% reliability.

mcguire 7y ago

Odds are, at least one of the links between the source and destination will be shared. If so, sending two packets is an expensive attempt at reliability; it will cut the bandwidth in half. Further, one data packet will arrive with a highish success rate.

snisarenko 7y ago
Depends on the use-case. If you are streaming a live sports event you don't want the video frames to fall behind from the real clock too far, but you also don't want the video to stutter when there is packet loss. I believe you can use the UDP trick I mentioned above to amortize loss of high quality frame data, with "double sent" low quality frame data.
- mcguire 7y ago
  As others have mentioned, you're looking for forward error correction, and there are better ways than sending duplicate packets, although they all do add redundancy in various ways.

ignoramous 7y ago

u/noselasd: > Also keep in mind this note: http://technet.microsoft.com/en-us/library/cc940021.aspx http://technet.microsoft.com/en-us/library/cc940021.aspx > Basically, if you send() 2 or more UDP datagrams in quick succession, and the OS has to resolve the destination with ARP, all but the 1 packet is dropped until you get an ARP reply (this behavior isn't entirely unique to windows, btw). https://news.ycombinator.com/item?id=8468313 https://news.ycombinator.com/item?id=8468313

zamadatix 7y ago
Note this only applies to things on the same subnet as you. Once you're off your local subnet routing takes over so you only need to know the ARP of the local gateway (which the OS handles as part of bringing network up).

tomohawk 7y ago

It depends on what the characteristics of the transmission line are. If it is purely random, that is one thing, but often if one packet is dropped or smashed, there is a higher probability of following ones to meet the same fate. For example, if the transmission is over a microwave link, it is easy to see how something could cause a few thousand packets in a row to go missing.

snisarenko 7y ago
> if one packet is dropped or smashed, there is a higher probability of following ones to meet the same fate. Not necessarily. I am not an expert on this topic, but I believe physical device failure is one of the least common reasons a packet is dropped (at least on physical cable transmission). I think link saturation, priority routing (for certain types of packets, or contractual reasons) might be more common. There are thousands of routers out there with a variety of different configurations that decide which packets go through under which conditions.

amelius 7y ago

> Optimizing UDP transmission over internet is an interesting topic. Can someone summarize the problem in abstract terms? What are the parameters that the algorithm can play with? Packet size and rate of transmission, anything else?

snisarenko 7y ago
Those two are pretty much it. I believe the ordering of the data transmission can matter as well (i.e. you are about to send 3 packets of 3 different sizes. the ordering might change deliverability behavior.) I guess I meant it was an interesting topic to me when I read the paper :) It might not be that exciting overall.

readmodifywrite 7y ago

Do you have a link to the paper? I'd love to read it.