5 ms·

A search for "ledbat" in github doesn't find any reference, which is a shame. Apple are the only one with ledbat support in the kernel[0] afaik, and it's functi

A search for "ledbat" in github doesn't find any reference, which is a shame. Apple are the only one with ledbat support in the kernel[0] afaik, and it's functionality that's sorely needed -- it's ridiculous that there's no way to say "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections". Some programs, e.g. rsync, let you limit bandwidth - but that's wasteful; I want this 2TB rsync job to be downloading at full speed at all time I'm not doing anything else. [0] yes, this is userland, but it's for fuchsia, so "kernel functionality" as far as its users are concerned.

unmole 9y ago
How would LEDBAT help for downloads unless the server supports it? Even then, how does the client indicate to the server that "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections"?
- deleted 9y ago
  [deleted]
- simcop2387 9y ago
  If it's TCP, adjust the congestion control to stop sending acks and slow the connection when others have priority. Can't do it for anything like udp though.
  - unmole 9y ago
    > adjust the congestion control to stop sending acks Huh? Congestion control is enforced on the sender side while ACKs are generated by the receiver. Apple has a patent on throttling lower priority TCP flows by advertising a reduced receive window based on inter-arrival jitter. I feel this is a cleaner way than delaying ACKs. Anyways, my point was that LEDBAT wouldn't help.
diamondlovesyou 9y ago
It's not implementable by local devices: your LAN is much faster than your WAN link, in most cases. Your router/modem does the packet dropping. But only your router/modem knows how much unused throughput is available at any moment, so rsync can't know how much it can use. Sure, rsync can look at TCP packet retransmits, but that will still cause other streams to experience an (assuming a lot of things) "equal" amount of drops too. So, you get a lot of complexity for not a lot of benefit. However, Fuchsia is destined for mobile devices, which may not have the LAN-WAN bottleneck, so this could see some benefit.
- beagle3 9y ago
  The way ledbat works is by measuring the derivative of round trip time with respect to a throttled receive bandwidth. As long as RTT is constant or decreasing, keep increasing bandwidth; if RTT is increasing decrease bandwidth. The server has to give you an indication of round trip for that to work - although I suspect there’s some clever way to get it using Some kind of URG or OOB or keep alive even if the server doesn’t natively support TCP ledbat.
  - toast0 9y ago
    If you've negotiated tcp timestamps, and you assume the server will send more data immediately when it gets an ack, you have a convenient round trip time signal in the echoed timestamps. If you don't have tcp timestamps, you can advertise a reduced window and time the round trip between acks that open up the window and packets with sequence numbers in the window.
vitus 9y ago
From my brief look into ledbat, it seems like it serves a similar role to BBR, which is in the Linux kernel now. Either way, it looks like the congestion control is just New Reno. https://github.com/google/netstack/blob/edcfca33ca347f59a1393986a5d216a5f183ae28/tcpip/transport/tcp/snd.go#L344 https://github.com/google/netstack/blob/edcfca33ca347f59a139... Also, this isn't an official Google product, per the disclaimer; just some code that happens to be owned by Google. Where are you seeing the reference to Fuchsia? edit: oh, I see. Fuchsia codebase has references to this netstack.
- slackingoff2017 9y ago
  LEDBAT is similar but works on L7 instead of L4 and needs support on both ends. BBR is excellent. It's the first congestion control strategy to really focus on fairness rather than trying to send packets as fast as possible. Combined with fq-CODEL, BBR finally offers a way to load balance traffic on your NAT without allowing someone to monopolize bandwidth
  - kevin2r 9y ago
    Wow this is awesome, It bothers me that one computer on a LAN making a big download, takes over the other devices, causing high latency and connections disruption. I mean we are talking about connectivity, it should not happen.
    - slackingoff2017 9y ago
      Yup, it's because for the longest time TCP was as aggressive as possible to get the most bandwidth. Over the last 10 years networks buffers have become gigantic and sending packets as fast as you can causes standing queues that can be a dozen seconds long. This is where the lag comes from when someone on your LAN is uploading/downloading. The lag actually gets worse with higher connection speed because the devices tend to have even bigger buffers. BBL and to some extent VEGAS are smarter going about things. They attempt to maintain constant latency instead of maximum bandwidth. For a few percent less connection speed all the buffers stay empty and delay stays minimal
slackingoff2017 9y ago
LEDBAT typically runs over UDP and doesn't need kernel support. Windows has something similar with "background intelligent transfer service" BITS, but LEDBAT works just fine outside kernel.
- beagle3 9y ago
  But you can apply it to TCP as well (only at a kernel/raw socket level), which Apple and Microsoft have already done, which makes it simple for software to use; Apple uses it to download o/s updates in the background. Implementing it in the kernel also makes programs like “trickle” much easier.
  - slackingoff2017 9y ago
    Are you sure they didn't use UDP? It seems like the kernel flow control in TCP would interact badly with LEDBAT
Fuzzy_Logic 9y ago
Windows now implements LEDBAT as well: https://blogs.technet.microsoft.com/networking/2016/07/18/announcing-new-transport-advancements-in-the-anniversary-update-for-windows-10-and-windows-server-2016/ https://blogs.technet.microsoft.com/networking/2016/07/18/an...
- beagle3 9y ago
  Good to know. Though it is still through an undocumented option and considered experimental.
  - Fuzzy_Logic 9y ago
    My understanding from a talk at Ignite 2017 is that certain OS functionality (like some Windows Updates) use it, and that it would get more official support later...but nothing I can find yet, so as you say for the developer it's still an undocumented 'experimental' feature. Hopefully this has changed in the latest build, I will have to look more. https://view.officeapps.live.com/op/embed.aspx?src=https%3A%2F%2F8gportalvhdsf9v440s15hrt.blob.core.windows.net%2Fignite2017%2Fsession-presentations%2FBRK3056.PPTX https://view.officeapps.live.com/op/embed.aspx?src=https%3A%... slide 31

unmole 9y ago

How would LEDBAT help for downloads unless the server supports it? Even then, how does the client indicate to the server that "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections"?

deleted 9y ago
[deleted]
simcop2387 9y ago
If it's TCP, adjust the congestion control to stop sending acks and slow the connection when others have priority. Can't do it for anything like udp though.
- unmole 9y ago
  > adjust the congestion control to stop sending acks Huh? Congestion control is enforced on the sender side while ACKs are generated by the receiver. Apple has a patent on throttling lower priority TCP flows by advertising a reduced receive window based on inter-arrival jitter. I feel this is a cleaner way than delaying ACKs. Anyways, my point was that LEDBAT wouldn't help.

diamondlovesyou 9y ago

It's not implementable by local devices: your LAN is much faster than your WAN link, in most cases. Your router/modem does the packet dropping. But only your router/modem knows how much unused throughput is available at any moment, so rsync can't know how much it can use. Sure, rsync can look at TCP packet retransmits, but that will still cause other streams to experience an (assuming a lot of things) "equal" amount of drops too. So, you get a lot of complexity for not a lot of benefit. However, Fuchsia is destined for mobile devices, which may not have the LAN-WAN bottleneck, so this could see some benefit.

beagle3 9y ago
The way ledbat works is by measuring the derivative of round trip time with respect to a throttled receive bandwidth. As long as RTT is constant or decreasing, keep increasing bandwidth; if RTT is increasing decrease bandwidth. The server has to give you an indication of round trip for that to work - although I suspect there’s some clever way to get it using Some kind of URG or OOB or keep alive even if the server doesn’t natively support TCP ledbat.
- toast0 9y ago
  If you've negotiated tcp timestamps, and you assume the server will send more data immediately when it gets an ack, you have a convenient round trip time signal in the echoed timestamps. If you don't have tcp timestamps, you can advertise a reduced window and time the round trip between acks that open up the window and packets with sequence numbers in the window.

vitus 9y ago

From my brief look into ledbat, it seems like it serves a similar role to BBR, which is in the Linux kernel now. Either way, it looks like the congestion control is just New Reno. https://github.com/google/netstack/blob/edcfca33ca347f59a1393986a5d216a5f183ae28/tcpip/transport/tcp/snd.go#L344 https://github.com/google/netstack/blob/edcfca33ca347f59a139... Also, this isn't an official Google product, per the disclaimer; just some code that happens to be owned by Google. Where are you seeing the reference to Fuchsia? edit: oh, I see. Fuchsia codebase has references to this netstack.

slackingoff2017 9y ago
LEDBAT is similar but works on L7 instead of L4 and needs support on both ends. BBR is excellent. It's the first congestion control strategy to really focus on fairness rather than trying to send packets as fast as possible. Combined with fq-CODEL, BBR finally offers a way to load balance traffic on your NAT without allowing someone to monopolize bandwidth
- kevin2r 9y ago
  Wow this is awesome, It bothers me that one computer on a LAN making a big download, takes over the other devices, causing high latency and connections disruption. I mean we are talking about connectivity, it should not happen.
  - slackingoff2017 9y ago
    Yup, it's because for the longest time TCP was as aggressive as possible to get the most bandwidth. Over the last 10 years networks buffers have become gigantic and sending packets as fast as you can causes standing queues that can be a dozen seconds long. This is where the lag comes from when someone on your LAN is uploading/downloading. The lag actually gets worse with higher connection speed because the devices tend to have even bigger buffers. BBL and to some extent VEGAS are smarter going about things. They attempt to maintain constant latency instead of maximum bandwidth. For a few percent less connection speed all the buffers stay empty and delay stays minimal

slackingoff2017 9y ago

LEDBAT typically runs over UDP and doesn't need kernel support. Windows has something similar with "background intelligent transfer service" BITS, but LEDBAT works just fine outside kernel.

beagle3 9y ago
But you can apply it to TCP as well (only at a kernel/raw socket level), which Apple and Microsoft have already done, which makes it simple for software to use; Apple uses it to download o/s updates in the background. Implementing it in the kernel also makes programs like “trickle” much easier.
- slackingoff2017 9y ago
  Are you sure they didn't use UDP? It seems like the kernel flow control in TCP would interact badly with LEDBAT

Fuzzy_Logic 9y ago

Windows now implements LEDBAT as well: https://blogs.technet.microsoft.com/networking/2016/07/18/announcing-new-transport-advancements-in-the-anniversary-update-for-windows-10-and-windows-server-2016/ https://blogs.technet.microsoft.com/networking/2016/07/18/an...

beagle3 9y ago
Good to know. Though it is still through an undocumented option and considered experimental.
- Fuzzy_Logic 9y ago
  My understanding from a talk at Ignite 2017 is that certain OS functionality (like some Windows Updates) use it, and that it would get more official support later...but nothing I can find yet, so as you say for the developer it's still an undocumented 'experimental' feature. Hopefully this has changed in the latest build, I will have to look more. https://view.officeapps.live.com/op/embed.aspx?src=https%3A%2F%2F8gportalvhdsf9v440s15hrt.blob.core.windows.net%2Fignite2017%2Fsession-presentations%2FBRK3056.PPTX https://view.officeapps.live.com/op/embed.aspx?src=https%3A%... slide 31