4 ms·

« Note the download isn’t going to hover exactly at 1.0K/sec — > the actual download speed as reported by wget is an average over time. In short, you’ll see num

« Note the download isn’t going to hover exactly at 1.0K/sec — > the actual download speed as reported by wget is an average over time. In short, you’ll see numbers closer to an even 1.0K/sec the longer the transfer. In this example, I didn’t wait to download an entire 4.2GB file, so the 10.5K/s you see above is just wget averaging the transfer speed over the short time I left wget running. » Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets! So, what you are really doing is rate limiting outgoing ACK packets to 1KB/s, which delays your outgoing ACKs, hence preventing the remote server from sending you data at full throttle. You can see that with a tool like "bwm-ng" to see Rx/Tx speed, and you'll see exactly 1KB/s on Tx, and something variable on Rx. The incoming rate will fluctuates depending on the TCP window setting used by the server which translate to how many non-acknowledged packets are allowed to be sent by the server.

gmazza 10y ago
> Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets! Yep. TC's default is to policy outgoing traffic, which in OP's example is a bunch of TCP ACKs essentially. Instead, they should be using ingress keyword, something like described here: http://blog.stevedoria.net/20050906/ingress-policing-with-linux-and-tc http://blog.stevedoria.net/20050906/ingress-policing-with-li... Caveat emptor: ingress rate-limiting is hard. Long story short, it all boils down to what you do with non-confirming packets: There are two alternatives, and both are rather sub-optimal. You can either buffer/delay packets in kernel space (default, which leads to bufferbload and memory waste), or drop (which author linked above opted for, which leads to excessive retransmits and bandwidth waste).
- wtallis 10y ago
  The drop vs buffer decision is no harder for outgoing or incoming packets. In either case: if you're trying to simulate a different kind of network, do what that network does. If you're just trying to get good QoS on your gateway router, then use a smart AQM that will buffer only to the extent that is reasonable, and then drop or ECN mark when buffering threatens to add too much latency.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
- mseebach 10y ago
  What actually happens when Internet traffic makes the jump from my ISPs big-pipe backbone connection to my much slower last-mile connection? They clearly can't cache the world, and "excessive retransmits and bandwidth waste" doesn't seem to be the case?
  - Sunset 10y ago
    They drop your packets, both for ingoing and outgoing overflow.
    - mseebach 10y ago
      But how is that different from the proposed method of rate limited which causes "excessive retransmits and bandwidth waste"? Not being pedantic, genuinely trying to understand how this stuff works.
      - gmazza 10y ago
        Middle ground between dropping [1] and buffering is Active Queue Management as wtallis pointed out above. State of the art AQM on Linux nowadays is CoDel [2]. Big network gear vendors that ship to ISPs have adopted early forms of AQM (both RED [3] and proprietary algorithms) quite a while ago - they had to: (a) backbone routers have much smaller buffer-to-bandwidth ratio (compared to a PC or even home router), so endless buffering is not an option; (b) buffer tail drops (i.e. what drop keyword does when added to tc filters/disciplines) interact really poorly with TCP bandwidth control algorithms, and rather badly with RTP streams, too (so drop is not an option either - it ruins user's connections; and (c) ISPs and carriers would typically run links (on average) much closer to saturation than your typical home router, so situation where router has to make buffer/drop decision is much, much more frequent. [1] My point above was that you don't really want to drop packets on the receiver side, after that packet already traversed expensive part of the network. [2] https://en.wikipedia.org/wiki/CoDel https://en.wikipedia.org/wiki/CoDel [3] https://en.wikipedia.org/wiki/Random_early_detection https://en.wikipedia.org/wiki/Random_early_detection
  - wtallis 10y ago

gmazza 10y ago

> Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets! Yep. TC's default is to policy outgoing traffic, which in OP's example is a bunch of TCP ACKs essentially. Instead, they should be using ingress keyword, something like described here: http://blog.stevedoria.net/20050906/ingress-policing-with-linux-and-tc http://blog.stevedoria.net/20050906/ingress-policing-with-li... Caveat emptor: ingress rate-limiting is hard. Long story short, it all boils down to what you do with non-confirming packets: There are two alternatives, and both are rather sub-optimal. You can either buffer/delay packets in kernel space (default, which leads to bufferbload and memory waste), or drop (which author linked above opted for, which leads to excessive retransmits and bandwidth waste).

wtallis 10y ago
The drop vs buffer decision is no harder for outgoing or incoming packets. In either case: if you're trying to simulate a different kind of network, do what that network does. If you're just trying to get good QoS on your gateway router, then use a smart AQM that will buffer only to the extent that is reasonable, and then drop or ECN mark when buffering threatens to add too much latency.
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
mseebach 10y ago
What actually happens when Internet traffic makes the jump from my ISPs big-pipe backbone connection to my much slower last-mile connection? They clearly can't cache the world, and "excessive retransmits and bandwidth waste" doesn't seem to be the case?
- Sunset 10y ago
  They drop your packets, both for ingoing and outgoing overflow.
  - mseebach 10y ago
    But how is that different from the proposed method of rate limited which causes "excessive retransmits and bandwidth waste"? Not being pedantic, genuinely trying to understand how this stuff works.
    - gmazza 10y ago
      Middle ground between dropping [1] and buffering is Active Queue Management as wtallis pointed out above. State of the art AQM on Linux nowadays is CoDel [2]. Big network gear vendors that ship to ISPs have adopted early forms of AQM (both RED [3] and proprietary algorithms) quite a while ago - they had to: (a) backbone routers have much smaller buffer-to-bandwidth ratio (compared to a PC or even home router), so endless buffering is not an option; (b) buffer tail drops (i.e. what drop keyword does when added to tc filters/disciplines) interact really poorly with TCP bandwidth control algorithms, and rather badly with RTP streams, too (so drop is not an option either - it ruins user's connections; and (c) ISPs and carriers would typically run links (on average) much closer to saturation than your typical home router, so situation where router has to make buffer/drop decision is much, much more frequent. [1] My point above was that you don't really want to drop packets on the receiver side, after that packet already traversed expensive part of the network. [2] https://en.wikipedia.org/wiki/CoDel https://en.wikipedia.org/wiki/CoDel [3] https://en.wikipedia.org/wiki/Random_early_detection https://en.wikipedia.org/wiki/Random_early_detection
- wtallis 10y ago