7 ms·

I was in the IETF meeting. It was actually a very civil discussion, and I think almost everyone in the room could see both sides on the issue. On the one hand

I was in the IETF meeting. It was actually a very civil discussion, and I think almost everyone in the room could see both sides on the issue. On the one hand, we've seen increasing problems with middleboxes making assumptions about end-to-end protocols, and this making it very hard to then deploy new end-to-end functionality. The problem is real; we measured this effect in this paper: http://www0.cs.ucl.ac.uk/staff/M.Handley/papers/extend-tcp.pdf http://www0.cs.ucl.ac.uk/staff/M.Handley/papers/extend-tcp.p... Indeed, one of the changes the IETF group has made since taking on QUIC is to also encrypt the QUIC sequence numbers, so middleboxes can't play silly games by observing them (they were already integrity protected). However, the flip side is that network operators do use observations of network round trip time gained from passive observation of traffic so as to discover if traffic is seeing excessive queuing somewhere. An inability to do this with QUIC may either lead to worse network behaviour, or to them using other methods to gain insight. If you're in a location where you can observe both directions of a flow, you can easily do this by, for example, delaying a bunch of packets by 200ms, then letting them go. When you see a burst of (encrypted) ack packets return, you can deduce the RTT from your observation point to the destination. I'd really like to avoid operators thinking they need to do such hacks just to measure RTT, and the spin bit lets a passive observer see the RTT. In the end, I was not convinced entirely by either argument, and neither was the consensus in the room. It's not a clear-cut decision; there are reasonable arguments either way. Such is engineering.

anameaname 9y ago
Realistically, are network operators going to add artificial delay to packets going through their network? If they don't get their bit, and they start slowing down communication, it would seem like market correction would solve it.
- mhandley 9y ago
  I've no idea what techniques will actually be applied by operators - I never expected middleboxes to go to the extremes they currently do either. But the technique I mentioned, of briefly delaying a short burst of packets, if done only occasionally, isn't really going to be noticed during a large transfer. I'm not claiming its a good idea though!
  - anameaname 9y ago
    Large transfer's wouldn't be the place where the effect would be noticeable, it would be JSON payloads that can fit in a few packets. If operators would do this seldomly enough to not be noticed, why do they need the bit at all? It feels like a blank (albeit single digit) check.
- bsder 9y ago
  Sarcastically: who will even notice a 200ms hiccup, nowadays, given the enormous waste of time that everybody shoves into their Javascript ad networks. Personally, I don't want them to expose anything. The RTT on my packets is yet another piece of metadata capable of being abused. In addition, I hope that the whole QUIC thing is a nice reboot of actually getting end-to-end connectivity back on the Internet so we can get some protocol experimentation moving again.
  - slededit 9y ago
    That's 200ms that could be used loading more tracking scripts.
  - supertrope 9y ago
    Sounds like AMP. Improve performance to fit more bloat instead of not adding superfluous elements in the first place. What else would you expect from an advertising company?
- zero_intp 9y ago
  Assume there are a minimum of three network operators involved in every flow; source, destination, and transport. It is highly likely that either the source or destination operator will have opportunity and motive to do said inspection/detection.
- Buge 9y ago
  Aren't network operators already adding artificial delay? I pay for X Mbps download speed, and my ISP artificially limits my download speed to that, even though their network is perfectly capable of providing me with more.
  - tialaramex 9y ago
    No. Bandwidth limits and delay aren't the same thing. CPE (the WiFi router or similar in your home) often adds delay because a few megabytes of RAM is cheap and the people who built it don't understand what they're doing. This is called "buffer bloat". But inside the network core this rarely comes up.
    - ksec 9y ago
      Does anyone know how much buffer bloat or delay is added in the router, or CPE or Modem? I assume they are less then 1ms? I have been trying to figure this out for a while but information aren't really available.
      - tialaramex 9y ago
        Bufferbloat is potentially almost unlimited, assuming that the people who built it are idiots (a safe assumption for most consumer gear) it basically just depends how much they were willing to spend on RAM. For example, let's say we can move 10Mbps, and we've decided to use 10 megabytes of buffers to make our new WiFi router super-duper fast. Do a big download, the buffer fills with ten megabytes of data, that's eight whole seconds of transmission, now the latency of packets is eight seconds, so that's 8000 times larger than your "I assume less then 1ms". {Edited to correct numbers}
        mikeash 9y ago
        Back around 2006, I had a RAZR that I figured out how to use to tether my computer onto Verizon’s 1xRTT service. I’m not sure which part of the system was to blame, but something had massive buffers and a strong aversion to dropping packets. If the connection got saturated I could easily see ping times of two minutes or more.
- floatingatoll 9y ago
  Yes. Market correction isn’t applicable in most US markets, which have either a broadband monopoly or duopoly for broadband service.
IshKebab 9y ago
The thing I don't understand is why these network operators care about RTT outside their network? Shouldn't they only care about getting packets through their network as quickly as possible, which you don't need this spin bit for.
- gmueckl 9y ago
  The global optimum might not be the combination of locally optimal routes. The shortest route through a network segment to a peer may end up being globally slower than a locally slower route to another peer which can handle the traffic faster.
  - catern 9y ago
    That's fine. If global optimization turns out to be significant, we can do it at the edge with overlay networks. No need for network operators to be involved.
    - nine_k 9y ago
      > we can do it at the edge with overlay networks Is this really easy to do, with negligible overhead, and no need to seriously reconfigure what you already have?
Steltek 9y ago
To me, excessive queueing seems indicative of network operators making assumptions about traffic and their solutions ultimately lead to worse outcomes, buffer bloat etc. Isn't QUIC meant to prevent the network from making these assumptions, forcing a more agnostic network, and letting QUIC manage congestion, latency, etc?
- peterwwillis 9y ago
  I think the point is that network operators want to use QUIC to measure network segments and make routing decisions. They are not necessarily trying to improve QUIC performance.
ElijahLynn 9y ago
You may have done this already, but I encourage you to copy/paste this as a comment on the article at apnic.net so it has more visibility. The opening line about "It was actually a very civil discussion, and I think almost everyone in the room could see both sides on the issue." needs to be shared.

anameaname 9y ago

Realistically, are network operators going to add artificial delay to packets going through their network? If they don't get their bit, and they start slowing down communication, it would seem like market correction would solve it.

mhandley 9y ago
I've no idea what techniques will actually be applied by operators - I never expected middleboxes to go to the extremes they currently do either. But the technique I mentioned, of briefly delaying a short burst of packets, if done only occasionally, isn't really going to be noticed during a large transfer. I'm not claiming its a good idea though!
- anameaname 9y ago
  Large transfer's wouldn't be the place where the effect would be noticeable, it would be JSON payloads that can fit in a few packets. If operators would do this seldomly enough to not be noticed, why do they need the bit at all? It feels like a blank (albeit single digit) check.
bsder 9y ago
Sarcastically: who will even notice a 200ms hiccup, nowadays, given the enormous waste of time that everybody shoves into their Javascript ad networks. Personally, I don't want them to expose anything. The RTT on my packets is yet another piece of metadata capable of being abused. In addition, I hope that the whole QUIC thing is a nice reboot of actually getting end-to-end connectivity back on the Internet so we can get some protocol experimentation moving again.
- slededit 9y ago
  That's 200ms that could be used loading more tracking scripts.
- supertrope 9y ago
  Sounds like AMP. Improve performance to fit more bloat instead of not adding superfluous elements in the first place. What else would you expect from an advertising company?
zero_intp 9y ago
Assume there are a minimum of three network operators involved in every flow; source, destination, and transport. It is highly likely that either the source or destination operator will have opportunity and motive to do said inspection/detection.
Buge 9y ago
Aren't network operators already adding artificial delay? I pay for X Mbps download speed, and my ISP artificially limits my download speed to that, even though their network is perfectly capable of providing me with more.
- tialaramex 9y ago
  No. Bandwidth limits and delay aren't the same thing. CPE (the WiFi router or similar in your home) often adds delay because a few megabytes of RAM is cheap and the people who built it don't understand what they're doing. This is called "buffer bloat". But inside the network core this rarely comes up.
  - ksec 9y ago
    Does anyone know how much buffer bloat or delay is added in the router, or CPE or Modem? I assume they are less then 1ms? I have been trying to figure this out for a while but information aren't really available.
    - tialaramex 9y ago
      Bufferbloat is potentially almost unlimited, assuming that the people who built it are idiots (a safe assumption for most consumer gear) it basically just depends how much they were willing to spend on RAM. For example, let's say we can move 10Mbps, and we've decided to use 10 megabytes of buffers to make our new WiFi router super-duper fast. Do a big download, the buffer fills with ten megabytes of data, that's eight whole seconds of transmission, now the latency of packets is eight seconds, so that's 8000 times larger than your "I assume less then 1ms". {Edited to correct numbers}
      - mikeash 9y ago
        Back around 2006, I had a RAZR that I figured out how to use to tether my computer onto Verizon’s 1xRTT service. I’m not sure which part of the system was to blame, but something had massive buffers and a strong aversion to dropping packets. If the connection got saturated I could easily see ping times of two minutes or more.
floatingatoll 9y ago
Yes. Market correction isn’t applicable in most US markets, which have either a broadband monopoly or duopoly for broadband service.

IshKebab 9y ago

The thing I don't understand is why these network operators care about RTT outside their network? Shouldn't they only care about getting packets through their network as quickly as possible, which you don't need this spin bit for.

gmueckl 9y ago
The global optimum might not be the combination of locally optimal routes. The shortest route through a network segment to a peer may end up being globally slower than a locally slower route to another peer which can handle the traffic faster.
- catern 9y ago
  That's fine. If global optimization turns out to be significant, we can do it at the edge with overlay networks. No need for network operators to be involved.
  - nine_k 9y ago
    > we can do it at the edge with overlay networks Is this really easy to do, with negligible overhead, and no need to seriously reconfigure what you already have?

Steltek 9y ago

To me, excessive queueing seems indicative of network operators making assumptions about traffic and their solutions ultimately lead to worse outcomes, buffer bloat etc. Isn't QUIC meant to prevent the network from making these assumptions, forcing a more agnostic network, and letting QUIC manage congestion, latency, etc?

peterwwillis 9y ago
I think the point is that network operators want to use QUIC to measure network segments and make routing decisions. They are not necessarily trying to improve QUIC performance.

ElijahLynn 9y ago

You may have done this already, but I encourage you to copy/paste this as a comment on the article at apnic.net so it has more visibility. The opening line about "It was actually a very civil discussion, and I think almost everyone in the room could see both sides on the issue." needs to be shared.