6 ms·

That's one of the issues HTTP/2 has. Because all the requests and responses are now multiplexed over a single TCP connection, losing a single packet is going to

by zimbatm 10y ago

That's one of the issues HTTP/2 has. Because all the requests and responses are now multiplexed over a single TCP connection, losing a single packet is going to block everything. This makes HTTP/2 actually work on unreliable networks. This problem is also known as Head-Of-Line (HOL) blocking. Google is aware of that issue, that's why they explored SCTP first and now QUIC as alternate layer 3 protocols to replace TCP in the future.

riobard 10y ago
SCTP is problematic because middle boxes need to be made aware of it since it uses a different IP protocol. This is gonna require massive infrastructure changes and probably won't fly (like IPv6). QUIC, which runs over UDP, is easier to adopt. However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable. TCP is still the safest bet. We need to figure out a better (and magical!) way somehow...
- AndrewDucker 10y ago
  IPv6 is over 10% of Google's traffic, and is doubling year-on-year. If SCTP was failing to fly that badly it would make me very happy!
  - daenney 10y ago
    But its taken us 20 years to get to this point and even with the IPv4 exhaustion it's still relatively slow to be picked up on by providers in Europe or North America (except for mobile, now). Not to say I wouldn't want SCTP to succeed but IPv6 doesn't exactly spring to mind as a model for the successful adoption of a new internet protocol.
- panic 10y ago
  SCTP is problematic because middle boxes need to be made aware of it since it uses a different IP protocol. Is this true? As far as I can tell SCTP works over plain IP, just like UDP and TCP do.
  - riobard 10y ago
    SCTP uses IP protocol number 132. TCP is 6 and UDP is 17. Many middle boxes, especially cheap home routers, are not aware of IP protocols other than TCP/UDP (and maybe GRE/ESP if you're lukcy). Their NAT will not work with SCTP.
    - garaetjjte 10y ago
      But there's IPv6 coming. (so NATs will be gone)
      - alyandon 10y ago
        That really assumes residential ISPs out there decide to fully embrace the idea behind IPv6 instead versus doing something asinine like handing out a single /128 address. Fortunately, mine hands me a /64 so I'm not in that situation. :)
- staunch 10y ago
  > However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable. What makes you think this? I play a UDP-based FPS game where "ping" times are critically important and it seems like latencies across the board have continued to drop over the years. Latencies are ~2x the speed of light in most case. Not too shabby.
  - riobard 10y ago
    Your connection looks pretty good. My home ISP throttles UDP traffic to about 5Mbps while TCP reaches 80+Mbps. Packet loss during peak hours can be as high as 60% for UDP but a manageable single digit percent for TCP. My workplace has anti-BitTorrent mechanism deployed and any meaningful amount of UDP outbound traffic is dropped.
- DanielDent 10y ago
  [quickly becoming standard comment for me]: IPv6 is in widespread use. IPv6 is not "the future". It's the present. I have services in production where >30% of users access the service over v6. Use is growing. IPv6-only mobile operators are also now a thing. The idea that middle boxes need to be aware of protocols is incredibly destructive to the end-to-end principal - the critical property of IP networks that enables a rapid pace of permissionless innovation. Middleboxes are evil incarnate. Fortunately, academic research as well as operational experience has shown that TCP and UDP provide the primitives required to build any protocol. QUIC, which you mentioned, is an example of this. Google has made the explicit - and probably wise - decision to keep protocol state encrypted. While there would be some nice optimizations possible if routers could be aware of things like 'this session is closed now', the risk of middleboxes making it so that the protocol can't continue to evolve is too great. The solution is to specifically prevent middleboxes from having the knowledge needed meddle in an 'intelligent' way. The last statistics I saw from Google were that 95% of networks don't cause problems for QUIC (i.e. QUIC connections are as fast or faster than HTTP/2). The main issue I'm aware of is not so actually de-prioritization but rather rate limiting (it's often implemented as a quick - and very very dirty - DDoS mitigation technique). Google's stated solution is to fallback to HTTP/2 based on measurements of user experience on a per-AS basis. Networks which try to be "intelligent" rather than just provide adequate bandwidth will tend towards providing a crappy user experience. Solutions which optimize for well behaved adequately provisioned networks and have an acceptable-though-degraded fallback seems like a reasonable way to promote progress. SCTP has an RFC-defined UDP encapsulation which will deal with many evil networks.
  - eeeeeeeeeeeee 10y ago
    Until you can run an IPv6 service without dual-stack, it's not in the present in the same way you're talking about. We are still very much in the transition period. Yes, it's positive to see IPv6 growth, but the fact remains that any new service today must provide both IPv4 and IPv6 end-points and that is likely to continue for years.
  - nucleardog 10y ago
    > IPv6 is in widespread use. IPv6 is not "the future". It's the present. I have services in production where >30% of users access the service over v6. Use is growing. IPv6-only mobile operators are also now a thing. That's... great, and I'm glad there's progress being made somewhere, but I don't think it's fair to say it's "the present". It's on its way. My ISP has rolled out FttH across a wide area but still only offers IPv6 on business buildouts. Even if they offered IPv6 to general customers, they still have a ton of home gateway devices out there that don't support any IPv6 at all. The other ISPs don't offer it or have any external roadmap either. I don't think this situation is all that unusual or unique - it's just not something that the people paying the bills are clamouring for and it's a long road from where we are to significant penetration. Oh, and EC2 still doesn't support IPv6 on anything but their load balancers. What portion of the market and traffic do you think they make up?
panic 10y ago
I wonder why SCTP isn't more widely-supported. It seems like a lot of Internet traffic is message-based rather than stream-based and would benefit from a more appropriate protocol.
- abstractbeliefs 10y ago
  I've actually been looking into (and started using) SCTP. It's pretty good, except some places seem to have it turned off. In particular, Windows has little support.
- mikeash 10y ago
  Seems to me that the basic Internet protocols got more or less frozen in time in the 90s, not because they had reached perfection, but just because it became too difficult to support new ones. With ubiquitous firewalls and NAT devices, anything that didn't fit into the existing TCP/UDP system would fail to work with so much of the Internet that people just wouldn't bother with it. It really is unfortunate that we ended up with UDP and TCP as the only choices. There are so many more things you might want outside of "unreliable packets" and "automatically recovered streams."
  - nocarrier 10y ago
    The list of firewall-punchable level 4 protocols is frozen in time to just TCP and UDP, and it's sad at first glance, but it's not as bad as we think: * Most ISPs don't mangle UDP, and if you care enough about perf to be using UDP in the first place, you can afford to figure out who mangles UDP and fall back to TCP for those networks. It's a very small number of networks. * Any TCP/SCTP/etc inspired protocol you can think of can be implemented on top of UDP. Its header is only eight bytes long and it adds minimal overhead. UDP socket semantics are simple enough that you can treat them as a slightly smarter IP socket and build your protocol as if you were running directly on top of IPv6. * There's a huge advantage to being able to implement your protocol in userland since you get fine grained control over congestion and connection logic. There's a huge disadvantage too, since it's actually hard to implement that stuff yourself. But for the people who are large enough to build their own protocols (FB, Google, etc), it's worth it. * You can even encapsulate SCTP inside UDP to punch through middleware firewalls if you want to. I don't know of anyone doing that for real in the wild, but there's a RFC for it (RFC 6951).
hitr 10y ago
My understanding is that http/2 completely fixed HOL blocking .If any streams has any issues ,server or client can reset that stream independently [1]. Isn't that the case ? [1] https://http2.github.io/http2-spec/#StreamErrorHandler https://http2.github.io/http2-spec/#StreamErrorHandler
- rgbrenner 10y ago
  There's http1's head of line blocking.. and tcp's head of line blocking. http2 fixed the former.. the latter applies to any tcp connection--including http 1.x, http2, etc (anything that uses tcp). In other words.. he's saying, because HTTP 2 is multiplexed, tcp's head of line blocking becomes a larger problem. Which is true.
rgbrenner 10y ago
Couldn't agree more. When I was implementing adaptive routing on NuevoCloud that was one of the problems I worked on. Ended up de-multiplexing http2, once the requests hit the edge node. So there's still one connection between the edge and the browser (can't do anything about that), but once it gets to the edge node, each request is sent over a prenegotiated connection to an edge node near the server.. then it drops down to a more reasonable number when sending the requests to the customer's server. So if a packet is dropped, it only affects that one file/connection. Don't think any other CDNs are doing this yet.. but I think it'll become more common as they invest more resources into their http2 implementations.
- nocarrier 10y ago
  This works for your use case which is great, but in my experience, you get gains with full http2 between edge and origin that you miss out on if you demux or fall back to http 1.1. We saw slightly better median latencies and much better >90th percentile latencies with full http2 from edge to origin, versus when we did http2 to edge users and http 1.1 to origin. At Facebook, we had clients talk http2 to edge nodes near the users, which then edge terminated TLS and opened up the request to figure out which upstream was best to forward the request to. We kept http2 conn pools to each upstream from a given edge location. We also had tunable amounts of domain sharding so that the client could open N http2 conns at once to a given edge hostname (cdn[1-4].facebook.com vs just api.facebook.com or www.facebook.com). This helped with head of line blocking, assuming that your ISP wasn't hopelessly over-congested. In that case, no amount of sharding will help you, and sometimes it even works against you depending on what tcp middleware your mobile ISP is using. We did different TCP settings for edge<->origin conns than edge<->user which let us balance between latency and throughput for origin traffic as needed, and we did some interesting work tuning edge tcp settings differently for different user networks too. Things like determining the best congestion window for a given ISP's users, how large to set the send buffers, etc. Admittedly, this work was much easier since we had good http and tcp instrumentation, and we had high quality links between edge and origin. I just wanted people reading your comment to understand the tradeoffs. We saw a measurable difference in perf with a full http2 stack that made up for the extra complexity of running http2 top to bottom.

by zimbatm 10y ago

riobard 10y ago

SCTP is problematic because middle boxes need to be made aware of it since it uses a different IP protocol. This is gonna require massive infrastructure changes and probably won't fly (like IPv6). QUIC, which runs over UDP, is easier to adopt. However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable. TCP is still the safest bet. We need to figure out a better (and magical!) way somehow...

AndrewDucker 10y ago
IPv6 is over 10% of Google's traffic, and is doubling year-on-year. If SCTP was failing to fly that badly it would make me very happy!
- daenney 10y ago
  But its taken us 20 years to get to this point and even with the IPv4 exhaustion it's still relatively slow to be picked up on by providers in Europe or North America (except for mobile, now). Not to say I wouldn't want SCTP to succeed but IPv6 doesn't exactly spring to mind as a model for the successful adoption of a new internet protocol.
panic 10y ago
SCTP is problematic because middle boxes need to be made aware of it since it uses a different IP protocol. Is this true? As far as I can tell SCTP works over plain IP, just like UDP and TCP do.
- riobard 10y ago
  SCTP uses IP protocol number 132. TCP is 6 and UDP is 17. Many middle boxes, especially cheap home routers, are not aware of IP protocols other than TCP/UDP (and maybe GRE/ESP if you're lukcy). Their NAT will not work with SCTP.
  - garaetjjte 10y ago
    But there's IPv6 coming. (so NATs will be gone)
    - alyandon 10y ago
      That really assumes residential ISPs out there decide to fully embrace the idea behind IPv6 instead versus doing something asinine like handing out a single /128 address. Fortunately, mine hands me a /64 so I'm not in that situation. :)
staunch 10y ago
> However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable. What makes you think this? I play a UDP-based FPS game where "ping" times are critically important and it seems like latencies across the board have continued to drop over the years. Latencies are ~2x the speed of light in most case. Not too shabby.
- riobard 10y ago
  Your connection looks pretty good. My home ISP throttles UDP traffic to about 5Mbps while TCP reaches 80+Mbps. Packet loss during peak hours can be as high as 60% for UDP but a manageable single digit percent for TCP. My workplace has anti-BitTorrent mechanism deployed and any meaningful amount of UDP outbound traffic is dropped.
DanielDent 10y ago
[quickly becoming standard comment for me]: IPv6 is in widespread use. IPv6 is not "the future". It's the present. I have services in production where >30% of users access the service over v6. Use is growing. IPv6-only mobile operators are also now a thing. The idea that middle boxes need to be aware of protocols is incredibly destructive to the end-to-end principal - the critical property of IP networks that enables a rapid pace of permissionless innovation. Middleboxes are evil incarnate. Fortunately, academic research as well as operational experience has shown that TCP and UDP provide the primitives required to build any protocol. QUIC, which you mentioned, is an example of this. Google has made the explicit - and probably wise - decision to keep protocol state encrypted. While there would be some nice optimizations possible if routers could be aware of things like 'this session is closed now', the risk of middleboxes making it so that the protocol can't continue to evolve is too great. The solution is to specifically prevent middleboxes from having the knowledge needed meddle in an 'intelligent' way. The last statistics I saw from Google were that 95% of networks don't cause problems for QUIC (i.e. QUIC connections are as fast or faster than HTTP/2). The main issue I'm aware of is not so actually de-prioritization but rather rate limiting (it's often implemented as a quick - and very very dirty - DDoS mitigation technique). Google's stated solution is to fallback to HTTP/2 based on measurements of user experience on a per-AS basis. Networks which try to be "intelligent" rather than just provide adequate bandwidth will tend towards providing a crappy user experience. Solutions which optimize for well behaved adequately provisioned networks and have an acceptable-though-degraded fallback seems like a reasonable way to promote progress. SCTP has an RFC-defined UDP encapsulation which will deal with many evil networks.
- eeeeeeeeeeeee 10y ago
  Until you can run an IPv6 service without dual-stack, it's not in the present in the same way you're talking about. We are still very much in the transition period. Yes, it's positive to see IPv6 growth, but the fact remains that any new service today must provide both IPv4 and IPv6 end-points and that is likely to continue for years.
- nucleardog 10y ago
  > IPv6 is in widespread use. IPv6 is not "the future". It's the present. I have services in production where >30% of users access the service over v6. Use is growing. IPv6-only mobile operators are also now a thing. That's... great, and I'm glad there's progress being made somewhere, but I don't think it's fair to say it's "the present". It's on its way. My ISP has rolled out FttH across a wide area but still only offers IPv6 on business buildouts. Even if they offered IPv6 to general customers, they still have a ton of home gateway devices out there that don't support any IPv6 at all. The other ISPs don't offer it or have any external roadmap either. I don't think this situation is all that unusual or unique - it's just not something that the people paying the bills are clamouring for and it's a long road from where we are to significant penetration. Oh, and EC2 still doesn't support IPv6 on anything but their load balancers. What portion of the market and traffic do you think they make up?

panic 10y ago

I wonder why SCTP isn't more widely-supported. It seems like a lot of Internet traffic is message-based rather than stream-based and would benefit from a more appropriate protocol.

abstractbeliefs 10y ago
I've actually been looking into (and started using) SCTP. It's pretty good, except some places seem to have it turned off. In particular, Windows has little support.
mikeash 10y ago
Seems to me that the basic Internet protocols got more or less frozen in time in the 90s, not because they had reached perfection, but just because it became too difficult to support new ones. With ubiquitous firewalls and NAT devices, anything that didn't fit into the existing TCP/UDP system would fail to work with so much of the Internet that people just wouldn't bother with it. It really is unfortunate that we ended up with UDP and TCP as the only choices. There are so many more things you might want outside of "unreliable packets" and "automatically recovered streams."
- nocarrier 10y ago
  The list of firewall-punchable level 4 protocols is frozen in time to just TCP and UDP, and it's sad at first glance, but it's not as bad as we think: * Most ISPs don't mangle UDP, and if you care enough about perf to be using UDP in the first place, you can afford to figure out who mangles UDP and fall back to TCP for those networks. It's a very small number of networks. * Any TCP/SCTP/etc inspired protocol you can think of can be implemented on top of UDP. Its header is only eight bytes long and it adds minimal overhead. UDP socket semantics are simple enough that you can treat them as a slightly smarter IP socket and build your protocol as if you were running directly on top of IPv6. * There's a huge advantage to being able to implement your protocol in userland since you get fine grained control over congestion and connection logic. There's a huge disadvantage too, since it's actually hard to implement that stuff yourself. But for the people who are large enough to build their own protocols (FB, Google, etc), it's worth it. * You can even encapsulate SCTP inside UDP to punch through middleware firewalls if you want to. I don't know of anyone doing that for real in the wild, but there's a RFC for it (RFC 6951).

hitr 10y ago

My understanding is that http/2 completely fixed HOL blocking .If any streams has any issues ,server or client can reset that stream independently [1]. Isn't that the case ? [1] https://http2.github.io/http2-spec/#StreamErrorHandler https://http2.github.io/http2-spec/#StreamErrorHandler

rgbrenner 10y ago
There's http1's head of line blocking.. and tcp's head of line blocking. http2 fixed the former.. the latter applies to any tcp connection--including http 1.x, http2, etc (anything that uses tcp). In other words.. he's saying, because HTTP 2 is multiplexed, tcp's head of line blocking becomes a larger problem. Which is true.

rgbrenner 10y ago

Couldn't agree more. When I was implementing adaptive routing on NuevoCloud that was one of the problems I worked on. Ended up de-multiplexing http2, once the requests hit the edge node. So there's still one connection between the edge and the browser (can't do anything about that), but once it gets to the edge node, each request is sent over a prenegotiated connection to an edge node near the server.. then it drops down to a more reasonable number when sending the requests to the customer's server. So if a packet is dropped, it only affects that one file/connection. Don't think any other CDNs are doing this yet.. but I think it'll become more common as they invest more resources into their http2 implementations.

nocarrier 10y ago
This works for your use case which is great, but in my experience, you get gains with full http2 between edge and origin that you miss out on if you demux or fall back to http 1.1. We saw slightly better median latencies and much better >90th percentile latencies with full http2 from edge to origin, versus when we did http2 to edge users and http 1.1 to origin. At Facebook, we had clients talk http2 to edge nodes near the users, which then edge terminated TLS and opened up the request to figure out which upstream was best to forward the request to. We kept http2 conn pools to each upstream from a given edge location. We also had tunable amounts of domain sharding so that the client could open N http2 conns at once to a given edge hostname (cdn[1-4].facebook.com vs just api.facebook.com or www.facebook.com). This helped with head of line blocking, assuming that your ISP wasn't hopelessly over-congested. In that case, no amount of sharding will help you, and sometimes it even works against you depending on what tcp middleware your mobile ISP is using. We did different TCP settings for edge<->origin conns than edge<->user which let us balance between latency and throughput for origin traffic as needed, and we did some interesting work tuning edge tcp settings differently for different user networks too. Things like determining the best congestion window for a given ISP's users, how large to set the send buffers, etc. Admittedly, this work was much easier since we had good http and tcp instrumentation, and we had high quality links between edge and origin. I just wanted people reading your comment to understand the tradeoffs. We saw a measurable difference in perf with a full http2 stack that made up for the extra complexity of running http2 top to bottom.