4 ms·

Is anyone else feeling quite sad reading this article? ALPN being used because only 80/443 are realistic these days, middleboxes causing the TLS handshake to ha

Is anyone else feeling quite sad reading this article? ALPN being used because only 80/443 are realistic these days, middleboxes causing the TLS handshake to have padding so it's not misinterpreted with an ancient protocol (SSLv2). It feels like the Internet is so fragile.

deleted 6y ago
[deleted]
profmonocle 6y ago
ALPN would make sense for something like HTTP2 even if you didn't have the problem of ports being blocked. If HTTP2 had its own port clients would have to make multiple TCP connection attempts for each host they connect to.
mrkurt 6y ago
I have the opposite feeling, the clever "hacks" people use to build very useful stuff that bypasses most problems with legacy infrastructure are pretty exciting. It's very much like watching a complex organism evolve into something you never really could have imagined 8000 iterations ago.
SahAssar 6y ago
Most of this could have been avoided by using DOH and SRV records for HTTP/HTTPS. I still don't understand why SRV records is not supported for HTTP/HTTPS in browsers.
- ancarda 6y ago
  I remember looking into why A/AAAA is still used over SRV, and it would seem performance is one of the big concerns; browsers do not want to make more DNS lookups than necessary. I think they'd end up with 4 lookups; A, AAAA, SRV (_http2._tls), and SRV (_http._tls). Though perhaps you are suggesting DoH could mean the resolver also returns SRV records if you request A or AAAA? i.e. proactively point out there's an HTTP server?
  - SahAssar 6y ago
    IIRC chrome is now racing QUIC and HTTP(2) connections instead of doing negotiation/upgrade to detect QUIC support. So this argument (if true) has fallen apart just a few years later.
    - tptacek 6y ago
      Isn't that because QUIC doesn't overload 443/tcp, and sending a UDP packet is extremely cheap? It's not the packets that kill you, it's the round trip latency.
      - SahAssar 6y ago
        My point was that they could just as well race the DNS lookups, right?
        yencabulator 6y ago
        That doesn't solve all the problems listed in https://news.ycombinator.com/item?id=23648497 https://news.ycombinator.com/item?id=23648497 and likely makes the buggy DNS resolver issue worse.
  - jiggawatts 6y ago
    This debate comes up a lot, and it's hilarious how misguided it was. I regularly work with load-balancers such as Citrix ADC (NetScaler) or F5 BIG IP. These do DNS-based load-balancing, dynamically returning "A" records to that the browsers so that they can get the "single working IP address" they're expecting. The browsers don't try very hard to fail over to secondary IPs because this is the established standard architecture, but they don't need to because of this common setup. Sounds like an optimal solution, right? It does at first glance anyway, as long as you ignore the eye-watering price tag on those load balancer boxes. The subtle but critical issue is that by returning "A" records, the load balancers have to use a short time to live (TTL)! This is because there's a trade-off: You can have fast failover, OR long-lived DNS caching. With A records you can't have both! Typical response TTL times are 5-30 seconds, 5 minutes tops if you hate your users. This means that many browsers will be forced to repeatedly re-query the DNS servers on every page load for typical end-user workflows. It also means that for all but the biggest, most popular sites, the ISP DNS cache does practically nothing for these records. Meanwhile with SRV records the TTL times can be much higher, hours even. This is how Active Directory works, for example, all of the Domain Controllers add themselves to various SRV records so that if you query "_ldap._tcp.dc._msdcs.test.com" you get back all the DCs. These records include priorities and weightings, so you can pull tricks like incrementally demote a DC or prioritise the shiny new one. If you watch the AD connection traffic in WireShark, it's incredible. It very quickly steps through alternate services and then reorders the successful hits in front of the failures so that subsequent queries are lightning fast. It is astonishingly tolerant of partial networking failures, yet still fast to connect despite that! The key mistake made by the original DNS design working groups was that SRV records should have returned a list of IP addresses instead of a list of host names.
- mrkurt 6y ago
  We actually run into problems that are similar to what you'd have with SRV records. Fly.io apps can define different service "handlers" (like TLS and HTTP). If you want to, you can accept TCP connections and bypass our logic. Which is great and flexible. The problem is, when someone is deploying a new version of their app where they _change_ one of those things, we have to e really careful about how we (a) load balance and (b) decide to do things like TLS. If we're not careful we can end up sending the wrong type of connection to a new VM that's expecting something else. SRV records sound like they'd have it worse. If you do a DNS lookup to detect something like http2, the IP you connect to _can't_ do anything else. It's much simpler / safer to negotiate stuff like that at connection time.
  - SahAssar 6y ago
    That all assumes that you use the same port for all those things, right? Which is one of the points of SRV, to not have to squeeze everything into one canonical port. With a SRV record you could route https to whatever port suited you and rotate that out when rolling out changes,
    - mrkurt 6y ago
      Yeah that's feasible, although it still sounds more complicated than doing per connection protocol negotiations to me. But I like ALPN so I'm biased.
- sleevi 6y ago
  While folks may be thinking “number of queries” must mean it was performance concerns, the answer for Chromium was simpler: SRV records just didn’t resolve for a number of users. Neither do TXT records, for that matter ( https://www.imperialviolet.org/2015/01/17/notdane.html https://www.imperialviolet.org/2015/01/17/notdane.html ). Early on in Chrome, it was discovered that attempting to resolve too many domains at once could cause 2Wire routers used by AT&T for DSL to lock up, requiring you to physically reboot, due to a bug in their port of dnsmasq. Stories like this are stories like those mentioned in the post: everything on the Internet is awful, and you work with what you have and work around things by adding JPEGs of cats and silly session IDs (As TLS 1.3 does) and whatever it takes to “make it work”. DoH/DoT, if it’s able to be adopted before ISPs and middleboxes ossify it with subpar implementations as part of their effort to avoid net neutrality ( https://www.theverge.com/2019/10/4/20898779/fcc-net-neutrality-court-of-appeals-decision-ruling https://www.theverge.com/2019/10/4/20898779/fcc-net-neutrali... ), is the first time in a long time that innovation and experimentation in DNS might happen again. This is why we see things like ESNI and SVCB ( https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc ) having a chance at being deployed. At least, if clients can get there before the middleboxes ruin the party, like they’ve tried to do for QUIC.

deleted 6y ago

[deleted]

profmonocle 6y ago

ALPN would make sense for something like HTTP2 even if you didn't have the problem of ports being blocked. If HTTP2 had its own port clients would have to make multiple TCP connection attempts for each host they connect to.

mrkurt 6y ago

I have the opposite feeling, the clever "hacks" people use to build very useful stuff that bypasses most problems with legacy infrastructure are pretty exciting. It's very much like watching a complex organism evolve into something you never really could have imagined 8000 iterations ago.

SahAssar 6y ago

Most of this could have been avoided by using DOH and SRV records for HTTP/HTTPS. I still don't understand why SRV records is not supported for HTTP/HTTPS in browsers.

ancarda 6y ago
I remember looking into why A/AAAA is still used over SRV, and it would seem performance is one of the big concerns; browsers do not want to make more DNS lookups than necessary. I think they'd end up with 4 lookups; A, AAAA, SRV (_http2._tls), and SRV (_http._tls). Though perhaps you are suggesting DoH could mean the resolver also returns SRV records if you request A or AAAA? i.e. proactively point out there's an HTTP server?
- SahAssar 6y ago
  IIRC chrome is now racing QUIC and HTTP(2) connections instead of doing negotiation/upgrade to detect QUIC support. So this argument (if true) has fallen apart just a few years later.
  - tptacek 6y ago
    Isn't that because QUIC doesn't overload 443/tcp, and sending a UDP packet is extremely cheap? It's not the packets that kill you, it's the round trip latency.
    - SahAssar 6y ago
      My point was that they could just as well race the DNS lookups, right?
      - yencabulator 6y ago
        That doesn't solve all the problems listed in https://news.ycombinator.com/item?id=23648497 https://news.ycombinator.com/item?id=23648497 and likely makes the buggy DNS resolver issue worse.
- jiggawatts 6y ago
  This debate comes up a lot, and it's hilarious how misguided it was. I regularly work with load-balancers such as Citrix ADC (NetScaler) or F5 BIG IP. These do DNS-based load-balancing, dynamically returning "A" records to that the browsers so that they can get the "single working IP address" they're expecting. The browsers don't try very hard to fail over to secondary IPs because this is the established standard architecture, but they don't need to because of this common setup. Sounds like an optimal solution, right? It does at first glance anyway, as long as you ignore the eye-watering price tag on those load balancer boxes. The subtle but critical issue is that by returning "A" records, the load balancers have to use a short time to live (TTL)! This is because there's a trade-off: You can have fast failover, OR long-lived DNS caching. With A records you can't have both! Typical response TTL times are 5-30 seconds, 5 minutes tops if you hate your users. This means that many browsers will be forced to repeatedly re-query the DNS servers on every page load for typical end-user workflows. It also means that for all but the biggest, most popular sites, the ISP DNS cache does practically nothing for these records. Meanwhile with SRV records the TTL times can be much higher, hours even. This is how Active Directory works, for example, all of the Domain Controllers add themselves to various SRV records so that if you query "_ldap._tcp.dc._msdcs.test.com" you get back all the DCs. These records include priorities and weightings, so you can pull tricks like incrementally demote a DC or prioritise the shiny new one. If you watch the AD connection traffic in WireShark, it's incredible. It very quickly steps through alternate services and then reorders the successful hits in front of the failures so that subsequent queries are lightning fast. It is astonishingly tolerant of partial networking failures, yet still fast to connect despite that! The key mistake made by the original DNS design working groups was that SRV records should have returned a list of IP addresses instead of a list of host names.
mrkurt 6y ago
We actually run into problems that are similar to what you'd have with SRV records. Fly.io apps can define different service "handlers" (like TLS and HTTP). If you want to, you can accept TCP connections and bypass our logic. Which is great and flexible. The problem is, when someone is deploying a new version of their app where they _change_ one of those things, we have to e really careful about how we (a) load balance and (b) decide to do things like TLS. If we're not careful we can end up sending the wrong type of connection to a new VM that's expecting something else. SRV records sound like they'd have it worse. If you do a DNS lookup to detect something like http2, the IP you connect to _can't_ do anything else. It's much simpler / safer to negotiate stuff like that at connection time.
- SahAssar 6y ago
  That all assumes that you use the same port for all those things, right? Which is one of the points of SRV, to not have to squeeze everything into one canonical port. With a SRV record you could route https to whatever port suited you and rotate that out when rolling out changes,
  - mrkurt 6y ago
    Yeah that's feasible, although it still sounds more complicated than doing per connection protocol negotiations to me. But I like ALPN so I'm biased.
sleevi 6y ago
While folks may be thinking “number of queries” must mean it was performance concerns, the answer for Chromium was simpler: SRV records just didn’t resolve for a number of users. Neither do TXT records, for that matter ( https://www.imperialviolet.org/2015/01/17/notdane.html https://www.imperialviolet.org/2015/01/17/notdane.html ). Early on in Chrome, it was discovered that attempting to resolve too many domains at once could cause 2Wire routers used by AT&T for DSL to lock up, requiring you to physically reboot, due to a bug in their port of dnsmasq. Stories like this are stories like those mentioned in the post: everything on the Internet is awful, and you work with what you have and work around things by adding JPEGs of cats and silly session IDs (As TLS 1.3 does) and whatever it takes to “make it work”. DoH/DoT, if it’s able to be adopted before ISPs and middleboxes ossify it with subpar implementations as part of their effort to avoid net neutrality ( https://www.theverge.com/2019/10/4/20898779/fcc-net-neutrality-court-of-appeals-decision-ruling https://www.theverge.com/2019/10/4/20898779/fcc-net-neutrali... ), is the first time in a long time that innovation and experimentation in DNS might happen again. This is why we see things like ESNI and SVCB ( https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc ) having a chance at being deployed. At least, if clients can get there before the middleboxes ruin the party, like they’ve tried to do for QUIC.