8 ms·

The Illustrated TLS 1.3 Connection: Every Byte Explained

568 points by syncsynchalt 8y ago

mcguire 8y ago
"Because middleboxes have been created and widely deployed that do not allow protocol versions that they do not recognize, the TLS 1.3 session must be disguised as a TLS 1.2 session." And, from the bottom of my network-protocol-guy's heart, I hate them and everyone who's ever even seen one with the fiery passion of a million very angry suns.
- sigjuice 8y ago
  What would be some examples of these middle boxes?
  - detaro 8y ago
    Typically "smart" firewalls/internet filtering things.
  - jeroenhd 8y ago
    Load balancers, IDS systems, traffic analysis, bad firewalls, content filtering systems, transparent proxies, some kinds of "intelligent switches", consumer routers with AV capabilities. Also anything sold by Blue Coat. You can read detailed examples on Cloudflare's blog, they've been very eager to experiment with TLS 1.3: https://blog.cloudflare.com/why-tls-1-3-isnt-in-browsers-yet/ https://blog.cloudflare.com/why-tls-1-3-isnt-in-browsers-yet...
  - LinuxBender 8y ago
    IDP's / IPS's, firewalls that need to inspect payload, MitM Proxies such as Bluecoat, Websense, etc.., network appliances that create layer-7 flow data for debugging applications such as TrueSight and many more. Most big companies have contractual requirements to have an IPS/IDP to protect their customers. TLS1.3 means putting the decryption at the edge, which not everyone has done, then re-encrypting with something their IDP can decrypt.
    - LeonM 8y ago
      I visited an infosec trade show recently, you wouldn't believe how many of these boxes are offered to 'improve' corporate security... Most of these boxes will break security chains, spoof DNS responses, require self-signed certs, etc. It's garbage. Kinda reminded me of 'the box' from the Silicon Valley TV-show. I guess corporate managers will prefer something tangible over actual good practices.
      - supertrope 8y ago
        Training is expensive and people will still click on ransomware payloads. Security appliances outsource blame.
      - jandrese 8y ago
        My favorite are the ones that re-sign connections without ever verifying that the original connection was signed by a legitimate authority.
        Tharkun 8y ago
        I encountered an interesting one at $bigcorp which required everyone in the corp to install shady certificates, and which would then inject <script> tags in all HTTP responses, including in XML and JSON REST responses. That was fun to debug.
        jandrese 8y ago
        It's pretty common for these devices to require you to install their generated CA cert on every machine in the corporation. Administrators deploy it to the domain with a group policy object IIRC. Firefox users end up having to install it by hand, or else every website comes up as "you are being attacked by a MITM!!!!", which is technically true.
        spockz 8y ago
        Sadly this also puts laptop users in the situation that they always need to vpn to the office otherwise sites that use HSTS will break either when on or outside the corporate networks, depending on when you visited the site first.
- nerdponx 8y ago
  Was explicit version whitelisting ever a legitimate security recommendation? Naively, it seems like a sane defensive measure against unanticipated behavior.
  - deleted 8y ago
    [deleted]
  - tialaramex 8y ago
    The version numbers are for negotiation purposes. One reasonable thing to do when the remote peer says "I want to speak TLS 8.6" is to say "Oh, I'm afraid we only allow TLS 1.2 here" and insist on talking TLS 1.2 anyway. But that's not what these boxes did. They go "TLS 8.6! Set Defence Condition One. Attack In Progress! Prepare For Immediate Nuclear Strike!" and they tear down the whole network connection. They did this with every previous version of TLS. The accepted lesson at this point is that if these idiots CAN break something they will, we don't care whether it's because they're too stupid not to break it or because they're actively assholes breaking things so nobody can have nice things, it doesn't matter. This is basically the network equivalent of that guy who just shoots anybody he doesn't recognise on his front step. We don't call that "a sane defensive measure" we call it murder, even if he thinks he was just on guard for "unanticipated behaviour" he's going to jail.
    - snazz 8y ago
      The reason for this is to prevent downgrade attacks where an attacker can perform a man-in-the-middle by forcing your client to use an insecure protocol version. It makes sense, even if it occasionally breaks websites.
      - mcguire 8y ago
        Preventing downgrade attacks could be done by blacklisting known-bad versions, no? Assuming all future versions are bad leads to situations like this, where TLS 1.3 (and presumably all future versions) have to tunnel through a pseudo-TLS 1.2 connection.
- deleted 8y ago
  [deleted]
- rkagerer 8y ago
  Don't worry, this clears it up: The particular version number (3.3 representing version 1.2) is due to TLS 1.0 being a minor revision of the SSL 3.0 protocol. Therefore TLS 1.0 is numeric version 3.1, TLS 1.1 is version 3.2, and so on. Talk about versioning schemes bastardized to hell. If people do that to my specs I hope I'm not around to see it.
- rlpb 8y ago
  There's an interesting approach to tackling this problem described here: https://www.ietf.org/mail-archive/web/tls/current/msg26385.html https://www.ietf.org/mail-archive/web/tls/current/msg26385.h... Summary: constantly make up new protocol "versions". If everything did this, then middleboxes "that do not allow protocol versions that they do not recognize" will never be functional in the first place.
mcguire 8y ago
And in honor of the records + messages structure of the protocol, I link David Tennenhouse's beloved 1989 paper, "Layered Multiplexing Considered Harmful". https://www.cl.cam.ac.uk/teaching/0708/DigiCommI/tennenhouse1989layered.pdf https://www.cl.cam.ac.uk/teaching/0708/DigiCommI/tennenhouse...
rnotaro 8y ago
Other thread from 26 days ago with the author: https://news.ycombinator.com/item?id=18200749 https://news.ycombinator.com/item?id=18200749
- odensc 8y ago
  That one's TLS 1.2, it's cool to see the differences though.
ecesena 8y ago
Can anyone confirm if the server certificate in the server hello message is encrypted or not? I've heard many time it's encrypted, but I can't find it explicitly written in the RFC. And it seems unencrypted in this example.
- rwbhn 8y ago
  According to the post the server cert is not sent in the server hello, but later in the handshake, after encryption has been established.
  - rwbhn 8y ago
    Section 4.4 of the rfc[1] makes it clear the cert is sent encrypted. [1] https://tools.ietf.org/html/rfc8446#section-4.4 https://tools.ietf.org/html/rfc8446#section-4.4
- tialaramex 8y ago
  As someone explained the certificate isn't part of ServerHello. After ServerHello is sent, the server knows a client now has everything it needs to establish all the keys, so it encrypts all subsequent data. The pretence that this is just a TLS 1.2 session resumption continues though. So there's a wrapper shown in the document. This wrapper claims to be TLS 1.2 encrypted application data. In fact it's TLS 1.3 encrypted handshake data. For clarity in the document they show both the whole encrypted wrapper, its decrypted data, and then the separate components inside that decrypted data.
quotemstr 8y ago
I wish we were more willing to break bad middleboxes. It'd cause more short-term pain than TLS1.3's compatibility hacks, but it'll get us a better internet in the long term. We can't let random bad intermediate nodes ossify protocol development.
- pilif 8y ago
  They did that for TLS 1.2. It took the better part of a decade before it became usable and could be enabled by browsers as a version to try. Being able to use the improved protocol now is very nice.
  - spacenick88 8y ago
    That's why this time it should be tied to the power of the biggest IT firms. If Apple, Google, Facebook and Microsoft were willing to let their servers break the middle boxes even the most fucked up, stupid, slow IT department would have to kill their broken middle boxes. I'd argue it would take less than 48h hours to kill all middle boxes worldwide and probably all broken browser versions too. Just imagine what it would mean for a Big Corp not to be able to access Google, Bing, Facebook and iCloud from their networks. It's either move or go bankrupt within days. In a way that's kind of the nuclear option, but hell nukes do work.
benkillin 8y ago
What changes in the exchange if mutual tls authentication is being performed (aka client cert authentication)?
- syncsynchalt 8y ago
  I wrote up my (possibly incomplete) understanding of client cert auth on the reddit post when someone had a similar question: https://www.reddit.com/r/programming/comments/9v0xr8/rewritten_for_tls_13_the_new_illustrated_tls/e99hfo9/ https://www.reddit.com/r/programming/comments/9v0xr8/rewritt...
- tialaramex 8y ago
  It's actually more similar in TLS 1.3 than earlier versions. In earlier versions a client uses a very different method to prove its identity then the server does. In TLS 1.3 they're completely symmetrical. First they show their certificate and then they sign the entire transcript of the session handshake using their private key and send that signature. The peer can use the public key from the certificate and their own copy of the transcript to verify the signature. If your peer doesn't offer an acceptable cert you abort. If they show you a signature that doesn't verify you abort. This could be because they're an imposter who doesn't have the private key, or it could be that a MitM has changed the handshake, which will mean the transcripts don't match. Either way you haven't got a secure connection. The TLS 1.3 server gets to be more specific about the cert it wants from a client. Where before it just says "Here's a list of CA roots I trust" now it can specify other filters such as here are some OIDs I want to see in the Certificate Policy.
  - brians 8y ago
    That’s a great write up. Another important change is that renegotiation to add client cents is no longer supported. Old HTTPS could work like this: C->S: handshake, no client cert C->S: GET / C<-S: here you go C->S: POST /launch?target=Moscow C<-S: server-initiated-renegotiation C->S: everything from *this* byte on is covered by C->s: client cert C<-S: okay, launched That is, the entangling of HTTP & TLS made some assumptions about the authentication parts being there for controlling read access, not about authenticating writes. Whoops. Nearly nobody using client cents wanted that behavior. Now there are much simpler mechanisms with many fewer bizarre side effects proposed at https://tools.ietf.org/id/draft-ietf-httpbis-http2-secondary-certs-01.html https://tools.ietf.org/id/draft-ietf-httpbis-http2-secondary...
deleted 8y ago
[deleted]
profmonocle 8y ago
Once these middlebox vendors catch up to TLS 1.3, what's to stop them from breaking the supported versions extension the same way they broke the client version field? Anyone know if Chrome is using/planning to use GREASE with this extension? (Couldn't find anything about it on Google.)
- pilif 8y ago
  > Once these middlebox vendors catch up to TLS 1.3, what's to stop them from breaking the supported versions extension the same way they broke the client version field Nothing. It will happen just the same, so TLS 1.4 will have a “actually, this is the extension to use to determine the version”-extension
- laggyluke 8y ago
  Hopefully efforts like this will make further ossification harder: https://www.ietf.org/mail-archive/web/tls/current/msg26385.html https://www.ietf.org/mail-archive/web/tls/current/msg26385.h...
_pmf_ 8y ago
It's so sad that examples like these are not part of every spec (in fact, the very first thing in any spec). I understand the dangers of programming by example rather than by spec, but a lack of canonical, verified examples has been a problem with every protocol I've had to implement.
- syncsynchalt 8y ago
  Good news: for cryptographic protocols and methods they are! Many of the cryptographic primitives that I implemented for my test TLS 1.3 implementation (https://github.com/syncsynchalt/tincan-tls https://github.com/syncsynchalt/tincan-tls) come with test vectors in the spec itself, and the rest always had test vectors supplied in a separate doc. For an example see Appendix C of the AES standard, FIPS-197: https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197.pdf https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197.pdf In the case of TLS 1.3 I was able to use this IETF draft: https://tools.ietf.org/html/draft-ietf-tls-tls13-vectors-07 https://tools.ietf.org/html/draft-ietf-tls-tls13-vectors-07
edwinyzh 8y ago
Is there any similar illustrated explanations for other network protocols? I mostly interested in IMAP4 and HTTP.

568 points by syncsynchalt 8y ago

"Because middleboxes have been created and widely deployed that do not allow protocol versions that they do not recognize, the TLS 1.3 session must be disguised as a TLS 1.2 session." And, from the bottom of my network-protocol-guy's heart, I hate them and everyone who's ever even seen one with the fiery passion of a million very angry suns.

sigjuice 8y ago
What would be some examples of these middle boxes?
- detaro 8y ago
  Typically "smart" firewalls/internet filtering things.
- jeroenhd 8y ago
  Load balancers, IDS systems, traffic analysis, bad firewalls, content filtering systems, transparent proxies, some kinds of "intelligent switches", consumer routers with AV capabilities. Also anything sold by Blue Coat. You can read detailed examples on Cloudflare's blog, they've been very eager to experiment with TLS 1.3: https://blog.cloudflare.com/why-tls-1-3-isnt-in-browsers-yet/ https://blog.cloudflare.com/why-tls-1-3-isnt-in-browsers-yet...
- LinuxBender 8y ago
  IDP's / IPS's, firewalls that need to inspect payload, MitM Proxies such as Bluecoat, Websense, etc.., network appliances that create layer-7 flow data for debugging applications such as TrueSight and many more. Most big companies have contractual requirements to have an IPS/IDP to protect their customers. TLS1.3 means putting the decryption at the edge, which not everyone has done, then re-encrypting with something their IDP can decrypt.
  - LeonM 8y ago
    I visited an infosec trade show recently, you wouldn't believe how many of these boxes are offered to 'improve' corporate security... Most of these boxes will break security chains, spoof DNS responses, require self-signed certs, etc. It's garbage. Kinda reminded me of 'the box' from the Silicon Valley TV-show. I guess corporate managers will prefer something tangible over actual good practices.
    - supertrope 8y ago
      Training is expensive and people will still click on ransomware payloads. Security appliances outsource blame.
    - jandrese 8y ago
      My favorite are the ones that re-sign connections without ever verifying that the original connection was signed by a legitimate authority.
      - Tharkun 8y ago
        I encountered an interesting one at $bigcorp which required everyone in the corp to install shady certificates, and which would then inject <script> tags in all HTTP responses, including in XML and JSON REST responses. That was fun to debug.
        jandrese 8y ago
        It's pretty common for these devices to require you to install their generated CA cert on every machine in the corporation. Administrators deploy it to the domain with a group policy object IIRC. Firefox users end up having to install it by hand, or else every website comes up as "you are being attacked by a MITM!!!!", which is technically true.
        spockz 8y ago
        Sadly this also puts laptop users in the situation that they always need to vpn to the office otherwise sites that use HSTS will break either when on or outside the corporate networks, depending on when you visited the site first.
nerdponx 8y ago
Was explicit version whitelisting ever a legitimate security recommendation? Naively, it seems like a sane defensive measure against unanticipated behavior.
- deleted 8y ago
  [deleted]
- tialaramex 8y ago
  The version numbers are for negotiation purposes. One reasonable thing to do when the remote peer says "I want to speak TLS 8.6" is to say "Oh, I'm afraid we only allow TLS 1.2 here" and insist on talking TLS 1.2 anyway. But that's not what these boxes did. They go "TLS 8.6! Set Defence Condition One. Attack In Progress! Prepare For Immediate Nuclear Strike!" and they tear down the whole network connection. They did this with every previous version of TLS. The accepted lesson at this point is that if these idiots CAN break something they will, we don't care whether it's because they're too stupid not to break it or because they're actively assholes breaking things so nobody can have nice things, it doesn't matter. This is basically the network equivalent of that guy who just shoots anybody he doesn't recognise on his front step. We don't call that "a sane defensive measure" we call it murder, even if he thinks he was just on guard for "unanticipated behaviour" he's going to jail.
  - snazz 8y ago
    The reason for this is to prevent downgrade attacks where an attacker can perform a man-in-the-middle by forcing your client to use an insecure protocol version. It makes sense, even if it occasionally breaks websites.
    - mcguire 8y ago
      Preventing downgrade attacks could be done by blacklisting known-bad versions, no? Assuming all future versions are bad leads to situations like this, where TLS 1.3 (and presumably all future versions) have to tunnel through a pseudo-TLS 1.2 connection.
deleted 8y ago
[deleted]
rkagerer 8y ago
Don't worry, this clears it up: The particular version number (3.3 representing version 1.2) is due to TLS 1.0 being a minor revision of the SSL 3.0 protocol. Therefore TLS 1.0 is numeric version 3.1, TLS 1.1 is version 3.2, and so on. Talk about versioning schemes bastardized to hell. If people do that to my specs I hope I'm not around to see it.
rlpb 8y ago
There's an interesting approach to tackling this problem described here: https://www.ietf.org/mail-archive/web/tls/current/msg26385.html https://www.ietf.org/mail-archive/web/tls/current/msg26385.h... Summary: constantly make up new protocol "versions". If everything did this, then middleboxes "that do not allow protocol versions that they do not recognize" will never be functional in the first place.

mcguire 8y ago

And in honor of the records + messages structure of the protocol, I link David Tennenhouse's beloved 1989 paper, "Layered Multiplexing Considered Harmful". https://www.cl.cam.ac.uk/teaching/0708/DigiCommI/tennenhouse1989layered.pdf https://www.cl.cam.ac.uk/teaching/0708/DigiCommI/tennenhouse...

rnotaro 8y ago

Other thread from 26 days ago with the author: https://news.ycombinator.com/item?id=18200749 https://news.ycombinator.com/item?id=18200749

odensc 8y ago
That one's TLS 1.2, it's cool to see the differences though.

ecesena 8y ago

Can anyone confirm if the server certificate in the server hello message is encrypted or not? I've heard many time it's encrypted, but I can't find it explicitly written in the RFC. And it seems unencrypted in this example.

rwbhn 8y ago
According to the post the server cert is not sent in the server hello, but later in the handshake, after encryption has been established.
- rwbhn 8y ago
  Section 4.4 of the rfc[1] makes it clear the cert is sent encrypted. [1] https://tools.ietf.org/html/rfc8446#section-4.4 https://tools.ietf.org/html/rfc8446#section-4.4
tialaramex 8y ago
As someone explained the certificate isn't part of ServerHello. After ServerHello is sent, the server knows a client now has everything it needs to establish all the keys, so it encrypts all subsequent data. The pretence that this is just a TLS 1.2 session resumption continues though. So there's a wrapper shown in the document. This wrapper claims to be TLS 1.2 encrypted application data. In fact it's TLS 1.3 encrypted handshake data. For clarity in the document they show both the whole encrypted wrapper, its decrypted data, and then the separate components inside that decrypted data.

quotemstr 8y ago

I wish we were more willing to break bad middleboxes. It'd cause more short-term pain than TLS1.3's compatibility hacks, but it'll get us a better internet in the long term. We can't let random bad intermediate nodes ossify protocol development.

pilif 8y ago
They did that for TLS 1.2. It took the better part of a decade before it became usable and could be enabled by browsers as a version to try. Being able to use the improved protocol now is very nice.
- spacenick88 8y ago
  That's why this time it should be tied to the power of the biggest IT firms. If Apple, Google, Facebook and Microsoft were willing to let their servers break the middle boxes even the most fucked up, stupid, slow IT department would have to kill their broken middle boxes. I'd argue it would take less than 48h hours to kill all middle boxes worldwide and probably all broken browser versions too. Just imagine what it would mean for a Big Corp not to be able to access Google, Bing, Facebook and iCloud from their networks. It's either move or go bankrupt within days. In a way that's kind of the nuclear option, but hell nukes do work.

benkillin 8y ago

What changes in the exchange if mutual tls authentication is being performed (aka client cert authentication)?

syncsynchalt 8y ago
I wrote up my (possibly incomplete) understanding of client cert auth on the reddit post when someone had a similar question: https://www.reddit.com/r/programming/comments/9v0xr8/rewritten_for_tls_13_the_new_illustrated_tls/e99hfo9/ https://www.reddit.com/r/programming/comments/9v0xr8/rewritt...
tialaramex 8y ago
It's actually more similar in TLS 1.3 than earlier versions. In earlier versions a client uses a very different method to prove its identity then the server does. In TLS 1.3 they're completely symmetrical. First they show their certificate and then they sign the entire transcript of the session handshake using their private key and send that signature. The peer can use the public key from the certificate and their own copy of the transcript to verify the signature. If your peer doesn't offer an acceptable cert you abort. If they show you a signature that doesn't verify you abort. This could be because they're an imposter who doesn't have the private key, or it could be that a MitM has changed the handshake, which will mean the transcripts don't match. Either way you haven't got a secure connection. The TLS 1.3 server gets to be more specific about the cert it wants from a client. Where before it just says "Here's a list of CA roots I trust" now it can specify other filters such as here are some OIDs I want to see in the Certificate Policy.
- brians 8y ago
  That’s a great write up. Another important change is that renegotiation to add client cents is no longer supported. Old HTTPS could work like this: C->S: handshake, no client cert C->S: GET / C<-S: here you go C->S: POST /launch?target=Moscow C<-S: server-initiated-renegotiation C->S: everything from *this* byte on is covered by C->s: client cert C<-S: okay, launched That is, the entangling of HTTP & TLS made some assumptions about the authentication parts being there for controlling read access, not about authenticating writes. Whoops. Nearly nobody using client cents wanted that behavior. Now there are much simpler mechanisms with many fewer bizarre side effects proposed at https://tools.ietf.org/id/draft-ietf-httpbis-http2-secondary-certs-01.html https://tools.ietf.org/id/draft-ietf-httpbis-http2-secondary...

deleted 8y ago

[deleted]

profmonocle 8y ago

Once these middlebox vendors catch up to TLS 1.3, what's to stop them from breaking the supported versions extension the same way they broke the client version field? Anyone know if Chrome is using/planning to use GREASE with this extension? (Couldn't find anything about it on Google.)

pilif 8y ago
> Once these middlebox vendors catch up to TLS 1.3, what's to stop them from breaking the supported versions extension the same way they broke the client version field Nothing. It will happen just the same, so TLS 1.4 will have a “actually, this is the extension to use to determine the version”-extension
laggyluke 8y ago
Hopefully efforts like this will make further ossification harder: https://www.ietf.org/mail-archive/web/tls/current/msg26385.html https://www.ietf.org/mail-archive/web/tls/current/msg26385.h...

_pmf_ 8y ago

It's so sad that examples like these are not part of every spec (in fact, the very first thing in any spec). I understand the dangers of programming by example rather than by spec, but a lack of canonical, verified examples has been a problem with every protocol I've had to implement.

syncsynchalt 8y ago
Good news: for cryptographic protocols and methods they are! Many of the cryptographic primitives that I implemented for my test TLS 1.3 implementation (https://github.com/syncsynchalt/tincan-tls https://github.com/syncsynchalt/tincan-tls) come with test vectors in the spec itself, and the rest always had test vectors supplied in a separate doc. For an example see Appendix C of the AES standard, FIPS-197: https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197.pdf https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.197.pdf In the case of TLS 1.3 I was able to use this IETF draft: https://tools.ietf.org/html/draft-ietf-tls-tls13-vectors-07 https://tools.ietf.org/html/draft-ietf-tls-tls13-vectors-07

edwinyzh 8y ago

Is there any similar illustrated explanations for other network protocols? I mostly interested in IMAP4 and HTTP.