6 ms·

An interesting aspect of this is that it's a bug in NPN, which is more or less a historic artifact. When SPDY, the precursor of HTTP/2, got introduced, it need

by hannob 2y ago

An interesting aspect of this is that it's a bug in NPN, which is more or less a historic artifact. When SPDY, the precursor of HTTP/2, got introduced, it needed a mechanism to signal that within TLS, a different protocol (SPDY instead of HTTP/1.1) was spoken. That mechanism was originally NPN. I don't know the exact details and motivation, but eventually people seem to have decided that NPN wasn't exactly what they wanted, and they invented a new mechanism called ALPN. Now, that was a decade ago (the ALPN RFC is from 2014), so the question is: why do we still have NPN code in OpenSSL? I don't think anyone uses it any more. Shouldn't it have been removed long ago? To put this in a larger context: it appears to me that OpenSSL still has a strong tendency to bloat. Heartbleed was essentially a "we added this feature, although noone knows why we need it" kind of bug, but it doesn't look to me they've changed. I still get the feeling that OpenSSL adds many features that they probably just should ignore (e.g. obscure "not invented here"-type algorithms), and they don't remove features that are obsolete.

lazide 2y ago
Removing old code, especially in open source, has a problem. Unless it is obviously breaking something someone wants to do, it has roughly a zero cost (in maintenance) being left alone, especially since bugs can often be ignored, but a non-zero cost to remove it - as it generally will only garner complaints and whining from any remaining users. And in open source in particular, it’s damn near impossible finding remaining users in advance, or even telling if there are no remaining users.
- RetpolineDrama 2y ago
  Is it really "zero cost in maintenance" if that old code presents attack surface?
  - c0balt 2y ago
    No, it's zero upfront cost in maintenance. Owning code always generates some maintenance/ tech debt but it often is opaque nor easily quantifiable.
    - lazide 2y ago
      IMO the secondary cost is really more ‘friction’ that results when making changes. If no changes get made, then no friction. And open source code can and often is just abandoned. Defacto, sometimes even de jur. User beware, use at your own risk, etc.
      - crngefest 2y ago
        Proprietary code is just as often if not more abandoned - you just don’t notice it
        lazide 2y ago
        Sure, but proprietary code you can (somewhat) see who is calling it, and the group maintaining it is at least sometimes the group calling it - so has an incentive to not make it a bigger mess. Somewhat. So that part of the equation has less weight. They have a counter balancing thing which is no one can see it to shame them.
  - lazide 2y ago
    Does the open source code maintainer have liability for it, or necessarily need to do work in that case? At worst they generally just suffer reputational damage, not actual cost or lost revenue, like a business would.
fullspectrumdev 2y ago
Why it’s not removed: backward compatibility I bet. I have to maintain a few VM’s and statically linked tools specifically for interacting with old network appliances that modern Linux hosts won’t “talk to” without crippling their SSL/TLS configuration or doing other horrendous workarounds like specific OpenSSL/OpenSSH config files for those things. It’s actually an interesting problem when performing security assessments - some tools for scanning will have false negatives because they can’t “talk to” old shit.
- jopsen 2y ago
  Couldn't you just fallback to HTTP/1.1? I guess it could be that it's not possible.
- hannob 2y ago
  You have some VMs that talk to old network appliances that require NPN? And if NPN is not supported they don't just fall back to simple HTTP/1.1? I find that sounds extremely unlikely. I am of course well aware of backwards compatibility issues. But I don't see how they'd impact NPN. You might have to update some code that hard-codes NPN functionality, but well, OpenSSL has done API changes in the past, quite significantly so.
  - throwway120385 2y ago
    Not OP, but I have some old network-connected appliances in the field that I can't upgrade and that our current OpenSSH clients have deprecated all of the ciphers and methods for. You still have to be able to talk to these things over the network, and ideally you'd want to upgrade at least some of the systems out there. You have to understand that outside of silicon valley and the hobbyist realm there are tons and tons of systems out there where the security posture is "don't allow anyone to have physical or network access to these systems who doesn't belong here" because the companies that made them deprecated them a decade ago or they were built before the widespread practice of supporting the Linux BSP for more than a couple of months. These systems still work and you won't convince very many users of these systems to spend a bunch of money upgrading them for some notional security risk. You may also be working for a company that is contractually obligated to support these systems in some way. So what do you do? In many cases, you do the best you can to mitigate the risks you know are there with tools like firewalls, port knocking, non-standard ports, and so on. And you hope that the 2-3% of attackers that actually know what they're doing never try to crack these things.
toast0 2y ago
> I don't know the exact details and motivation, but eventually people seem to have decided that NPN wasn't exactly what they wanted, and they invented a new mechanism called ALPN. Looking at the spec for NPN and knowing about ALPN from implementing just enough of if to enable TLS false start way back when, ALPN is better because it's less steps. With ALPN, the client includes its supported protocols in the Client Hello, and the server picks one or discards the extension in the Server Hello. With NPN, the client indicates it supports NPN in the Client Hello, the server sends back a list of supported protocols in the Server Hello, and the client sends an extra message to indicate its selection between change cipher spec and finished. There's perhaps more client profiling information on the wire with ALPN, but it's not very interesting when 99% of connections just advertise h2,http/1.1. During the development of http/2, you could probably have a pretty tight range on client versions depending on what pre-standard versions were declared.
tadbit 2y ago
LibreSSL removed NPN support seven years ago. https://marc.info/?l=openbsd-announce&m=150996307120987&w=2 https://marc.info/?l=openbsd-announce&m=150996307120987&w=2 I wonder how many memory leaks it'll take for OpenSSL to finally get their act together or for major projects to drop it entirely.
- doublepg23 2y ago
  I feel like a lot of projects did drop OpenSSL post-Heartbleed and then went back to OpenSSL some years later. I know Gentoo did https://bugs.gentoo.org/762847 https://bugs.gentoo.org/762847 Python 3.10 did https://peps.python.org/pep-0644/ https://peps.python.org/pep-0644/ Void Linux did https://voidlinux.org/news/2021/02/OpenSSL.html https://voidlinux.org/news/2021/02/OpenSSL.html Etc. Seems like a lot of that was due to OpenSSL breaking API compat that LibreSSL promised not to break though.
asveikau 2y ago
I could read that in the opposite direction: why were the SPDY people pushing their half-finished protocols into a bunch of upstream projects? Wasn't that irresponsible? Shouldn't they have pushed those same projects to remove support for WIP protocols?
tialaramex 2y ago
It was and remains a stamp collecting library. "Oooh, the Half Cent South African Reverse Yellow Eagle. Only six hundred of these were ever issued and since all postage by that point already cost at least four rand the half cent stamp was entirely useless. Nine in mint condition unperforated were sold at auction in 1956 but none have been seen since" So it's useless? Yes! So why do you want it? Because it's rare! The fact that lots of real world code uses OpenSSL is one of those things it's going to be hard for our descendants to understand. Like explaining that people used to Smoke cigarettes on aeroplanes. But why though, that's crazy. Yes, yes it is.
- znpy 2y ago
  I remember the early days of libressl, they bragged A LOT about removing tens of thousands of lines of code that they deemed useless.
  - touisteur 2y ago
    Heavy use of unifdef there. Following their progress was amazing from an 'let's own this thing now' perspective, seeing the size of code changes, how they decided which things had to go. I would have been bragging too, were I removing ifdefs by the shovelload every day for weeks.
    - sillywalk 2y ago
      The OpenSSL Valhalla Rampage. It was kind of amusing, though horrifying with what they found. https://opensslrampage.org/ https://opensslrampage.org/
olliej 2y ago
Old protocols truly are the gift that keeps on giving (see the numerous protocol and cipher downgrades over the years as well)
extraduder_ire 2y ago
Why does TLS need to know what's being passed through it? I thought the encryption it provides is meant to be transparent.

by hannob 2y ago

lazide 2y ago

Removing old code, especially in open source, has a problem. Unless it is obviously breaking something someone wants to do, it has roughly a zero cost (in maintenance) being left alone, especially since bugs can often be ignored, but a non-zero cost to remove it - as it generally will only garner complaints and whining from any remaining users. And in open source in particular, it’s damn near impossible finding remaining users in advance, or even telling if there are no remaining users.

RetpolineDrama 2y ago
Is it really "zero cost in maintenance" if that old code presents attack surface?
- c0balt 2y ago
  No, it's zero upfront cost in maintenance. Owning code always generates some maintenance/ tech debt but it often is opaque nor easily quantifiable.
  - lazide 2y ago
    IMO the secondary cost is really more ‘friction’ that results when making changes. If no changes get made, then no friction. And open source code can and often is just abandoned. Defacto, sometimes even de jur. User beware, use at your own risk, etc.
    - crngefest 2y ago
      Proprietary code is just as often if not more abandoned - you just don’t notice it
      - lazide 2y ago
        Sure, but proprietary code you can (somewhat) see who is calling it, and the group maintaining it is at least sometimes the group calling it - so has an incentive to not make it a bigger mess. Somewhat. So that part of the equation has less weight. They have a counter balancing thing which is no one can see it to shame them.
- lazide 2y ago
  Does the open source code maintainer have liability for it, or necessarily need to do work in that case? At worst they generally just suffer reputational damage, not actual cost or lost revenue, like a business would.

fullspectrumdev 2y ago

Why it’s not removed: backward compatibility I bet. I have to maintain a few VM’s and statically linked tools specifically for interacting with old network appliances that modern Linux hosts won’t “talk to” without crippling their SSL/TLS configuration or doing other horrendous workarounds like specific OpenSSL/OpenSSH config files for those things. It’s actually an interesting problem when performing security assessments - some tools for scanning will have false negatives because they can’t “talk to” old shit.

jopsen 2y ago
Couldn't you just fallback to HTTP/1.1? I guess it could be that it's not possible.
hannob 2y ago
You have some VMs that talk to old network appliances that require NPN? And if NPN is not supported they don't just fall back to simple HTTP/1.1? I find that sounds extremely unlikely. I am of course well aware of backwards compatibility issues. But I don't see how they'd impact NPN. You might have to update some code that hard-codes NPN functionality, but well, OpenSSL has done API changes in the past, quite significantly so.
- throwway120385 2y ago
  Not OP, but I have some old network-connected appliances in the field that I can't upgrade and that our current OpenSSH clients have deprecated all of the ciphers and methods for. You still have to be able to talk to these things over the network, and ideally you'd want to upgrade at least some of the systems out there. You have to understand that outside of silicon valley and the hobbyist realm there are tons and tons of systems out there where the security posture is "don't allow anyone to have physical or network access to these systems who doesn't belong here" because the companies that made them deprecated them a decade ago or they were built before the widespread practice of supporting the Linux BSP for more than a couple of months. These systems still work and you won't convince very many users of these systems to spend a bunch of money upgrading them for some notional security risk. You may also be working for a company that is contractually obligated to support these systems in some way. So what do you do? In many cases, you do the best you can to mitigate the risks you know are there with tools like firewalls, port knocking, non-standard ports, and so on. And you hope that the 2-3% of attackers that actually know what they're doing never try to crack these things.

toast0 2y ago

> I don't know the exact details and motivation, but eventually people seem to have decided that NPN wasn't exactly what they wanted, and they invented a new mechanism called ALPN. Looking at the spec for NPN and knowing about ALPN from implementing just enough of if to enable TLS false start way back when, ALPN is better because it's less steps. With ALPN, the client includes its supported protocols in the Client Hello, and the server picks one or discards the extension in the Server Hello. With NPN, the client indicates it supports NPN in the Client Hello, the server sends back a list of supported protocols in the Server Hello, and the client sends an extra message to indicate its selection between change cipher spec and finished. There's perhaps more client profiling information on the wire with ALPN, but it's not very interesting when 99% of connections just advertise h2,http/1.1. During the development of http/2, you could probably have a pretty tight range on client versions depending on what pre-standard versions were declared.

tadbit 2y ago

LibreSSL removed NPN support seven years ago. https://marc.info/?l=openbsd-announce&m=150996307120987&w=2 https://marc.info/?l=openbsd-announce&m=150996307120987&w=2 I wonder how many memory leaks it'll take for OpenSSL to finally get their act together or for major projects to drop it entirely.

doublepg23 2y ago
I feel like a lot of projects did drop OpenSSL post-Heartbleed and then went back to OpenSSL some years later. I know Gentoo did https://bugs.gentoo.org/762847 https://bugs.gentoo.org/762847 Python 3.10 did https://peps.python.org/pep-0644/ https://peps.python.org/pep-0644/ Void Linux did https://voidlinux.org/news/2021/02/OpenSSL.html https://voidlinux.org/news/2021/02/OpenSSL.html Etc. Seems like a lot of that was due to OpenSSL breaking API compat that LibreSSL promised not to break though.

asveikau 2y ago

I could read that in the opposite direction: why were the SPDY people pushing their half-finished protocols into a bunch of upstream projects? Wasn't that irresponsible? Shouldn't they have pushed those same projects to remove support for WIP protocols?

tialaramex 2y ago

It was and remains a stamp collecting library. "Oooh, the Half Cent South African Reverse Yellow Eagle. Only six hundred of these were ever issued and since all postage by that point already cost at least four rand the half cent stamp was entirely useless. Nine in mint condition unperforated were sold at auction in 1956 but none have been seen since" So it's useless? Yes! So why do you want it? Because it's rare! The fact that lots of real world code uses OpenSSL is one of those things it's going to be hard for our descendants to understand. Like explaining that people used to Smoke cigarettes on aeroplanes. But why though, that's crazy. Yes, yes it is.

znpy 2y ago
I remember the early days of libressl, they bragged A LOT about removing tens of thousands of lines of code that they deemed useless.
- touisteur 2y ago
  Heavy use of unifdef there. Following their progress was amazing from an 'let's own this thing now' perspective, seeing the size of code changes, how they decided which things had to go. I would have been bragging too, were I removing ifdefs by the shovelload every day for weeks.
  - sillywalk 2y ago
    The OpenSSL Valhalla Rampage. It was kind of amusing, though horrifying with what they found. https://opensslrampage.org/ https://opensslrampage.org/

olliej 2y ago

Old protocols truly are the gift that keeps on giving (see the numerous protocol and cipher downgrades over the years as well)

extraduder_ire 2y ago

Why does TLS need to know what's being passed through it? I thought the encryption it provides is meant to be transparent.