3 ms·

My take away from this is that people are still far too blasé about introducing hard dependencies and complexity, even after the left-pad incident as a warning.

My take away from this is that people are still far too blasé about introducing hard dependencies and complexity, even after the left-pad incident as a warning. OpenSSH is a massive wall of code. Such complex systems are inherently untrustworthy to me, no matter what language they might be written in. Even with ernest devs there are still more opportunities for mistakes.

lrvick 3y ago
Agreed, which is why IMO ssh in production is a terrible idea that reveals even worse problems. A server with ssh generally implies there is a shell, and cli tools, and an administration model that involves humans connecting to servers to manually update them in place like pets. Having a full workstation-optimized distro like ubuntu or debian with hundreds of packages constantly shifting and updating as a critical production server is wildly high risk in terms of both reliability and security. I understand this is how most sysadmins were taught but it is a 70s unix mainframe mindset from a time before security was a thing and everyone on the internet was a good actor. Only a workstation or dev server should have tools for humans like ssh installed. Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. No production systems I launch today even have xz installed, or a package manager, or a shell.
- ris 3y ago
  > Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. And then you need to debug something. What do?
  - nuc1e0n 3y ago
    Have good logging and virtual machines that can replay those logs offline?
    - lrvick 3y ago
      One of many viable methods depending on the application, yes.
  - lrvick 3y ago
    Debug it with a binary with debugging enabled on a debug dev system. Just like you would any other immutable firmware appliance. You could also in some cases have a debug container you pull in on demand, say if the host OS is an appliance-style k8s runtime like TalosOS.
    - ris 3y ago
      Except the debug environment isn't _quite_ like the production environment, is it? Now you've got it working on the debug environment but weirdly broken on prod still..
      - lrvick 3y ago
        They should be identical except for any debug tools. In fact they should ideally share the exact same root filesystem bit-for-bit, but then you can overlay mount debug tools in place in a dev/staging instance when required. E.g. if you are running an immutable k8s runtime distro like TalosOS or Metropolis, you can choose to pull and mount a debug container with strace and gdb into the same process namespace of a problem pod, and when you are done all those debugging tools evaporate. System mutability is why most bugs happen in the first place. There should be no tools or services in prod except those required to run or monitor prod at runtime.
noobermin 3y ago
This wasn't a vulnerability in openssh though, this was from systemd, where I think your point in fact is stronger.
- ninkendo 3y ago
  It’s not a vulnerability in systemd either, it’s a vulnerability in xz, which libsystemd depends on, and libsystemd is only integrated into OpenSSH in certain distributions, and only to avoid writing ~5 lines of C code to write the string “READY=1” to a socket specified in an environment variable. Systemd itself doesn’t even use lzma/xz… libsystemd does, and that’s a library meant to help other software integrate with systemd. It’s not really the same thing.

lrvick 3y ago

Agreed, which is why IMO ssh in production is a terrible idea that reveals even worse problems. A server with ssh generally implies there is a shell, and cli tools, and an administration model that involves humans connecting to servers to manually update them in place like pets. Having a full workstation-optimized distro like ubuntu or debian with hundreds of packages constantly shifting and updating as a critical production server is wildly high risk in terms of both reliability and security. I understand this is how most sysadmins were taught but it is a 70s unix mainframe mindset from a time before security was a thing and everyone on the internet was a good actor. Only a workstation or dev server should have tools for humans like ssh installed. Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. No production systems I launch today even have xz installed, or a package manager, or a shell.

ris 3y ago
> Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. And then you need to debug something. What do?
- nuc1e0n 3y ago
  Have good logging and virtual machines that can replay those logs offline?
  - lrvick 3y ago
    One of many viable methods depending on the application, yes.
- lrvick 3y ago
  Debug it with a binary with debugging enabled on a debug dev system. Just like you would any other immutable firmware appliance. You could also in some cases have a debug container you pull in on demand, say if the host OS is an appliance-style k8s runtime like TalosOS.
  - ris 3y ago
    Except the debug environment isn't _quite_ like the production environment, is it? Now you've got it working on the debug environment but weirdly broken on prod still..
    - lrvick 3y ago
      They should be identical except for any debug tools. In fact they should ideally share the exact same root filesystem bit-for-bit, but then you can overlay mount debug tools in place in a dev/staging instance when required. E.g. if you are running an immutable k8s runtime distro like TalosOS or Metropolis, you can choose to pull and mount a debug container with strace and gdb into the same process namespace of a problem pod, and when you are done all those debugging tools evaporate. System mutability is why most bugs happen in the first place. There should be no tools or services in prod except those required to run or monitor prod at runtime.

noobermin 3y ago

This wasn't a vulnerability in openssh though, this was from systemd, where I think your point in fact is stronger.

ninkendo 3y ago
It’s not a vulnerability in systemd either, it’s a vulnerability in xz, which libsystemd depends on, and libsystemd is only integrated into OpenSSH in certain distributions, and only to avoid writing ~5 lines of C code to write the string “READY=1” to a socket specified in an environment variable. Systemd itself doesn’t even use lzma/xz… libsystemd does, and that’s a library meant to help other software integrate with systemd. It’s not really the same thing.