4 ms·
> Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim in
by ris 3y ago
> Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust.
And then you need to debug something. What do?
- nuc1e0n 3y agoHave good logging and virtual machines that can replay those logs offline?
- lrvick 3y agoOne of many viable methods depending on the application, yes.
- lrvick 3y agoDebug it with a binary with debugging enabled on a debug dev system. Just like you would any other immutable firmware appliance. You could also in some cases have a debug container you pull in on demand, say if the host OS is an appliance-style k8s runtime like TalosOS.
- ris 3y agoExcept the debug environment isn't _quite_ like the production environment, is it? Now you've got it working on the debug environment but weirdly broken on prod still..
- lrvick 3y agoThey should be identical except for any debug tools. In fact they should ideally share the exact same root filesystem bit-for-bit, but then you can overlay mount debug tools in place in a dev/staging instance when required. E.g. if you are running an immutable k8s runtime distro like TalosOS or Metropolis, you can choose to pull and mount a debug container with strace and gdb into the same process namespace of a problem pod, and when you are done all those debugging tools evaporate. System mutability is why most bugs happen in the first place. There should be no tools or services in prod except those required to run or monitor prod at runtime.