4 ms·

> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `s

by akx 4y ago

> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.

capableweb 4y ago
Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
- jacquesm 4y ago
  That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
- dspillett 4y ago
  If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
  - bravetraveler 4y ago
    Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem. There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.
- gtirloni 4y ago
  Sorry, what? You don't lose complete control of a server because it's rebooting nonstop.
- markstos 4y ago
  AWS at least provides serial console access so have the option to access it during the boot cycle. Alternatively, you umount the drive, attach it to another machine, chroot into it, fix grub or whatever, reverse the process and boot again. It's a few steps, but can be done in a few minutes with practice.
- bravetraveler 4y ago
  Out of band management is common and highly recommended
- mrintegrity 4y ago
  Actually networking and ssh come up for a couple of seconds before containerd triggers the kernel panic so you can fix it by doing this: while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done While rebooting the system

by akx 4y ago

capableweb 4y ago

Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.

jacquesm 4y ago
That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
dspillett 4y ago
If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
- bravetraveler 4y ago
  Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem. There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.
gtirloni 4y ago
Sorry, what? You don't lose complete control of a server because it's rebooting nonstop.
markstos 4y ago
AWS at least provides serial console access so have the option to access it during the boot cycle. Alternatively, you umount the drive, attach it to another machine, chroot into it, fix grub or whatever, reverse the process and boot again. It's a few steps, but can be done in a few minutes with practice.
bravetraveler 4y ago
Out of band management is common and highly recommended
mrintegrity 4y ago
Actually networking and ssh come up for a couple of seconds before containerd triggers the kernel panic so you can fix it by doing this: while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done While rebooting the system