4 ms·

Wouldn't rollback of kernel be a choice in grub menu? It's pretty standard for all distros to have that choice.

• withinboredom 4y ago
  That usually requires physical access to the server to select it during boot.
  • taspeotis 4y ago
    Or a real server with Lights Out Management.
    • withinboredom 4y ago
      That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
      • phillu 4y ago
        That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.
        • withinboredom 4y ago
          My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?
          • laumars 4y ago
            Serial connections will only cost you a Raspberry Pi (there's probably some really cheap console servers on eBay too). I don't think the issue is so much cost but more this kind of systems administration is becoming a forgotten art because 99% of the time modern tooling removes the need for it. So younger sysadmins are never taught how to do these kinds things. However when I started out, I worked in a few small companies that had their physical hosts connected to a console server (which was a Cisco device like a network switch) via serial cables and you'd then connect to that console server remotely.
          • topranks 4y ago
            Depends on the infra and how it’s set up. If you can afford to have something down for an extended period then fine. But even with a small team some services are built such that certain device outages cannot be tolerated, at least for an extended period. So out-of-band/console servers or whatever still make a lot of sense and a relatively high priority.
      • jacquesm 4y ago
        You can do this kind of thing across the network if you have to.
      • corobo 4y ago
        I'd be more inclined to say that physical servers usually have some sort of console access available. I'm not sure I've ever worked with any (2008-present) that don't in any case.
  • akx 4y ago
    Not necessarily. With good timing and some luck, you can connect the serial/"recovery" console before GRUB's timeout ends and either change the running kernel or add the `systemd.mask=docker.service` boot parameter to prevent Docker from starting.
    • withinboredom 4y ago
      Sounds like a VM and not a physical server.
      • akx 4y ago
        Sure. For a physical server, you'd use its lights-out management to the same effect.
      • laumars 4y ago
        Nope. Back before VMs were thing it was common to do "lights out" style remote management via a console server. That console server would then have a serial connection (the old 9 pin d-sub plug[1]) to your individual physical servers. You could then connect to your remote servers local TTY via the console server a little like jumping to remote servers via an SSH bastion. However it did sometimes require a little bit of prior configuration, depending on your distro[2]. This wasn't just limited to Linux either. It was a common UNIX trick :) This is a bit of a lost art these days though. iLo, IPMI have replaced the need for serial. Then virtualisation and, to a lesser extent, containerisation have lowered the bar even further plus also moving the industry towards more ephemeral systems that can be destroyed and rebuilt automatically rather than the old habits of nursing failed hosts back to health. [1] https://duckduckgo.com/?q=9+pin+d-sub+plug&t=newext&atb=v316-1&iar=images&iax=images&ia=images https://duckduckgo.com/?q=9+pin+d-sub+plug&t=newext&atb=v316... [2] https://www.kernel.org/doc/html/v5.3/admin-guide/serial-console.html https://www.kernel.org/doc/html/v5.3/admin-guide/serial-cons... (a lot of distros at the time did ship a kernel with this support compiled in. I don't know how common it is now).
        • jacquesm 4y ago
          And quite a few implementations actually emulate the serial console allowing for the exact same access. (Serial Over Lan or SOL for short.)
  • ape4 4y ago
    If its in the cloud you'd have a virtual console.
    • zurn 4y ago
      Unsurprisingly AWS is ghetto about this.
  • hansel_der 4y ago
    no. it requires acces to the serial console or baseband management controller or whatever terms have emerged. have never rented a physical server w/o this.
  • darkwater 4y ago
    If you have unattended-upgrade and automatic reboot in the cloud to benefit from security updates for long-lived instances, then you better make sure to have a tty console attached to it. You are treating it like a physical machine, you must have the same tooling around.
  • bravetraveler 4y ago
    Not really, console access through IPMI found on most servers Exceptions tend to be white boxes built with desktop components, at which point, yea. The proverbial You asked for this problem