4 ms·

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ra

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that address - or a filesystem path, as the case may be. Meaning that the command (under root) overwrote /bin/bash with 0 bytes. Within minutes, jobs started failing, the machine couldn't be SSH'd into, but funnily enough, as /bin/bash was the login shell for all users, not even logging in via a tty through KVM worked. Perhaps not the best story, but certainly a fun way to blow your foot off on a Monday morning :)

soneil 5y ago
That's beautiful. I'm not sure I'd have had a clue what just happened even if it was me making the typo.
hamburglar 5y ago
Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”
- linsomniac 5y ago
  Probably not if it was overwritten (": >/bin/bash") rather than removed and recreated ("rm -f /bin/bash; : >/bin/bash"). The former will cause all processes to see the empty file, the latter would leave processes with access to the old contents. In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you could pull a copy from another machine or backups.
  - fulafel 5y ago
    In the wsgi case of replacing a regular file with an AF_UNIX socket, the destination would have needed to to be unlinked first - the bind() call will otherwise fail.
  - cyphar 5y ago
    On Linux you can't overwrite a binary that is currently being executed (you'll -ETXTBSY when you try to open it for writing, and truncate will also fail).
  - mijoharas 5y ago
    I was unaware of the first part of this, and decided to find a reference. This stack overflow answer[0] provides some detail on ETXTBUSY and the fact that writing directly writing to a running executable file will fail. (I had thought it would unlink the file like in the second case, simply because that's what I always see when recompiling something that's already running. This obviously uses the second method though). [0] https://unix.stackexchange.com/questions/187931/modifying-binary-during-execution https://unix.stackexchange.com/questions/187931/modifying-bi...
jhugo 5y ago
`ssh $host /bin/sh` (or another shell) should work?
- bch 5y ago
  On a Linux box isn’t that just a link to bash?
  - yjftsjthsd-h 5y ago
    It depends on the distro; on RHEL yes it is BASH, but Debian uses dash, and Alpine uses busybox sh.
- dotancohen 5y ago
  Logging into another shell would be attempted only if someone knew at the time why the logins were failing. But thanks, I've just added another technique to my toolbox.
- thayne 5y ago
  Depending on the distribution, /bin/sh might be a symlink to bash
- hddqsb 5y ago
  That won't work because sshd runs the command using the user's login shell. From https://man.openbsd.org/sshd#LOGIN_PROCESS https://man.openbsd.org/sshd#LOGIN_PROCESS: > When a user successfully logs in, sshd does the following: > ... > 9. Runs user's shell or command. All commands are run under the user's login shell as specified in the system password database.
bentcorner 5y ago
Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.
- lights0123 5y ago
  Recovery disk, then either copy the disk's copy of bash (if it doesn't depend on a later glibc version), copy another shell to /bin/bash (as the system probably doesn't depend on bash-specific commands to boot), chroot and use the package manager, or use the package manager with an explicit sysroot (e.g. pacman --sysroot). The first two steps are very easy compared to the latter two, but should be followed by a reinstallation of the package that provides bash.
  - r00f 5y ago
    i wonder, wouldn't copy over sftp work? i don't think it depends on shell
  - andrewstuart2 5y ago
    You should also usually be able to just reboot and as long as another shell is installed, choose that shell as your startup command in the bootloader.

soneil 5y ago

That's beautiful. I'm not sure I'd have had a clue what just happened even if it was me making the typo.

hamburglar 5y ago

Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”

linsomniac 5y ago
Probably not if it was overwritten (": >/bin/bash") rather than removed and recreated ("rm -f /bin/bash; : >/bin/bash"). The former will cause all processes to see the empty file, the latter would leave processes with access to the old contents. In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you could pull a copy from another machine or backups.
- fulafel 5y ago
  In the wsgi case of replacing a regular file with an AF_UNIX socket, the destination would have needed to to be unlinked first - the bind() call will otherwise fail.
- cyphar 5y ago
  On Linux you can't overwrite a binary that is currently being executed (you'll -ETXTBSY when you try to open it for writing, and truncate will also fail).
- mijoharas 5y ago
  I was unaware of the first part of this, and decided to find a reference. This stack overflow answer[0] provides some detail on ETXTBUSY and the fact that writing directly writing to a running executable file will fail. (I had thought it would unlink the file like in the second case, simply because that's what I always see when recompiling something that's already running. This obviously uses the second method though). [0] https://unix.stackexchange.com/questions/187931/modifying-binary-during-execution https://unix.stackexchange.com/questions/187931/modifying-bi...

jhugo 5y ago

`ssh $host /bin/sh` (or another shell) should work?

bch 5y ago
On a Linux box isn’t that just a link to bash?
- yjftsjthsd-h 5y ago
  It depends on the distro; on RHEL yes it is BASH, but Debian uses dash, and Alpine uses busybox sh.
dotancohen 5y ago
Logging into another shell would be attempted only if someone knew at the time why the logins were failing. But thanks, I've just added another technique to my toolbox.
thayne 5y ago
Depending on the distribution, /bin/sh might be a symlink to bash
hddqsb 5y ago
That won't work because sshd runs the command using the user's login shell. From https://man.openbsd.org/sshd#LOGIN_PROCESS https://man.openbsd.org/sshd#LOGIN_PROCESS: > When a user successfully logs in, sshd does the following: > ... > 9. Runs user's shell or command. All commands are run under the user's login shell as specified in the system password database.

bentcorner 5y ago

Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.

lights0123 5y ago
Recovery disk, then either copy the disk's copy of bash (if it doesn't depend on a later glibc version), copy another shell to /bin/bash (as the system probably doesn't depend on bash-specific commands to boot), chroot and use the package manager, or use the package manager with an explicit sysroot (e.g. pacman --sysroot). The first two steps are very easy compared to the latter two, but should be followed by a reinstallation of the package that provides bash.
- r00f 5y ago
  i wonder, wouldn't copy over sftp work? i don't think it depends on shell
- andrewstuart2 5y ago
  You should also usually be able to just reboot and as long as another shell is installed, choose that shell as your startup command in the bootloader.