6 ms·

When you deleted /lib on Linux while still connected via SSH (2022)

191 points by todsacerdoti 2y ago

smw 2y ago
I had someone do this on an important production Solaris machine once, many years ago. Luckily they just moved /lib instead of deleting it -- and on Solaris (some of?) the binaries in /sbin were statically linked, including ln. Hard linking /lib back to the correct path was enough to recover.
- ryao 2y ago
  Do you mean moving it? You cannot make a hard link for a directory. That is disallowed on Unix to prevent the creation of orphaned subtrees (i.e. ln a directory into itself and then rmdir the original link).
  - yjftsjthsd-h 2y ago
    GNU ln has this ( https://man.archlinux.org/man/ln.1#d https://man.archlinux.org/man/ln.1#d ): -d, -F, --directory allow the superuser to attempt to hard link directories (this will probably fail due to system restrictions, even for the superuser) which implies that that's not quite an absolute limit. I don't see any comment either way on https://illumos.org/man/1/ln https://illumos.org/man/1/ln , but it's plausible that some version of Solaris had wiggle room; it's a terrible idea for obvious reasons, but there's really no hard technical reason why a system couldn't allow you to create hard links to directories.
    - ryao 2y ago
      The Solaris VFS is flexible enough for a filesystem to support this, but any filesystem that does violates POSIX according to a comment in ZFS. A check of the illumos UFS driver source code reveals that it violates POSIX by permitting root to do this. I wrote more about this with links to relevant illumos source code here: https://news.ycombinator.com/item?id=43447288 https://news.ycombinator.com/item?id=43447288
  - wizzwizz4 2y ago
    There are two checks prohibiting it: • Non-root users aren't allowed to make directory hard links. • Many versions of the userspace program `ln` don't let you do it. But the `link` system call can, at least on Solaris, if called by user 0. (Not sure about Linux: I tried it once and it didn't work, but I was doing weird things with FUSE and also trying to name the link `..`, so I don't know why it failed.)
    - deleted 2y ago
      [deleted]
    - ryao 2y ago
      It is more nuanced than that. First, the Solaris 11.4 documentation says that this is not allowed: https://docs.oracle.com/cd/E88353_01/html/E72487/link-8.html https://docs.oracle.com/cd/E88353_01/html/E72487/link-8.html The illumos man page is less clear: https://illumos.org/man/8/link https://illumos.org/man/8/link The illumos ZFS driver makes it very clear that this is not allowed under POSIX in a comment and explicitly disallows it in zfs_link(): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/zfs_vnops.c#L4476 https://github.com/illumos/illumos-gate/blob/master/usr/src/... However, It appears that the illumos UFS driver supports this: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/ufs/ufs_vnops.c#L3195 https://github.com/illumos/illumos-gate/blob/master/usr/src/... Presumably, the Solaris 10 UFS driver also supports it (or supported it in older versions of Solaris 10). Given that someone at Oracle likely modified the Solaris man page to differ from the older OpenSolaris man page in illumos, I would expect recent versions of Solaris to disallow this on UFS, but someone would need to check. That said, I have to recant my previous comment. smw likely linked the directory, which is insane, but would have worked on older Solaris versions if we assume the modern illumos UFS driver is unchanged in this regard.
jmclnx 2y ago
Interesting, at one time ages ago I thought Linux came with a static linked bash in /bin or maybe /sbin for this type of problem. Just checked Slackware and no more, I wonder if that is a casualty of the /bin /usr/bin merge ?
- ryao 2y ago
  Gentoo used to include a static busybox binary for recovery purposes, but it was removed from the system set years ago. Now you are expected to use the initramfs for recovery and can install bust box yourself if you are concerned. In fairness, this particular issue would be made worse by rebooting since the unlinked anonymous library files would be garbage collected, which is bad unless you have a filesystem that has snapshots like ZFS and had used them prior to this. The article author did not try to recover libraries from the anonymous files, which is probably good considering that only a subset would have been in use and thus only that subset would be recoverable from the anonymous files (unless there are filesystem snapshots).
- pengaru 2y ago
  once upon a time everything in /sbin was statically linked then only `sln` was statically linked (a variant of `ln`) today most distros statically link nothing and you're up shit's creek in this situation
ryao 2y ago
The technical term is unlinked. The files in use by running processes are still anonymous files in the filesystem. They will not be garbage collected until the last program that mmap’ed them is gone. If you could find the inode number from /proc (possibly from /proc/$PID/maps), you should be able to use a filesystem specific tool to retrieve them, such as debugfs or zdb.
- abound 2y ago
  I think the tricky piece is that you'd need to find that inode number without using any dynamically linked libraries, including ls and cat in the author's case. And debugfs is likely dynamically linked too (I just checked and it is on my machine)
  - ryao 2y ago
    Presumably, the read shell builtin can be used to read files in /proc. As per the original article, you can get a static version over the network using only bash builtins and overwrite a file with execute permission to be able to execute it.
eitland 2y ago
Something similar happened a place I worked somewhere between 10 and 20 years ago. Service technician did not see the . in the command rm -rf ./bin so he proceeded to run rm -rf /bin When that didn't work he did what everyone who knows a little bit Linux does and added sudo in front. I was on a terminal from the other side of the globe when the server suddenly started acting weird. We were able to use scp or rsync (one of then was in sbin or something) to get back the bits from an identical server, which saved me from three days of tedious work :-) In hindsight of course we should have written the docs in a way that would prevent this exact situation but in the beginning it was just the output of the history command after I had done it, dumped into a document with some explanations.
- ryao 2y ago
  It is not the same thing, but recently, my pfsense router started acting weird where dns stopped working. I was bust so I rebooted it and it failed to boot. I ended up bringing a monitor and keyboard to see what went wrong and it turned out that suricata’s logs had used 1.4TB on my 250GB SSD (ZFS zstd compression is awesome), causing it to run out of space a few years after I enabled the feature out of curiously. I wiped the logs, rebooted and things worked. My lesson from them is when a machine servers acting strangely, do not reboot and instead troubleshoot right away. Had I done that, I could have found the problem and had only minimal partial downtime (I would have had to restart dns afterward). Since I was too busy to do things the right way, my internet was out for a half hour. Your story reminded me of this, since you would have had a bigger headache if you had rebooted.
  - hinkley 2y ago
    This is the origin story for anyone who has ever set up alerts for 90% disk utilization on a machine. These are especially bad with services that generally grow their logs very slowly but when something like a net split happens or a server is down they generate as much log data in an hour as they typically do all week. So you get close to full and then an incident happens, and now you have two incidents because the screaming machine goes down with a full disk right after you lose your internal DNS server or what have you.
nunez 2y ago
Super interesting, though it feels like a response to an interview question!
Dwedit 2y ago
If you deleted /lib, you'd probably be better off reinstalling packages while booting off of USB or something. You're gonna have downtime because programs won't work correctly.
- LorenDB 2y ago
  I also had to wonder why not just liveboot from USB or attach the affected boot medium to another system, then use the recovery system's fully working tools to just relink the /lib folders?
  - ideasphere 2y ago
    “This is not a very important machine, and I could have just reimaged the MicroSD card and be done with it, but I was curious if I could recover from the error.”
nullorempty 2y ago
At the start of my career I removed the `x` attribute from all files :)
- ivanjermakov 2y ago
  chmod -R is too convenient lol
- ryao 2y ago
  Of all of the stories I have read here, this is the first to make me laugh. Congratulations. :)
  - nullorempty 2y ago
    ... we had an amazing sysadmin. He had a shell open on that box when I came to tell the news. He started to type quickly and thoughtfully, trying utilities I haven't even heard of. Then, he echoed a small C program that was supposed to set `x` on the chmod. Quickly he typed `cc x.c` to compile... and then it dawned on him.
    - ryao 2y ago
      Perhaps scp -p or sshfs would have worked. This is just a guess. I am not certain.
inejge 2y ago
The heroic version of the story is now almost 40 years old[1]. (One HN mention with the link to a HTMLized version is here[2].) In both cases, the upshot is that as long as you have a running shell with root privileges, at least one existing executable file on the filesystem, and the means to overwrite that file with arbitrary binary content, you can write a small program which can recreate a skeleton system structure and dig yourself out of the hole. The reason why this keeps happening is that in regular UNIX root is omnipotent and the filesystem is ultimately unprotected. Immutable systems and restricted execution environments may make this a thing of the past. [1] https://www.wolczko.com/rm.txt https://www.wolczko.com/rm.txt [2] https://news.ycombinator.com/item?id=7892471 https://news.ycombinator.com/item?id=7892471
lloeki 2y ago
A long long time ago the team I was part of managed old unix systems. A coworker telnet'd (or rsh, can't recall) into such a machine to do some maintenance and after a while fat fingered: umount / Would you believe it, back then being root meant this was absolutely unprotected and the minicomputer OS (some ancient AIX) dutifully complied. The chaos that ensued is but a blur.
- teaearlgraycold 2y ago
  Did you just restart?
  - WesolyKubeczek 2y ago
    Wasn’t restarting those AIX dinosaurs a nontrivial thing?
    - teaearlgraycold 2y ago
      I could believe it. I’m just hoping OP can provide more info.
  - dingaling 2y ago
    Using what command, though?
    - teaearlgraycold 2y ago
      Perhaps there's a restart button?
deleted 2y ago
[deleted]
gleenn 2y ago
One time I was flipping back and forth between directories, compiling some code, then checking it, then rm -rf'ing it. I accidentally hit up an cd ..'d one too many times. Suddenly the rm command hung and I was confused because it should be nearly instant as it was a few files. I stared in horror as I was accidentally deleting everything in mybhome directory. Luckily back then I had a .pr0n directory with a significant amount of content. A few things were lost but that .pr0n folder was luckily early enough in the list and big enough to slow down the deletion of my photos and documents. That's why I always recommend having a big "buffer" of video content for such situations, ya know, for data integrity ;)
- hinkley 2y ago
  Unless you were using tcsh you had to rewrite your shell configuration though.
- OKRainbowKid 2y ago
  Great idea, I shall see to it right away.
- winwang 2y ago
  That's wild and inspirational. Like the scene in Rush Hour when a stack of bills saves Tucker from a bullet. Though I'm moreso tempted to just create a `.1111aaaa-antidumb` directory and store my caches and backups there. This has also un-inspired me from creating a fast `rm`-esque utility.
  - woleium 2y ago
    You could of course alias rm -rf to rm -rf -i
    - marc_abonce 2y ago
      I always use trash-cli and alias rm to 'echo NO! #'. Only if the file is too big to fit into the garbage bin, I can unalias rm, rm the thing and then reset the alias immediately after.
o11c 2y ago
> Please note that busybox can’t function with a name that is not a busybox applet name. This is somewhat wrong. From ksh/bash/zsh, you can run: (exec -a someappname /arbitraryexecutablepath args...) This won't work on most ash derivatives (including /bin/sh on Debian, FreeBSD, or NetBSD), but does work on busybox ash. The parentheses prevent the `exec` from actually replacing your current shell, which might be less important for emergency rescues, but which otherwise is often what you want with `exec -a`.
nurple 2y ago
My workstation seems fine: $ ls -R /{lib,usr,bin,sbin} ls: cannot access '/sbin': No such file or directory /bin: sh /lib: ld-linux.so.2 /usr: bin /usr/bin: env Oh right... $ ls -l /usr/bin/env lrwxrwxrwx 1 root root 65 Mar 21 23:39 /usr/bin/env -> /nix/store/9m68vvhnsq5cpkskphgw84ikl9m6wjwp-coreutils-9.5/bin/env $ ldd /usr/bin/env linux-vdso.so.1 (0x00007ffff7fc4000) libacl.so.1 => /nix/store/dyizbk50iglbibrbwbgw2mhgskwb6ham-acl-2.3.2/lib/libacl.so.1 (0x00007ffff7fb3000) libattr.so.1 => /nix/store/vlgwyb076hkz7yv96sjnj9msb1jn1ggz-attr-2.5.2/lib/libattr.so.1 (0x00007ffff7fab000) libgmp.so.10 => /nix/store/dsxb6qvi21bzy21c98kb71wfbdj4lmz7-gmp-with-cxx-6.3.0/lib/libgmp.so.10 (0x00007ffff7f06000) libc.so.6 => /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/libc.so.6 (0x00007ffff7d0e000) /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/ld-linux-x86-64.so.2 => /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fc6000)
- MadnessASAP 2y ago
  Don't have to worry about trashing FHS if your OS doesn't use FHS :-P
zavec 2y ago
I actually started a short blog series about a similar problem where a friend had blown away /bin and a bunch of other stuff, but/lib was still there. Unfortunately it didn't end up getting anywhere because even though I was able to drop executables on the machine with echo and make them executable with a .so from lib I wasn't able to get back to root permissions as sudo and everything had been blown away and I didn't think I'd have great luck trying to find a zero-day in the kernel. It was still a lot of fun though.
Fizzadar 2y ago
I once nuked the entire OS partition on an openvz host. Vz data was still good so we ended up copying the root fs from another similar box, manually updated the network config and it ran for another 4 years until retired.
throwanem 2y ago
"When you discover yourself to be in a hole, the best first thing to do is stop digging." Some of the best early professional advice I ever received was, in moments like these, to keep my hands off the keyboard for at least a timed minute.
ofalkaed 2y ago
Many years ago I wrote a backup script where I did "rm -rf /etc/" instead "cp -r /etc/ /mnt/whatever," not sure how I managed that. Took ages to figure out what was causing /etc/ to disappear since /etc/ going missing often went unnoticed for awhile and I was running Arch back in the days when running pacman -syu always caused exciting things to happen. I even did a complete backup and reinstall trying to figure that one out, and I was extra cautious about making a backup of that backup script which I had spent so much time on and was fairly proud of since it was my first non-trivial bash script. I also once did "rm -rf /", was deleting a dir which started with a "[" and accidentally hit "enter" instead of "\." That one taught me the dangers of absolute paths. Edit: That last one is not quite right, would not have been an absolute path issue, that dir must have ended up in root somehow, can't quite remember the details, been too long.
- Vilian 2y ago
  Isn't absolute path better practice than relative paths?
  - ofalkaed 2y ago
    In scripts, yes, but they are not without their dangers and can be especially troublesome when working in the terminal since almost no one types out and checks the full paths constantly, they just let tab completion take care of it and assume it worked. In scripts things like ../../../../file are a pain to read and assume everything in the script before all those previous dirs worked as it should have and everything is where it should be. Cd to an incorrect absolute path produces an error code so we can be sure we are in the proper dir, cd ../../../ will never produce an error and always succeeds even if you are at root.
    - 3np 2y ago
      > almost no one types out and checks the full paths constantly, they just let tab completion take care of it and assume it worked. Are you a bootcamp coach having watched countless of people at the terminal? What makes you assume this is a general habit?
chatmasta 2y ago
I’ve seen this same story in exploit writeups - it’s funny to see “living off the land” techniques used non-maliciously.
mastax 2y ago
chattr +i /lib/ld-linux.so.2 Sounds tempting.
userbinator 2y ago
This could be considered a variant of the "bootstrapping" problem.
asah 2y ago
This happened to me once. I caught my breath, found an identical server and copied the files over using uuencode and cat, until I had enough working to create a second terminal window and could use better tools to verify everything. 5 minutes that felt a lot longer.
- doubled112 2y ago
  Sounds similar to a time I accidentally removed a bunch of system libraries with pip. I wish I would have had a similar server to recover from. Yum being a Python program made recovery tedious, since some of the libraries it depended on were installed via RPM and gone. Not a difficult recovery, but tedious.
conceptme 2y ago
A long time ago I started working at this company and I fat fingered 'rm -rf /' instead with a dot. When realizing what I had done and it was taking too long I powered off the machine. When I told the sysops he looked horrified and asked me how far it got. The fun thing was all websites and data where mounted from the server to each workstation to make it easy to update source code.
nasretdinov 2y ago
I think I now know why on macOS "Applications" (with capital A) comes first in the list of root-level directories. I've been showing to my colleague that on modern systems "sudo rm -rf /" does nothing, however that turned out to be GNU-specific thing, and macOS of course uses BSD tools. So I've started noticing that something is off very quickly and stopped the command. On HFS+ (probably on APFS too) the file list is always sorted, so it only deleted some of my Adobe apps and stuff like Addressbook, so I didn't even notice anything at first :)
vessenes 2y ago
I “rm -rf /“ ed my Linux box in 1996 and realized what I had done about 30 seconds in. It was through like a-d in /bin. ‘Recovery’ wasn’t a strong concept in Linux os installers then, and I wasn’t sure my home directory would survive a reinstall, (not on a separate partition) and I didn’t think I’d be able to reboot successfully in any event. My brother ran the same distribution but at a school across the country; I was able to recover by carefully pulling what I needed down with ftp to his dorm computers IP. I’m not sure scp even existed then on Linux; I guess maybe we could have used netcat in a pinch. Well, this was already a pinch. In a real pickle. This was one of the first moments where high bandwidth connections to endpoints really saved me / impacted me, and I haven’t forgotten. For many years after university this kind of direct high bandwidth connection got much harder to achieve, first because we were back in a low bandwidth residential world, then because ipv4 was mostly denied to consumers, then because of nat. Today this would again be achievable, but with vastly more complexity. For a home computer rescue you’d want Tailscale in both sides. And it’s extremely unlikely you’d be using the same distribution as your sib, much less have the same libc linking. And god help you if you had to restore your systemd directory by hand.
udev4096 2y ago
Reminds me of this: https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123 https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...
scripturial 2y ago
‘rm —rf’ should have default behavior to prompt “are you sure” when deleting top level folders. People are literally talking about secret video files saving their files from being accidentally destroyed. How in 2025 do we not have these types of innovations? Are people afraid of breaking old scripts? (macOS saves us from a lot of this stupidity, Linux should have something similar. I would love a way to mark folders as “permanent” or as “restore on reboot”)

191 points by todsacerdoti 2y ago

smw 2y ago

I had someone do this on an important production Solaris machine once, many years ago. Luckily they just moved /lib instead of deleting it -- and on Solaris (some of?) the binaries in /sbin were statically linked, including ln. Hard linking /lib back to the correct path was enough to recover.

ryao 2y ago
Do you mean moving it? You cannot make a hard link for a directory. That is disallowed on Unix to prevent the creation of orphaned subtrees (i.e. ln a directory into itself and then rmdir the original link).
- yjftsjthsd-h 2y ago
  GNU ln has this ( https://man.archlinux.org/man/ln.1#d https://man.archlinux.org/man/ln.1#d ): -d, -F, --directory allow the superuser to attempt to hard link directories (this will probably fail due to system restrictions, even for the superuser) which implies that that's not quite an absolute limit. I don't see any comment either way on https://illumos.org/man/1/ln https://illumos.org/man/1/ln , but it's plausible that some version of Solaris had wiggle room; it's a terrible idea for obvious reasons, but there's really no hard technical reason why a system couldn't allow you to create hard links to directories.
  - ryao 2y ago
    The Solaris VFS is flexible enough for a filesystem to support this, but any filesystem that does violates POSIX according to a comment in ZFS. A check of the illumos UFS driver source code reveals that it violates POSIX by permitting root to do this. I wrote more about this with links to relevant illumos source code here: https://news.ycombinator.com/item?id=43447288 https://news.ycombinator.com/item?id=43447288
- wizzwizz4 2y ago
  There are two checks prohibiting it: • Non-root users aren't allowed to make directory hard links. • Many versions of the userspace program `ln` don't let you do it. But the `link` system call can, at least on Solaris, if called by user 0. (Not sure about Linux: I tried it once and it didn't work, but I was doing weird things with FUSE and also trying to name the link `..`, so I don't know why it failed.)
  - deleted 2y ago
    [deleted]
  - ryao 2y ago
    It is more nuanced than that. First, the Solaris 11.4 documentation says that this is not allowed: https://docs.oracle.com/cd/E88353_01/html/E72487/link-8.html https://docs.oracle.com/cd/E88353_01/html/E72487/link-8.html The illumos man page is less clear: https://illumos.org/man/8/link https://illumos.org/man/8/link The illumos ZFS driver makes it very clear that this is not allowed under POSIX in a comment and explicitly disallows it in zfs_link(): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/zfs_vnops.c#L4476 https://github.com/illumos/illumos-gate/blob/master/usr/src/... However, It appears that the illumos UFS driver supports this: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/ufs/ufs_vnops.c#L3195 https://github.com/illumos/illumos-gate/blob/master/usr/src/... Presumably, the Solaris 10 UFS driver also supports it (or supported it in older versions of Solaris 10). Given that someone at Oracle likely modified the Solaris man page to differ from the older OpenSolaris man page in illumos, I would expect recent versions of Solaris to disallow this on UFS, but someone would need to check. That said, I have to recant my previous comment. smw likely linked the directory, which is insane, but would have worked on older Solaris versions if we assume the modern illumos UFS driver is unchanged in this regard.

jmclnx 2y ago

Interesting, at one time ages ago I thought Linux came with a static linked bash in /bin or maybe /sbin for this type of problem. Just checked Slackware and no more, I wonder if that is a casualty of the /bin /usr/bin merge ?

ryao 2y ago
Gentoo used to include a static busybox binary for recovery purposes, but it was removed from the system set years ago. Now you are expected to use the initramfs for recovery and can install bust box yourself if you are concerned. In fairness, this particular issue would be made worse by rebooting since the unlinked anonymous library files would be garbage collected, which is bad unless you have a filesystem that has snapshots like ZFS and had used them prior to this. The article author did not try to recover libraries from the anonymous files, which is probably good considering that only a subset would have been in use and thus only that subset would be recoverable from the anonymous files (unless there are filesystem snapshots).
pengaru 2y ago
once upon a time everything in /sbin was statically linked then only `sln` was statically linked (a variant of `ln`) today most distros statically link nothing and you're up shit's creek in this situation

ryao 2y ago

The technical term is unlinked. The files in use by running processes are still anonymous files in the filesystem. They will not be garbage collected until the last program that mmap’ed them is gone. If you could find the inode number from /proc (possibly from /proc/$PID/maps), you should be able to use a filesystem specific tool to retrieve them, such as debugfs or zdb.

abound 2y ago
I think the tricky piece is that you'd need to find that inode number without using any dynamically linked libraries, including ls and cat in the author's case. And debugfs is likely dynamically linked too (I just checked and it is on my machine)
- ryao 2y ago
  Presumably, the read shell builtin can be used to read files in /proc. As per the original article, you can get a static version over the network using only bash builtins and overwrite a file with execute permission to be able to execute it.

eitland 2y ago

Something similar happened a place I worked somewhere between 10 and 20 years ago. Service technician did not see the . in the command rm -rf ./bin so he proceeded to run rm -rf /bin When that didn't work he did what everyone who knows a little bit Linux does and added sudo in front. I was on a terminal from the other side of the globe when the server suddenly started acting weird. We were able to use scp or rsync (one of then was in sbin or something) to get back the bits from an identical server, which saved me from three days of tedious work :-) In hindsight of course we should have written the docs in a way that would prevent this exact situation but in the beginning it was just the output of the history command after I had done it, dumped into a document with some explanations.

ryao 2y ago
It is not the same thing, but recently, my pfsense router started acting weird where dns stopped working. I was bust so I rebooted it and it failed to boot. I ended up bringing a monitor and keyboard to see what went wrong and it turned out that suricata’s logs had used 1.4TB on my 250GB SSD (ZFS zstd compression is awesome), causing it to run out of space a few years after I enabled the feature out of curiously. I wiped the logs, rebooted and things worked. My lesson from them is when a machine servers acting strangely, do not reboot and instead troubleshoot right away. Had I done that, I could have found the problem and had only minimal partial downtime (I would have had to restart dns afterward). Since I was too busy to do things the right way, my internet was out for a half hour. Your story reminded me of this, since you would have had a bigger headache if you had rebooted.
- hinkley 2y ago
  This is the origin story for anyone who has ever set up alerts for 90% disk utilization on a machine. These are especially bad with services that generally grow their logs very slowly but when something like a net split happens or a server is down they generate as much log data in an hour as they typically do all week. So you get close to full and then an incident happens, and now you have two incidents because the screaming machine goes down with a full disk right after you lose your internal DNS server or what have you.

nunez 2y ago

Super interesting, though it feels like a response to an interview question!

Dwedit 2y ago

If you deleted /lib, you'd probably be better off reinstalling packages while booting off of USB or something. You're gonna have downtime because programs won't work correctly.

LorenDB 2y ago
I also had to wonder why not just liveboot from USB or attach the affected boot medium to another system, then use the recovery system's fully working tools to just relink the /lib folders?
- ideasphere 2y ago
  “This is not a very important machine, and I could have just reimaged the MicroSD card and be done with it, but I was curious if I could recover from the error.”

nullorempty 2y ago

At the start of my career I removed the `x` attribute from all files :)

ivanjermakov 2y ago
chmod -R is too convenient lol
ryao 2y ago
Of all of the stories I have read here, this is the first to make me laugh. Congratulations. :)
- nullorempty 2y ago
  ... we had an amazing sysadmin. He had a shell open on that box when I came to tell the news. He started to type quickly and thoughtfully, trying utilities I haven't even heard of. Then, he echoed a small C program that was supposed to set `x` on the chmod. Quickly he typed `cc x.c` to compile... and then it dawned on him.
  - ryao 2y ago
    Perhaps scp -p or sshfs would have worked. This is just a guess. I am not certain.

inejge 2y ago

The heroic version of the story is now almost 40 years old[1]. (One HN mention with the link to a HTMLized version is here[2].) In both cases, the upshot is that as long as you have a running shell with root privileges, at least one existing executable file on the filesystem, and the means to overwrite that file with arbitrary binary content, you can write a small program which can recreate a skeleton system structure and dig yourself out of the hole. The reason why this keeps happening is that in regular UNIX root is omnipotent and the filesystem is ultimately unprotected. Immutable systems and restricted execution environments may make this a thing of the past. [1] https://www.wolczko.com/rm.txt https://www.wolczko.com/rm.txt [2] https://news.ycombinator.com/item?id=7892471 https://news.ycombinator.com/item?id=7892471

lloeki 2y ago

A long long time ago the team I was part of managed old unix systems. A coworker telnet'd (or rsh, can't recall) into such a machine to do some maintenance and after a while fat fingered: umount / Would you believe it, back then being root meant this was absolutely unprotected and the minicomputer OS (some ancient AIX) dutifully complied. The chaos that ensued is but a blur.

teaearlgraycold 2y ago
Did you just restart?
- WesolyKubeczek 2y ago
  Wasn’t restarting those AIX dinosaurs a nontrivial thing?
  - teaearlgraycold 2y ago
    I could believe it. I’m just hoping OP can provide more info.
- dingaling 2y ago
  Using what command, though?
  - teaearlgraycold 2y ago
    Perhaps there's a restart button?

deleted 2y ago

[deleted]

gleenn 2y ago

One time I was flipping back and forth between directories, compiling some code, then checking it, then rm -rf'ing it. I accidentally hit up an cd ..'d one too many times. Suddenly the rm command hung and I was confused because it should be nearly instant as it was a few files. I stared in horror as I was accidentally deleting everything in mybhome directory. Luckily back then I had a .pr0n directory with a significant amount of content. A few things were lost but that .pr0n folder was luckily early enough in the list and big enough to slow down the deletion of my photos and documents. That's why I always recommend having a big "buffer" of video content for such situations, ya know, for data integrity ;)

hinkley 2y ago
Unless you were using tcsh you had to rewrite your shell configuration though.
OKRainbowKid 2y ago
Great idea, I shall see to it right away.
winwang 2y ago
That's wild and inspirational. Like the scene in Rush Hour when a stack of bills saves Tucker from a bullet. Though I'm moreso tempted to just create a `.1111aaaa-antidumb` directory and store my caches and backups there. This has also un-inspired me from creating a fast `rm`-esque utility.
- woleium 2y ago
  You could of course alias rm -rf to rm -rf -i
  - marc_abonce 2y ago
    I always use trash-cli and alias rm to 'echo NO! #'. Only if the file is too big to fit into the garbage bin, I can unalias rm, rm the thing and then reset the alias immediately after.

o11c 2y ago

> Please note that busybox can’t function with a name that is not a busybox applet name. This is somewhat wrong. From ksh/bash/zsh, you can run: (exec -a someappname /arbitraryexecutablepath args...) This won't work on most ash derivatives (including /bin/sh on Debian, FreeBSD, or NetBSD), but does work on busybox ash. The parentheses prevent the `exec` from actually replacing your current shell, which might be less important for emergency rescues, but which otherwise is often what you want with `exec -a`.

nurple 2y ago

My workstation seems fine: $ ls -R /{lib,usr,bin,sbin} ls: cannot access '/sbin': No such file or directory /bin: sh /lib: ld-linux.so.2 /usr: bin /usr/bin: env Oh right... $ ls -l /usr/bin/env lrwxrwxrwx 1 root root 65 Mar 21 23:39 /usr/bin/env -> /nix/store/9m68vvhnsq5cpkskphgw84ikl9m6wjwp-coreutils-9.5/bin/env $ ldd /usr/bin/env linux-vdso.so.1 (0x00007ffff7fc4000) libacl.so.1 => /nix/store/dyizbk50iglbibrbwbgw2mhgskwb6ham-acl-2.3.2/lib/libacl.so.1 (0x00007ffff7fb3000) libattr.so.1 => /nix/store/vlgwyb076hkz7yv96sjnj9msb1jn1ggz-attr-2.5.2/lib/libattr.so.1 (0x00007ffff7fab000) libgmp.so.10 => /nix/store/dsxb6qvi21bzy21c98kb71wfbdj4lmz7-gmp-with-cxx-6.3.0/lib/libgmp.so.10 (0x00007ffff7f06000) libc.so.6 => /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/libc.so.6 (0x00007ffff7d0e000) /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib/ld-linux-x86-64.so.2 => /nix/store/maxa3xhmxggrc5v2vc0c3pjb79hjlkp9-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fc6000)

MadnessASAP 2y ago
Don't have to worry about trashing FHS if your OS doesn't use FHS :-P

zavec 2y ago

I actually started a short blog series about a similar problem where a friend had blown away /bin and a bunch of other stuff, but/lib was still there. Unfortunately it didn't end up getting anywhere because even though I was able to drop executables on the machine with echo and make them executable with a .so from lib I wasn't able to get back to root permissions as sudo and everything had been blown away and I didn't think I'd have great luck trying to find a zero-day in the kernel. It was still a lot of fun though.

Fizzadar 2y ago

I once nuked the entire OS partition on an openvz host. Vz data was still good so we ended up copying the root fs from another similar box, manually updated the network config and it ran for another 4 years until retired.

throwanem 2y ago

"When you discover yourself to be in a hole, the best first thing to do is stop digging." Some of the best early professional advice I ever received was, in moments like these, to keep my hands off the keyboard for at least a timed minute.

ofalkaed 2y ago

Many years ago I wrote a backup script where I did "rm -rf /etc/" instead "cp -r /etc/ /mnt/whatever," not sure how I managed that. Took ages to figure out what was causing /etc/ to disappear since /etc/ going missing often went unnoticed for awhile and I was running Arch back in the days when running pacman -syu always caused exciting things to happen. I even did a complete backup and reinstall trying to figure that one out, and I was extra cautious about making a backup of that backup script which I had spent so much time on and was fairly proud of since it was my first non-trivial bash script. I also once did "rm -rf /", was deleting a dir which started with a "[" and accidentally hit "enter" instead of "\." That one taught me the dangers of absolute paths. Edit: That last one is not quite right, would not have been an absolute path issue, that dir must have ended up in root somehow, can't quite remember the details, been too long.

Vilian 2y ago
Isn't absolute path better practice than relative paths?
- ofalkaed 2y ago
  In scripts, yes, but they are not without their dangers and can be especially troublesome when working in the terminal since almost no one types out and checks the full paths constantly, they just let tab completion take care of it and assume it worked. In scripts things like ../../../../file are a pain to read and assume everything in the script before all those previous dirs worked as it should have and everything is where it should be. Cd to an incorrect absolute path produces an error code so we can be sure we are in the proper dir, cd ../../../ will never produce an error and always succeeds even if you are at root.
  - 3np 2y ago
    > almost no one types out and checks the full paths constantly, they just let tab completion take care of it and assume it worked. Are you a bootcamp coach having watched countless of people at the terminal? What makes you assume this is a general habit?

chatmasta 2y ago

I’ve seen this same story in exploit writeups - it’s funny to see “living off the land” techniques used non-maliciously.

mastax 2y ago

chattr +i /lib/ld-linux.so.2 Sounds tempting.

userbinator 2y ago

This could be considered a variant of the "bootstrapping" problem.

asah 2y ago

This happened to me once. I caught my breath, found an identical server and copied the files over using uuencode and cat, until I had enough working to create a second terminal window and could use better tools to verify everything. 5 minutes that felt a lot longer.

doubled112 2y ago
Sounds similar to a time I accidentally removed a bunch of system libraries with pip. I wish I would have had a similar server to recover from. Yum being a Python program made recovery tedious, since some of the libraries it depended on were installed via RPM and gone. Not a difficult recovery, but tedious.

conceptme 2y ago

A long time ago I started working at this company and I fat fingered 'rm -rf /' instead with a dot. When realizing what I had done and it was taking too long I powered off the machine. When I told the sysops he looked horrified and asked me how far it got. The fun thing was all websites and data where mounted from the server to each workstation to make it easy to update source code.

nasretdinov 2y ago

I think I now know why on macOS "Applications" (with capital A) comes first in the list of root-level directories. I've been showing to my colleague that on modern systems "sudo rm -rf /" does nothing, however that turned out to be GNU-specific thing, and macOS of course uses BSD tools. So I've started noticing that something is off very quickly and stopped the command. On HFS+ (probably on APFS too) the file list is always sorted, so it only deleted some of my Adobe apps and stuff like Addressbook, so I didn't even notice anything at first :)

vessenes 2y ago

I “rm -rf /“ ed my Linux box in 1996 and realized what I had done about 30 seconds in. It was through like a-d in /bin. ‘Recovery’ wasn’t a strong concept in Linux os installers then, and I wasn’t sure my home directory would survive a reinstall, (not on a separate partition) and I didn’t think I’d be able to reboot successfully in any event. My brother ran the same distribution but at a school across the country; I was able to recover by carefully pulling what I needed down with ftp to his dorm computers IP. I’m not sure scp even existed then on Linux; I guess maybe we could have used netcat in a pinch. Well, this was already a pinch. In a real pickle. This was one of the first moments where high bandwidth connections to endpoints really saved me / impacted me, and I haven’t forgotten. For many years after university this kind of direct high bandwidth connection got much harder to achieve, first because we were back in a low bandwidth residential world, then because ipv4 was mostly denied to consumers, then because of nat. Today this would again be achievable, but with vastly more complexity. For a home computer rescue you’d want Tailscale in both sides. And it’s extremely unlikely you’d be using the same distribution as your sib, much less have the same libc linking. And god help you if you had to restore your systemd directory by hand.

udev4096 2y ago

Reminds me of this: https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issues/123 https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/issue...

scripturial 2y ago

‘rm —rf’ should have default behavior to prompt “are you sure” when deleting top level folders. People are literally talking about secret video files saving their files from being accidentally destroyed. How in 2025 do we not have these types of innovations? Are people afraid of breaking old scripts? (macOS saves us from a lot of this stupidity, Linux should have something similar. I would love a way to mark folders as “permanent” or as “restore on reboot”)