5 ms·
Something similar happened a place I worked somewhere between 10 and 20 years ago. Service technician did not see the . in the command rm -rf ./bin so he p
by eitland 2y ago
Something similar happened a place I worked somewhere between 10 and 20 years ago.
Service technician did not see the . in the command
rm -rf ./bin
so he proceeded to run
rm -rf /bin
When that didn't work he did what everyone who knows a little bit Linux does and added sudo in front.
I was on a terminal from the other side of the globe when the server suddenly started acting weird.
We were able to use scp or rsync (one of then was in sbin or something) to get back the bits from an identical server, which saved me from three days of tedious work :-)
In hindsight of course we should have written the docs in a way that would prevent this exact situation but in the beginning it was just the output of the history command after I had done it, dumped into a document with some explanations.
- ryao 2y agoIt is not the same thing, but recently, my pfsense router started acting weird where dns stopped working. I was bust so I rebooted it and it failed to boot. I ended up bringing a monitor and keyboard to see what went wrong and it turned out that suricata’s logs had used 1.4TB on my 250GB SSD (ZFS zstd compression is awesome), causing it to run out of space a few years after I enabled the feature out of curiously. I wiped the logs, rebooted and things worked. My lesson from them is when a machine servers acting strangely, do not reboot and instead troubleshoot right away. Had I done that, I could have found the problem and had only minimal partial downtime (I would have had to restart dns afterward). Since I was too busy to do things the right way, my internet was out for a half hour. Your story reminded me of this, since you would have had a bigger headache if you had rebooted.
- hinkley 2y agoThis is the origin story for anyone who has ever set up alerts for 90% disk utilization on a machine. These are especially bad with services that generally grow their logs very slowly but when something like a net split happens or a server is down they generate as much log data in an hour as they typically do all week. So you get close to full and then an incident happens, and now you have two incidents because the screaming machine goes down with a full disk right after you lose your internal DNS server or what have you.
- 3eb7988a1663 2y agoOne trick I learned is to create a big random file (must be random to ensure no funny compression filesystem tricks out smart you) called something like BIG_DUMMY_DATA_SAFE_TO_DELETE. When you find yourself in a catastrophic space situation, you can delete the file and have a less panicky recovery process as the immediate problem is gone.
- nneonneo 2y agoWorks right up until you have another incident, go to delete the file, and realize you already deleted it the first time…
- PhilipRoman 2y agoThe funniest consequence of full disk was that I could not log in to a server, because the login process required writing some tiny temporary file in the user's home. Did not find a way to recover it, other than plugging a keyboard in it.
- kees99 2y agoYou were lucky scp uses a binary that typically lives outside /bin - in /usr/lib/openssh/ or some such. Many years ago, I've got to recover a remote server where /usr was nuked (and /bin, /sbin, and /lib were all symlinks into now-empty /usr). Ended up writing a one-liner perl to convert /bin/busybox-static from my local machine into a series of: echo -ne "\x7f\x45..." >>~/busybox-static ...and copy-pasting that, chunk-by-chunk, into a single surviving ssh/bash connection, and then used that busybox binary to pull in from a backup.
- deleted 2y ago[deleted]
- hinkley 2y agoI have learned through hard experience that is you ever user sudo to edit the sudoers file, create two shell windows logged in as root before doing so. Use visudo to edit the file of course, because not doing so can blow everything up by rendering the sudoers file unparseable and then everyone is gonna have a bad time. But also the temptation when altering sudo is to immediately log out as super user and try using sudo to do the new command. If you’ve fucked up the file you might not be able to sudo anymore. So now use your second shell window to undo whatever you just did in the first window.
- ryao 2y agoI have setup ssh remote forwards to a jump host in the past to allow remote access through a firewall. daemontools executes scripts exec’ing ssh. ExitOnForwardFailure and ServerAliveInterval are set client side with ClientAliveInterval and ClientAliveCountMax set server side to enable rapid recovery if something goes wrong. Whenever one of the daemon tools scripts doing remote forwards needs to be modified, a second reverse forward script is added and the reverse forward from that is used for ssh, before changing the original script. The second script is removed only after confirming the first still works after the edit. This procedure prevents fatfingering from locking out remote access, since if something goes wrong, you just need to redo the previous step(s) until you get things working. If anyone wants to replicate that, I suggest setting -nNT as arguments to ssh and restricting what the user login can do via sshd_config.
- malkia 2y agoWhat would be safer alternative? pwd # Then check something? pushd bin # rm -rf . Probably still with pitfalls
- 01HNNWZ0MV43FF 2y ago`rm -rf bin`
- OJFord 2y agoI think the misguided belief that `./` means 'execute script' or something (program that isn't 'installed'?) is single-handedly to blame for so much script spaghetti.
- bigstrat2003 2y agoAnd not just for bin. There's probably an edge case where you would need to give the ./ prefix to rm, but I've never come across it. The vast, vast majority of the time just entering the name of the thing is easier and less error-prone.
- deleted 2y ago[deleted]
- pphysch 2y agoUse the full path of the bin dir in your rm rf
- hinkley 2y agorm -rf ~/bin could have some nasty consequences if you fat finger a return key anywhere in the middle. Run enough commands enough times and you will find Murphy is waiting for you. If you’re just removing a bin directory one time, odds are low but not zero. If you’re writing a run book for people to use, odds are 100% that you will have to help someone rebuild at least once.
- cortesoft 2y ago
- trelane 2y ago> In hindsight of course we should have written the docs in a way that would prevent this exact situation I would say the bigger failure is relying on a human typing things into a terminal rather than automating the tasks or changing the system so the task is no longer needed.
- micw 2y agoIt's always a good idea to allow sudo to untrained people on critical systems...
- hinkley 2y agoMy second boss was a Sun Microsystems enjoyer and he always pronounced “superuser” as “stupid user”. After a raised eyebrow he went on to explain, “because when the machine is broken it’s always because some stupid user did something. It took me a couple more stupiduser incidents of my own before I instituted a rule of counting to five before hitting enter on any `rm -rf` command.
- Ferret7446 2y ago> I instituted a rule of counting to five before hitting enter on any `rm -rf` command. That's just (should be) standard practice. Another risky command is `sudo dd`/`sudo cat` for writing disk images, always chant the disk device against an fdisk -l listing like a magical spell, lest you nuke your main drive.
- cafeinux 2y agoWhen using `dd` I always verbally say "`if` is input file, because I want to take data from <whatever I wrote there> and `of` is output file, because I want that data to overwrite <whatever I wrote here>". Never have I ever had a `dd` incident, but that's such a fear I have that I developed this habit.
- ryan-c 2y agoof stands for obliterated file
- thedrexster 2y agoThis is the way!
- bear8642 2y agoand dd is data destroyer
- hinkley 2y agoBut it’s surprising how often you see the wheels turn when you explain it to someone else. It’s not obvious to everyone.
- rav 2y agoI often run rm -rf as part of operations (for reasons), and my habit is to first run "sudo du -csh" on the paths to be deleted, check that the total size makes sense, and then up-arrow and replace "du -csh" with "rm -rf".
- mekster 2y agoUse trash-cli and additionally git commit the target if you’re nervous before deletion.
- mekster 2y agoFirst, you don’t let people read and type commands by hand. Second, hire a better guy than someone who blindly does sudo because the command didn’t work.
- eitland 2y agoFirst, there is a difference between rolling out 5 such systems a year and modern server farms, second, replacing a brilliant electronics guy because he once messed up when he helped us out on site isn't something I'd want to have on my cv. Third this is well over a decade ago.