9 ms·

Unix Recovery Legend (1986)

168 points by signa11 6y ago

oofabz 6y ago
If this happened to me, my gut reaction would be to connect the hard drive to another, working computer, and do the recovery in a fully functional environment. With all the physical work required, it probably would have taken longer. But if anything else unexpected happened, it would have been easier to deal with.
- ncmncm 6y ago
  Spare vaxen were not just sitting on shelves back then.
Galanwe 6y ago
Could someone shed some light on the constraints of Unix & VAX at the time? The decision making here does not make much sense to a youngster (36) like me. Why couldn't they just mount their half deleted drive on an other machine and restore it from there? I have the feeling all these other hundreds of seemingly easier solutions were out of the realm of possibilities back then, but I am unsure of what were the actual limitations / constraints of that era.
- karim 6y ago
  Most likely they didn't have a VAX in the same building they could do that with, so they had to improvise. These things were pretty big and expensive (https://en.wikipedia.org/wiki/VAX#/media/File:VAX_11-780_intero.jpg https://en.wikipedia.org/wiki/VAX#/media/File:VAX_11-780_int...)!
- pram 6y ago
  They could have restored it on their current machine/disk FWIW. You could do a tape boot, boot to single user, and copy the OS and rebuild the devices. I think they were just unfamiliar with that process.
- gdavisson 6y ago
  I'm not that familiar with the relevant systems, but this was before plug-and-play was a thing, and moving drives around wasn't a minor operation. Mario mentions a similar option in the article: "Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. "
- SulphurCrested 6y ago
  In the late 1970s/early 1980s VAX hard drives usually took cartridges which could be conveniently unplugged and carried across to a drive on another machine. But by 1986 drives were often "Winchesters" - sealed, rack width, a foot high, and weighing maybe 100kg. The cables were an inch thick and the plugs the size of a large ham and cheese sandwich. Hence the comment in the article that they'd need to pay a DEC technician to come out and physically do the move.
userbinator 6y ago
Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. I've not used Unix on an early VAX before, but every other OS I've used, including DOS, Windows, Linux, and macOS has the option to use a "recovery mode" shell if you boot from the OS installation media.
- jasomill 6y ago
  4.3BSD VAX was distributed on open-reel magnetic tape. The second file of the first tape contained a miniroot filesystem which, at the very least, certainly would have included working copies of of the two utilities mknod(8) and restore(8) required to restore from his dump(8) tapes. As with DOS, Windows, Linux, etc., 4.3 BSD could only mount filesystems stored on random-access block devices (so not sequential-access magtape). Therefore, one of the first steps in "booting from distribution media" was to copy this miniroot filesystem to the swap partition of a formatted and labeled disk, at which point it could be mounted as the root filesystem (with swapping to this partition disabled, obviously). In the (likely) case that the existing disk layout contained a swap partition of sufficient size on a device supported by the boot loaders and kernel supplied by the distribution, this would have indeed been a reasonable road to recovery, and, assuming the existing disk labels remained valid, the required swap space was available, and the dump(8) tapes were good, shouldn't have required anything beyond: 1. Copying the miniroot from tape to a preexisting swap partition using the standalone copy program included with the distribution media (typically on a console tape or diskette as appropriate for the VAX model in question). 2. Booting vmunix from the miniroot in the swap partition, taking care to RTFM and include the special syntax required to prevent swapping to the partition containing the miniroot. 3. Recreating required special files with mknod(8) (possibly using the /dev/MAKEDEV script from the miniroot). 4. Using restore(8) to restore from dump(8) tapes as usual.
pram 6y ago
Another useful thing to remember: if you deleted a file and a process still has it open, you can copy it from the file descriptor in /proc! Just use lsof to find the PID and FD. This has saved my bacon a few times in the past ;P
- naikrovek 6y ago
  Only on Linux, and not everyone who uses Unix uses Linux.
  - anonacct37 6y ago
    I actually had to recover a deleted file on OSX once. I sure wished the proc filesystem was present there. I ended up using gdb (or maybe lldb I don't recall) to load a shared object that did a seek()/open()/copy(). Totally hacky but it worked!
  - btilly 6y ago
    Actually /proc was invented on Solaris in the mid-90s. However this story took place on DEC Unix about 10 years earlier. So /proc was not available.
    - anthk 6y ago
      False, from plan9.
      - wizzwizz4 6y ago
        Like with Euler, we generally credit the first one to do it after Plan9.
  - userbinator 6y ago
    ...and Windows won't let you delete a file that's still in use, so this scenario can't happen.
    - 6y ago
tyingq 6y ago
I remember accidentally removing the shared libc on a SunOS box and having to use things in /sbin to recreate it...though I don't remember exactly what I did.
- nneonneo 6y ago
  Hey, I did that exact thing (removing libc) on a QNX box years ago. I ended up restoring libc using a series of "echo -ne" commands (generated with a script, and then pasted into the SSH session I thankfully still had open).
enw 6y ago
At my first job right after graduation (less than 5 years ago), I managed to accidentally terminate production servers on a Friday night. When I realized my mistake, my heart sunk. I got cold sweats. I was afraid I'd get fired. To this day I'm so glad my team leader was still at the office at the time. I told him about the blunder, and he asked me about the details. He sat down next to me, and we started working together, exchanging information and getting replacements back up. There was no CloudFormation, so everything was manual. He never accused me in the postmortem, and I still feel so grateful for it. I learned from my mistakes and took action to prevent it from happening again.
- jhabdas 6y ago
  Never drink at the office. Even if they have a bar.
  - Something1234 6y ago
    Where did he mention booze?
    - croes 6y ago
      He probably infers from himself to others.
  - spicybright 6y ago
    Stop projecting your own issues.
  - iamtedd 6y ago
    I'd like to hear the story behind that one!
- siffland 6y ago
  At my first UNIX IT job, my team lead told me: Listen when you break something tell us. Explain what you did and we can get it fixed. You will make mistakes, we all still make mistakes. That is what i always do and expect my coworkers to do. Yeah we talk a little smack afterwards (only jokingly), but were able to resolve most self inflicted wounds quickly. Nowadays what scares me more is a coworker who hides mistakes rather than one who confesses to them.
alyandon 6y ago
I once fat-fingered crontab -r instead of crontab -e on a production accounting system running Solaris which immediately wiped out something on the order of about 200 cron entries. Had to restore the contents from off-site backup tapes. The jokes circling the office for the next few weeks were all variations on "Remember alyandon - r doesn't mean 'revise'". :(
- Waterluvian 6y ago
  Kind of a terribly bad engineering job to design a program that can do so much damage by being one key off. I've never understood why we accept such a terrible interface as some intractable reality.
  - nexuist 6y ago
    Most places don't. Cron is great for hobbyists and personal use, nobody should be using it in production. There are far easier and safer ways to schedule recurring tasks.
    - verroq 6y ago
      Those are?
      - jolux 6y ago
        Probably systemd these days.
        MertsA 6y ago
        Honestly all joking aside systemd timer units have a lot of nice aspects. The only drag is that crontab entries are so much more terse and quick to add. Just one line instead of two unit files plus enabling/starting the timer. You can actually get the best of both with systemd-crontab-generator (not from the systemd project, just an unaffiliated third party). https://manpages.debian.org/buster/systemd-cron/systemd-crontab-generator.8.en.html https://manpages.debian.org/buster/systemd-cron/systemd-cron... It creates transient unit files based off of a standard crontab on boot and after any file modifications. If you want to extend one of your cron jobs with something that you can't express in a crontab you can just mv it from /run to /etc and modify it however you'd like.
rsync 6y ago
I'm not going to go into details - they are long, torturous and (of course) occurred very early in the morning - as they always did. But, rsync.net used to run - exclusively - on UFS2 and, circa 2009/2010 we were pushing UFS2 to the very limits in terms of disk size, number of inodes per filesystem, size of memory used to fsck, etc.[1] Several times we came very, very close to destroying a filesystem and losing customer data. What made all the difference, every time, was: - don't panic - postulate what is happening - do something non destructive that will prove or disprove the theory - possibly repeat Go slow and don't take any steps until you have proven that you understand the problem. ALSO, when working with filesystems remember that you can run a filesystem, non-destructively, in read-only mode. There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive. Your first steps should always be read-only ... [1] I asked Kirk M., in 2010, what the long term solution to our problems with UFS was and he said "migrate to ZFS".
- gpderetta 6y ago
  > There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive. Some years ago (but not so long ago that I shouldn't have known better...), I wiped out by mistake my home directory were I had a few days of unpushed changes (yes, second mistake) to an important project I was working on. When I realized what I had done, I immediately remounted read-only and I managed to preserve the data; a few hours later, ext4-undelete saved the day.
  - jlokier 6y ago
    I did something like that recently, deleting a project directory with a mistyped git command intended to preserve the contents for safekeeping, ironically. (That's an interesting lesson in itself). It was in a VM, so rather than remount read-only, I snapshotted the VM drive, then rebooted to perform recovery, confident I could try out things non-destructively by reverting to the snapshot. Used ext4magic to undelete, and that tool did a really great job. Some files ended up with contents that should have been in other files, so it isn't perfect. But it was a project with only a few files, so I was able to reconstruct it easily enough because all the contents were at least recovered. Did something similar for a customer a few years ago who had wiped a significant directory on a live system. That was on a RAID-1 (mirrored). Rather than remounting read-only, to allow production to continue I detached one of the mirrored drives to preserve it for safekeeping, while allowing operations to continue using the other drive.
noisy_boy 6y ago
I remember fork-bombing a production server - the sheer panic as process table entries keep getting used up and commands getting slower and slower by second is unforgettable (which is probably why I'm super cautious about anything to do with Production now).
- anonymousiam 6y ago
  I once (accidentally) took down Netcom with a fork bomb. Not just one node, but all of them. When I tried to login again after they rebooted everything, I found that my account had been banned. I got a NOC admin on the phone and apologized for my mistake, but also asked him why they did not enforce a maximum number of processes per user (via ulimit). Apparently they had never thought of doing that. He thanked me and my account was restored immediately.
- bch 6y ago
  Experiences like this and a bit of imagination are why I try to be not-root as quickly as possible, and sometimes consciously physically remove my hands from the keyboard instead of having them “idling” at their home position.
  - anonymousiam 6y ago
    The thing about a fork-bomb is that you do not need to be root to spawn one. I never had root on Netcom, but still manged to take the whole thing off-line for about 15 minutes.
    - bch 6y ago
      Yea, I didn’t mean fork-bomb specifically, just “fuck ups” generally, or “critical sections” of operations as a human :)
c0l0 6y ago
What a beautiful story! I've read a similarly impressive account, where a GNU/Linux system was revived in comparably wizard-like manner, with Kernel hacker Al Viro involved: https://yarchive.net/comp/linux/extreme_system_recovery.html https://yarchive.net/comp/linux/extreme_system_recovery.html It was linked to in an LWN article's comment section, titled "If you think you're really good, read this.", or something. The story proved very effective at humbling its reader for me. That doesn't change the fact that I, too, once felt like the uncrowned teenage king of emergency system rescue operations back in 2002 or so, right in my parents' basement. I had started recursively deleting / by accident, and ended up with most directories, /bin and /sbin included, already turned to dust, and most of my as of then proudest achievement - a Gentoo Linux stage 1 installation - gone. I dug myself (and the standing remains of my /home) out of that particular hole by virtue of still having an instance of a browser (I think firefox) running. Mildly panicking, I still left everything untouched ... and after a few hours of thinking hard about what to do, I arrived at the following: I set up a webserver on my mom's windows PC with am embedded PHP interpreter, and hacked up a script that enabled file uploads via HTTP over the LAN. After using that from within my still-running browser (fortunately, it could still list the contents of my home directory) to evac the most precious artifacts my ~ had in store, I rebooted my machine into the lightless nothingness of not having a kernel image in /boot any more. Another few hours later, I had Gentoo reinstalled, and copied the rescued files over to my own machine again - and felt like I had levelled up a little bit.
- segfaultbuserr 6y ago
  > comparably wizard-like manner, with Kernel hacker Al Viro involved: https://yarchive.net/comp/linux/extreme_system_recovery.html https://yarchive.net/comp/linux/extreme_system_recovery.html If I were him, the best I could imagine was doing documentation reading and ASM writing on a separate computer with working toolchains, testing it, and finally entering some machine code in hex to the broken machine by hand. But he literally wrote an ASM program and constructed an ELF on a totally broken machine itself on the fly, with no documentation, no toolchain. Absolutely wizard. For example, it was how the syscall number was found without grep... > The next step was getting the syscall numbers. grep? We don't need no stinkin' grep. # while read i; do case $i in *__NR_link*) echo $i;; esac; done </usr/include/asm/unistd.h #define __NR_link 9 # while read i; do case $i in *__NR_exit*) echo $i;; esac; done </usr/include/asm/unistd.h #define __NR_exit 1 #define __NR__exit __NR_exit
trm42 6y ago
On day in my previous job our junior support guy started cursing loudly. He had done what he was asked for and did some some small, _almost_ day-to-day support request which was supposed to do some requested maintenance things for some credit card accounts our company was handling. He by some small accident ended up invalidating the cards with our CC operator and closing the accounts on our end. He was so wrecked about it and we, the more senior guys started helping him and comforting him that "yeah, this kind of blunders can happen to anyone, no worries. Let's figure this out together.". The other guys figured how to restore the data into correct setup on our end, and I found some unimplemented badly documented "undo invalidation" endpoint from the CC operator with which we could fix the stuff on the operators end. So we spent nice and fun evening together figuring out the problem as a team, but damn the guy was so torn up about his small and completely understandable fumble that nobody else needed to critique him about it. The funny thing is that with this kind of blunders one learns a lot more and a lot faster than usually. People need to make mistakes. It's usually the best way to learn.
- eitland 6y ago
  > It's usually the best way to learn. There's a Norwegian saying that damage makes you knowledgeable - but not rich. Learning from others mistakes are very often cheaper and/or less painful.
  - cpach 6y ago
    Just out of curiosity: How does that saying read in bokmål? (Or nynorsk) Hilsen fra Sverige (^_^)
    - unhammer 6y ago
      "av skade blir man klok, men sjelden rik" is one variant; there's also the predicative form "klok av skade". (though I'd translate "klok" with "wise")
      - eitland 6y ago
dang 6y ago
If curious see also 2015 https://news.ycombinator.com/item?id=10160417 https://news.ycombinator.com/item?id=10160417 2014 https://news.ycombinator.com/item?id=7892471 https://news.ycombinator.com/item?id=7892471 2010 https://news.ycombinator.com/item?id=1212051 https://news.ycombinator.com/item?id=1212051
sapeint 6y ago
Reminds me of a few years back when I was still new to Linux (and anything non-Windows in general tbqh), and I accidentally nuked /bin. That was a fun time. I happened to have Firefox and an IRC Client opened. I tried a few different things (on the advice of others in an IRC channel I happened to be in), but eventually a friend created a VM of the same Linux Distro and then compressed his /bin and sent it to me -- along with a few required programs to uncompress it just in case. That was a fun day or two. Taught me a few important things though!
danwills 6y ago
I blindly followed a 'news' item's advice once on my home Gentoo Linux that said a particular version of glibc should be removed. Learned the hard way that you definitely want to make sure you have emerged another version of glibc first! Hardly anything works without it (can't even auth as root) Unfortunately unlike the cool cucumbers in this story, I hastily rebooted, and it took a fair while for various reasons to get hold of a boot disk and get it repaired and running again. I love Gentoo but it does take a lot of patience generally, and this kind of thing seems to happen fairly continuously (which teaches you a lot but requires that you want to learn and solve the problems yourself).
simonh 6y ago
I did something like this on a Sun SPARCStation ELC back in the early 90s. I was setting the machine up and was new to Unix and the shell and deleted the contents of the root directory. Not the sub-folders just the files. That doesn't sound so bad right, there's nothing important just in the root directory - except in those days that's where the system kept the kernel binary! No problem, I'd learned how to do network file transfers the day before so I just copied the kernel across from another ELC and rebooted. Phew!
- h2odragon 6y ago
  I was playing with raw disk access and overwrote selected parts of my kernel image once. oopsie indeed but it was a dev system and so figuring that one out wasn't that bad in terms of impact on others. which was good, cuz it took me a week to figure out.
tragomaskhalos 6y ago
A key takeaway is "don't be root unless you have to be". Horrifies me the number of times at work, even on a production box, that "who" reveals a couple of roots logged on for no good reason. DBAs are the worst offenders.
- cafard 6y ago
  An old-timey sysadmin I met referred to superuser as "stupiduser".
foxtrottbravo 6y ago
Reminds me of the times I ran `killall` on a SCO machine only to learn that they take the name very literally...
g051051 6y ago
Not quite the same, but one time a guy I worked with decided to "clean up" things on a NCR Unix system and set the ownership of various system files to whatever he thought made the most sense. This broke all the stuff that relied on setuid to function correctly. He then went to lunch while the machine slowly ground to a halt. I had to boot from tape and figure out how to fix things from there.
dtoznayxvf 6y ago
In mid 2000s as a lead HPC sysadmin I accidentally pasted 'shutdown now' into the head node of a 200 CPU HPC cluster, which was 3 weeks into a 12 week simulation job! Fortunately that thing never shut down cleanly and it hung as usual, unable to unmount the NFS share used by all the compute nodes. For the next 9 weeks I was staring at that screen hoping it would keep going and I would not be fired. In the end nobody was ever the wiser.
peter_d_sherman 6y ago
This brings up an interesting issue in OS design, and that is: What are the responsibilities of the OS's Kernel if/when all files are deleted, and all processes are similarly deleted? In other words, the worst-case of all worst-case scenarios. Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel? Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only. Also, we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty" -- for future recycling. Between these two things, if they were implemented, we'd get a lot of leverage to resurrect a dead system, but there's more to this than that. You see, the above scenario should become an active unit test (tested at each recompilation) of that future OS. Also -- we want to know the fewest number of steps necessary (given the above scenario) to fix everything back to normal. In other words, a very interesting topic for future OS designers to think about!
- tremon 6y ago
  Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel? Practically speaking, none. The responsibility of the kernel (in Linux terms, but I think it's the same across all Unixen) is to start the init system (PID 1), and that's where it ends. Should PID 1 ever terminate, the kernel is in an undefined state. The VFS semantics ensure that even when the init binary is removed from the filesystem, it is not actually unlinked until its reference count is zero -- and since the kernel holds a reference to it, the kernel and init are "safe" from harm through deletion. Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only. While not in-kernel, systemd is ahead of you, as it offers a shutdown-time ramfs to use as root filesystem so the real root filesystem can be safely unmounted. Previously, Linux didn't really unmount the root filesystem (it couldn't, because the kernel still had an open handle to the init binary), but mounted it read-only before shutdown. we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty" This is already the case. "removing" a file in Unix only removes its name from the directory listing, the inode isn't reused until it's been cleared. And even when it's been cleared, only the inode entry is zeroed -- not the data contents. That's why tools like "shred" exist -- to make sure a file's contents are removed along with the file. a very interesting topic for future OS designers to think about I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.
janci 6y ago
I managed to break a btrfs filesystem by exhausting metadata blocks. The filesystem would show as 50% occupied, but 0 bytes free. Everything started crashing. The resolution is to run btrfs balance to free unused metadata blocks. There is a catch - you need at least some free space to run the balance. I had none, but managed to remove some unnecessary files (system package manager cache) to have just enough free space to run the balance. In meantime I discovered this condition was created by npm creating thousands of files while installing packages. Later I run into the same problem (again with npm), but this time there was not enough of unnecessary files to delete to be able run the balance (package manager cache was already purged). Normally this can only be solved by adding new partition to the btrfs filesystem and I had no unformatted or unused partiton on disk left. Instead I wanted to get rid of the btrfs that failed me twice and rsynced whole btrfs content into a loop-mounted ext4 disk image on a NTFS drive and then after unmounting I dd-ed the disk image over the btrfs. It booted! Had not any more filesystem trouble ever since.
mickotron 6y ago
Early in my Ops career I wrote a log handling script for Windows servers. It was a re-write of an existing one that pre-dated a lot of us. My version used an xml configuration file, but this iteration of the script did not properly validate the configuration items read from the file. Also, Powershell had this weird xml parsing bug that if a parent had a commented child element, it would show up as an empty child element. So I came in one afternoon shift and my colleague said "your script destroyed two Production servers last night". These servers were critical for monitoring Police radio GPS coordinates. "Impossible! It should only work on directories we configured it for!" I sat there looking at horror at the debug logs, and sure enough, it read the empty child element, changed directory into "", looking for files with "" name pattern, which for Powershell resulted in C:\Windows\system32 and all files. It proceeded to zip and remove "old" files. On a reboot, they did not come back up. To add insult to injury, when they rebuilt those servers manually, the same script with the commented out config was deployed, destroying the servers again. They realised that the script was the issue and decided not to deploy that the second time. So I learned a lot about config validation and defensive coding in the months following the incident.

168 points by signa11 6y ago

If this happened to me, my gut reaction would be to connect the hard drive to another, working computer, and do the recovery in a fully functional environment. With all the physical work required, it probably would have taken longer. But if anything else unexpected happened, it would have been easier to deal with.

ncmncm 6y ago
Spare vaxen were not just sitting on shelves back then.

Galanwe 6y ago

Could someone shed some light on the constraints of Unix & VAX at the time? The decision making here does not make much sense to a youngster (36) like me. Why couldn't they just mount their half deleted drive on an other machine and restore it from there? I have the feeling all these other hundreds of seemingly easier solutions were out of the realm of possibilities back then, but I am unsure of what were the actual limitations / constraints of that era.

karim 6y ago
Most likely they didn't have a VAX in the same building they could do that with, so they had to improvise. These things were pretty big and expensive (https://en.wikipedia.org/wiki/VAX#/media/File:VAX_11-780_intero.jpg https://en.wikipedia.org/wiki/VAX#/media/File:VAX_11-780_int...)!
pram 6y ago
They could have restored it on their current machine/disk FWIW. You could do a tape boot, boot to single user, and copy the OS and rebuild the devices. I think they were just unfamiliar with that process.
gdavisson 6y ago
I'm not that familiar with the relevant systems, but this was before plug-and-play was a thing, and moving drives around wasn't a minor operation. Mario mentions a similar option in the article: "Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. "
SulphurCrested 6y ago
In the late 1970s/early 1980s VAX hard drives usually took cartridges which could be conveniently unplugged and carried across to a drive on another machine. But by 1986 drives were often "Winchesters" - sealed, rack width, a foot high, and weighing maybe 100kg. The cables were an inch thick and the plugs the size of a large ham and cheese sandwich. Hence the comment in the article that they'd need to pay a DEC technician to come out and physically do the move.

userbinator 6y ago

Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. I've not used Unix on an early VAX before, but every other OS I've used, including DOS, Windows, Linux, and macOS has the option to use a "recovery mode" shell if you boot from the OS installation media.

jasomill 6y ago
4.3BSD VAX was distributed on open-reel magnetic tape. The second file of the first tape contained a miniroot filesystem which, at the very least, certainly would have included working copies of of the two utilities mknod(8) and restore(8) required to restore from his dump(8) tapes. As with DOS, Windows, Linux, etc., 4.3 BSD could only mount filesystems stored on random-access block devices (so not sequential-access magtape). Therefore, one of the first steps in "booting from distribution media" was to copy this miniroot filesystem to the swap partition of a formatted and labeled disk, at which point it could be mounted as the root filesystem (with swapping to this partition disabled, obviously). In the (likely) case that the existing disk layout contained a swap partition of sufficient size on a device supported by the boot loaders and kernel supplied by the distribution, this would have indeed been a reasonable road to recovery, and, assuming the existing disk labels remained valid, the required swap space was available, and the dump(8) tapes were good, shouldn't have required anything beyond: 1. Copying the miniroot from tape to a preexisting swap partition using the standalone copy program included with the distribution media (typically on a console tape or diskette as appropriate for the VAX model in question). 2. Booting vmunix from the miniroot in the swap partition, taking care to RTFM and include the special syntax required to prevent swapping to the partition containing the miniroot. 3. Recreating required special files with mknod(8) (possibly using the /dev/MAKEDEV script from the miniroot). 4. Using restore(8) to restore from dump(8) tapes as usual.

pram 6y ago

Another useful thing to remember: if you deleted a file and a process still has it open, you can copy it from the file descriptor in /proc! Just use lsof to find the PID and FD. This has saved my bacon a few times in the past ;P

naikrovek 6y ago
Only on Linux, and not everyone who uses Unix uses Linux.
- anonacct37 6y ago
  I actually had to recover a deleted file on OSX once. I sure wished the proc filesystem was present there. I ended up using gdb (or maybe lldb I don't recall) to load a shared object that did a seek()/open()/copy(). Totally hacky but it worked!
- btilly 6y ago
  Actually /proc was invented on Solaris in the mid-90s. However this story took place on DEC Unix about 10 years earlier. So /proc was not available.
  - anthk 6y ago
    False, from plan9.
    - wizzwizz4 6y ago
      Like with Euler, we generally credit the first one to do it after Plan9.
- userbinator 6y ago
  ...and Windows won't let you delete a file that's still in use, so this scenario can't happen.
  - 6y ago

tyingq 6y ago

I remember accidentally removing the shared libc on a SunOS box and having to use things in /sbin to recreate it...though I don't remember exactly what I did.

nneonneo 6y ago
Hey, I did that exact thing (removing libc) on a QNX box years ago. I ended up restoring libc using a series of "echo -ne" commands (generated with a script, and then pasted into the SSH session I thankfully still had open).

enw 6y ago

At my first job right after graduation (less than 5 years ago), I managed to accidentally terminate production servers on a Friday night. When I realized my mistake, my heart sunk. I got cold sweats. I was afraid I'd get fired. To this day I'm so glad my team leader was still at the office at the time. I told him about the blunder, and he asked me about the details. He sat down next to me, and we started working together, exchanging information and getting replacements back up. There was no CloudFormation, so everything was manual. He never accused me in the postmortem, and I still feel so grateful for it. I learned from my mistakes and took action to prevent it from happening again.

jhabdas 6y ago
Never drink at the office. Even if they have a bar.
- Something1234 6y ago
  Where did he mention booze?
  - croes 6y ago
    He probably infers from himself to others.
- spicybright 6y ago
  Stop projecting your own issues.
- iamtedd 6y ago
  I'd like to hear the story behind that one!
siffland 6y ago
At my first UNIX IT job, my team lead told me: Listen when you break something tell us. Explain what you did and we can get it fixed. You will make mistakes, we all still make mistakes. That is what i always do and expect my coworkers to do. Yeah we talk a little smack afterwards (only jokingly), but were able to resolve most self inflicted wounds quickly. Nowadays what scares me more is a coworker who hides mistakes rather than one who confesses to them.

alyandon 6y ago

I once fat-fingered crontab -r instead of crontab -e on a production accounting system running Solaris which immediately wiped out something on the order of about 200 cron entries. Had to restore the contents from off-site backup tapes. The jokes circling the office for the next few weeks were all variations on "Remember alyandon - r doesn't mean 'revise'". :(

Waterluvian 6y ago
Kind of a terribly bad engineering job to design a program that can do so much damage by being one key off. I've never understood why we accept such a terrible interface as some intractable reality.
- nexuist 6y ago
  Most places don't. Cron is great for hobbyists and personal use, nobody should be using it in production. There are far easier and safer ways to schedule recurring tasks.
  - verroq 6y ago
    Those are?
    - jolux 6y ago
      Probably systemd these days.
      - MertsA 6y ago
        Honestly all joking aside systemd timer units have a lot of nice aspects. The only drag is that crontab entries are so much more terse and quick to add. Just one line instead of two unit files plus enabling/starting the timer. You can actually get the best of both with systemd-crontab-generator (not from the systemd project, just an unaffiliated third party). https://manpages.debian.org/buster/systemd-cron/systemd-crontab-generator.8.en.html https://manpages.debian.org/buster/systemd-cron/systemd-cron... It creates transient unit files based off of a standard crontab on boot and after any file modifications. If you want to extend one of your cron jobs with something that you can't express in a crontab you can just mv it from /run to /etc and modify it however you'd like.

rsync 6y ago

I'm not going to go into details - they are long, torturous and (of course) occurred very early in the morning - as they always did. But, rsync.net used to run - exclusively - on UFS2 and, circa 2009/2010 we were pushing UFS2 to the very limits in terms of disk size, number of inodes per filesystem, size of memory used to fsck, etc.[1] Several times we came very, very close to destroying a filesystem and losing customer data. What made all the difference, every time, was: - don't panic - postulate what is happening - do something non destructive that will prove or disprove the theory - possibly repeat Go slow and don't take any steps until you have proven that you understand the problem. ALSO, when working with filesystems remember that you can run a filesystem, non-destructively, in read-only mode. There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive. Your first steps should always be read-only ... [1] I asked Kirk M., in 2010, what the long term solution to our problems with UFS was and he said "migrate to ZFS".

gpderetta 6y ago
> There are a lot of investigations that can be done on a filesystem in either explicitly read-only mode or with tools (like fsck) set to run no-write, or non-destructive. Some years ago (but not so long ago that I shouldn't have known better...), I wiped out by mistake my home directory were I had a few days of unpushed changes (yes, second mistake) to an important project I was working on. When I realized what I had done, I immediately remounted read-only and I managed to preserve the data; a few hours later, ext4-undelete saved the day.
- jlokier 6y ago
  I did something like that recently, deleting a project directory with a mistyped git command intended to preserve the contents for safekeeping, ironically. (That's an interesting lesson in itself). It was in a VM, so rather than remount read-only, I snapshotted the VM drive, then rebooted to perform recovery, confident I could try out things non-destructively by reverting to the snapshot. Used ext4magic to undelete, and that tool did a really great job. Some files ended up with contents that should have been in other files, so it isn't perfect. But it was a project with only a few files, so I was able to reconstruct it easily enough because all the contents were at least recovered. Did something similar for a customer a few years ago who had wiped a significant directory on a live system. That was on a RAID-1 (mirrored). Rather than remounting read-only, to allow production to continue I detached one of the mirrored drives to preserve it for safekeeping, while allowing operations to continue using the other drive.

noisy_boy 6y ago

I remember fork-bombing a production server - the sheer panic as process table entries keep getting used up and commands getting slower and slower by second is unforgettable (which is probably why I'm super cautious about anything to do with Production now).

anonymousiam 6y ago
I once (accidentally) took down Netcom with a fork bomb. Not just one node, but all of them. When I tried to login again after they rebooted everything, I found that my account had been banned. I got a NOC admin on the phone and apologized for my mistake, but also asked him why they did not enforce a maximum number of processes per user (via ulimit). Apparently they had never thought of doing that. He thanked me and my account was restored immediately.
bch 6y ago
Experiences like this and a bit of imagination are why I try to be not-root as quickly as possible, and sometimes consciously physically remove my hands from the keyboard instead of having them “idling” at their home position.
- anonymousiam 6y ago
  The thing about a fork-bomb is that you do not need to be root to spawn one. I never had root on Netcom, but still manged to take the whole thing off-line for about 15 minutes.
  - bch 6y ago
    Yea, I didn’t mean fork-bomb specifically, just “fuck ups” generally, or “critical sections” of operations as a human :)

c0l0 6y ago

What a beautiful story! I've read a similarly impressive account, where a GNU/Linux system was revived in comparably wizard-like manner, with Kernel hacker Al Viro involved: https://yarchive.net/comp/linux/extreme_system_recovery.html https://yarchive.net/comp/linux/extreme_system_recovery.html It was linked to in an LWN article's comment section, titled "If you think you're really good, read this.", or something. The story proved very effective at humbling its reader for me. That doesn't change the fact that I, too, once felt like the uncrowned teenage king of emergency system rescue operations back in 2002 or so, right in my parents' basement. I had started recursively deleting / by accident, and ended up with most directories, /bin and /sbin included, already turned to dust, and most of my as of then proudest achievement - a Gentoo Linux stage 1 installation - gone. I dug myself (and the standing remains of my /home) out of that particular hole by virtue of still having an instance of a browser (I think firefox) running. Mildly panicking, I still left everything untouched ... and after a few hours of thinking hard about what to do, I arrived at the following: I set up a webserver on my mom's windows PC with am embedded PHP interpreter, and hacked up a script that enabled file uploads via HTTP over the LAN. After using that from within my still-running browser (fortunately, it could still list the contents of my home directory) to evac the most precious artifacts my ~ had in store, I rebooted my machine into the lightless nothingness of not having a kernel image in /boot any more. Another few hours later, I had Gentoo reinstalled, and copied the rescued files over to my own machine again - and felt like I had levelled up a little bit.

segfaultbuserr 6y ago
> comparably wizard-like manner, with Kernel hacker Al Viro involved: https://yarchive.net/comp/linux/extreme_system_recovery.html https://yarchive.net/comp/linux/extreme_system_recovery.html If I were him, the best I could imagine was doing documentation reading and ASM writing on a separate computer with working toolchains, testing it, and finally entering some machine code in hex to the broken machine by hand. But he literally wrote an ASM program and constructed an ELF on a totally broken machine itself on the fly, with no documentation, no toolchain. Absolutely wizard. For example, it was how the syscall number was found without grep... > The next step was getting the syscall numbers. grep? We don't need no stinkin' grep. # while read i; do case $i in *__NR_link*) echo $i;; esac; done </usr/include/asm/unistd.h #define __NR_link 9 # while read i; do case $i in *__NR_exit*) echo $i;; esac; done </usr/include/asm/unistd.h #define __NR_exit 1 #define __NR__exit __NR_exit

trm42 6y ago

On day in my previous job our junior support guy started cursing loudly. He had done what he was asked for and did some some small, _almost_ day-to-day support request which was supposed to do some requested maintenance things for some credit card accounts our company was handling. He by some small accident ended up invalidating the cards with our CC operator and closing the accounts on our end. He was so wrecked about it and we, the more senior guys started helping him and comforting him that "yeah, this kind of blunders can happen to anyone, no worries. Let's figure this out together.". The other guys figured how to restore the data into correct setup on our end, and I found some unimplemented badly documented "undo invalidation" endpoint from the CC operator with which we could fix the stuff on the operators end. So we spent nice and fun evening together figuring out the problem as a team, but damn the guy was so torn up about his small and completely understandable fumble that nobody else needed to critique him about it. The funny thing is that with this kind of blunders one learns a lot more and a lot faster than usually. People need to make mistakes. It's usually the best way to learn.

eitland 6y ago
> It's usually the best way to learn. There's a Norwegian saying that damage makes you knowledgeable - but not rich. Learning from others mistakes are very often cheaper and/or less painful.
- cpach 6y ago
  Just out of curiosity: How does that saying read in bokmål? (Or nynorsk) Hilsen fra Sverige (^_^)
  - unhammer 6y ago
    "av skade blir man klok, men sjelden rik" is one variant; there's also the predicative form "klok av skade". (though I'd translate "klok" with "wise")
    - eitland 6y ago

dang 6y ago

If curious see also 2015 https://news.ycombinator.com/item?id=10160417 https://news.ycombinator.com/item?id=10160417 2014 https://news.ycombinator.com/item?id=7892471 https://news.ycombinator.com/item?id=7892471 2010 https://news.ycombinator.com/item?id=1212051 https://news.ycombinator.com/item?id=1212051

sapeint 6y ago

Reminds me of a few years back when I was still new to Linux (and anything non-Windows in general tbqh), and I accidentally nuked /bin. That was a fun time. I happened to have Firefox and an IRC Client opened. I tried a few different things (on the advice of others in an IRC channel I happened to be in), but eventually a friend created a VM of the same Linux Distro and then compressed his /bin and sent it to me -- along with a few required programs to uncompress it just in case. That was a fun day or two. Taught me a few important things though!

danwills 6y ago

I blindly followed a 'news' item's advice once on my home Gentoo Linux that said a particular version of glibc should be removed. Learned the hard way that you definitely want to make sure you have emerged another version of glibc first! Hardly anything works without it (can't even auth as root) Unfortunately unlike the cool cucumbers in this story, I hastily rebooted, and it took a fair while for various reasons to get hold of a boot disk and get it repaired and running again. I love Gentoo but it does take a lot of patience generally, and this kind of thing seems to happen fairly continuously (which teaches you a lot but requires that you want to learn and solve the problems yourself).

simonh 6y ago

I did something like this on a Sun SPARCStation ELC back in the early 90s. I was setting the machine up and was new to Unix and the shell and deleted the contents of the root directory. Not the sub-folders just the files. That doesn't sound so bad right, there's nothing important just in the root directory - except in those days that's where the system kept the kernel binary! No problem, I'd learned how to do network file transfers the day before so I just copied the kernel across from another ELC and rebooted. Phew!

h2odragon 6y ago
I was playing with raw disk access and overwrote selected parts of my kernel image once. oopsie indeed but it was a dev system and so figuring that one out wasn't that bad in terms of impact on others. which was good, cuz it took me a week to figure out.

tragomaskhalos 6y ago

A key takeaway is "don't be root unless you have to be". Horrifies me the number of times at work, even on a production box, that "who" reveals a couple of roots logged on for no good reason. DBAs are the worst offenders.

cafard 6y ago
An old-timey sysadmin I met referred to superuser as "stupiduser".

foxtrottbravo 6y ago

Reminds me of the times I ran `killall` on a SCO machine only to learn that they take the name very literally...

g051051 6y ago

Not quite the same, but one time a guy I worked with decided to "clean up" things on a NCR Unix system and set the ownership of various system files to whatever he thought made the most sense. This broke all the stuff that relied on setuid to function correctly. He then went to lunch while the machine slowly ground to a halt. I had to boot from tape and figure out how to fix things from there.

dtoznayxvf 6y ago

In mid 2000s as a lead HPC sysadmin I accidentally pasted 'shutdown now' into the head node of a 200 CPU HPC cluster, which was 3 weeks into a 12 week simulation job! Fortunately that thing never shut down cleanly and it hung as usual, unable to unmount the NFS share used by all the compute nodes. For the next 9 weeks I was staring at that screen hoping it would keep going and I would not be fired. In the end nobody was ever the wiser.

peter_d_sherman 6y ago

This brings up an interesting issue in OS design, and that is: What are the responsibilities of the OS's Kernel if/when all files are deleted, and all processes are similarly deleted? In other words, the worst-case of all worst-case scenarios. Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel? Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only. Also, we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty" -- for future recycling. Between these two things, if they were implemented, we'd get a lot of leverage to resurrect a dead system, but there's more to this than that. You see, the above scenario should become an active unit test (tested at each recompilation) of that future OS. Also -- we want to know the fewest number of steps necessary (given the above scenario) to fix everything back to normal. In other words, a very interesting topic for future OS designers to think about!

tremon 6y ago
Under that scenario, philosophically, OS-design-wise, what are the responsibilties of the kernel? Practically speaking, none. The responsibility of the kernel (in Linux terms, but I think it's the same across all Unixen) is to start the init system (PID 1), and that's where it ends. Should PID 1 ever terminate, the kernel is in an undefined state. The VFS semantics ensure that even when the init binary is removed from the filesystem, it is not actually unlinked until its reference count is zero -- and since the kernel holds a reference to it, the kernel and init are "safe" from harm through deletion. Well, we might start by designing our future OS with a tiny shell inside of the kernel, for emergency use only. While not in-kernel, systemd is ahead of you, as it offers a shutdown-time ramfs to use as root filesystem so the real root filesystem can be safely unmounted. Previously, Linux didn't really unmount the root filesystem (it couldn't, because the kernel still had an open handle to the init binary), but mounted it read-only before shutdown. we might consider using a file system that doesn't actually delete files when they're deleted, but rather flags the disk space as "empty" This is already the case. "removing" a file in Unix only removes its name from the directory listing, the inode isn't reused until it's been cleared. And even when it's been cleared, only the inode entry is zeroed -- not the data contents. That's why tools like "shred" exist -- to make sure a file's contents are removed along with the file. a very interesting topic for future OS designers to think about I think transactional filesystems, snapshots, and log filesystems already do most of what you're asking for.

janci 6y ago

I managed to break a btrfs filesystem by exhausting metadata blocks. The filesystem would show as 50% occupied, but 0 bytes free. Everything started crashing. The resolution is to run btrfs balance to free unused metadata blocks. There is a catch - you need at least some free space to run the balance. I had none, but managed to remove some unnecessary files (system package manager cache) to have just enough free space to run the balance. In meantime I discovered this condition was created by npm creating thousands of files while installing packages. Later I run into the same problem (again with npm), but this time there was not enough of unnecessary files to delete to be able run the balance (package manager cache was already purged). Normally this can only be solved by adding new partition to the btrfs filesystem and I had no unformatted or unused partiton on disk left. Instead I wanted to get rid of the btrfs that failed me twice and rsynced whole btrfs content into a loop-mounted ext4 disk image on a NTFS drive and then after unmounting I dd-ed the disk image over the btrfs. It booted! Had not any more filesystem trouble ever since.

mickotron 6y ago

Early in my Ops career I wrote a log handling script for Windows servers. It was a re-write of an existing one that pre-dated a lot of us. My version used an xml configuration file, but this iteration of the script did not properly validate the configuration items read from the file. Also, Powershell had this weird xml parsing bug that if a parent had a commented child element, it would show up as an empty child element. So I came in one afternoon shift and my colleague said "your script destroyed two Production servers last night". These servers were critical for monitoring Police radio GPS coordinates. "Impossible! It should only work on directories we configured it for!" I sat there looking at horror at the debug logs, and sure enough, it read the empty child element, changed directory into "", looking for files with "" name pattern, which for Powershell resulted in C:\Windows\system32 and all files. It proceeded to zip and remove "old" files. On a reboot, they did not come back up. To add insult to injury, when they rebuilt those servers manually, the same script with the commented out config was deployed, destroying the servers again. They realised that the script was the issue and decided not to deploy that the second time. So I learned a lot about config validation and defensive coding in the months following the incident.