12 ms·

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "Never admit to anything." I walked into the server room, switched off the PC and restored it from a backup tape that was a day old. 2 Hours later the machine was back on. No one ever noticed, and I never mentioned it either. Moral of the story - Don't let anyone give you access to production servers.

megaman22 9y ago
Did anyone ever notice that it happened?
- 1ba9115454 9y ago
  No one. It was lunchtime and it's possible no ne traded so far that day.
  - charlesdm 9y ago
    Well played
    - brianwawok 9y ago
      The flip side to the story.. what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup? Would be epic fired story. I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.
      - megaman22 9y ago
        I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad driver update or dodgy install of some software. The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
        FiveSquared 9y ago
        Well, I know I how to, and I am fairly young.
LoSboccacc 9y ago
Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on. Since then I’ve adopted custom prompt and color scheme for production. I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
- gumby 9y ago
  > Since then I’ve adopted custom prompt and color scheme for production. An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason. I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
  - lovehashbrowns 9y ago
    I always have so many terminals open that this is a must for me. I have different colors for production and I used ansible to make them consistent. It's nice to not have to waste brain power trying to guess which terminal I'm on.
- stephengillie 9y ago
  > I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management. These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
  - dasil003 9y ago
    I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.
seanwilson 9y ago
This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.
- 3131s 9y ago
  You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes. From man rm: -i prompt before every removal -I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
  - nisse72 9y ago
    This alias seems to be set by default in many linux distributions these days (and has been that way for many years now). The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't. I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
    - chias 9y ago
      > This alias seems to be set by default in many linux distributions these days (and has been that way for many years now) Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
      - nisse72 9y ago
        I stand corrected, it doesn't seem to be the case anymore. I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.
dkersten 9y ago
I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)
- mason55 9y ago
  > I used to setup my shell so that the text on the production servers was red I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
  - DigitalJack 9y ago
    I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point. So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
    - dkersten 9y ago
      That’s a pretty good idea. “Be careful” is a good default.
mfontani 9y ago
To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|
- busterarm 9y ago
  I have a different 'theme' for tmux on any production servers I set up than development/test servers and my local machine. It ends up that I nest my remote sessions in my local, but it's effective.
- mmjaa 9y ago
  Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..
  - foobarchu 9y ago
    Maybe this is an obvious question but...how do you do that if you ssh into your remote machines through a bastion that normalizes everyone to the same end user? I'm unable to set a custom .bashrc on the remote server, because its a shared user, so how would you change the ps1 on the remote box?
humblebee 9y ago
One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no idea if anything was deleted, but he probably had ~ 2PB of mounted data attached to his workstation. Everything from build archives, customer logs dumps, test artifacts, dogfooding clusters (the product was a parallel distributed networked file system), and possibly a few random other things. No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.
- kbenson 9y ago
  This is why --one-file-system should be default, with a warning if another file system is encountered that's skipped so if it's what is desired you can re-run it (or umount and try again).
- throwme211345 9y ago
  Oh fuck moment.
cr0sh 9y ago
> Moral of the story - Don't let anyone give you access to production servers. In my case, it was "Don't let anyone have you do testing/debugging on the production machine." I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines. Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run. Well - at one point I cleared the wrong table - cleared the production and not the test table. Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money. Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened. Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method. Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in. Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).
thomastjeffery 9y ago
> Moral of the story - Don't let anyone give you access to production servers. If you are root, make another user. you can delete it later.

megaman22 9y ago

Did anyone ever notice that it happened?

1ba9115454 9y ago
No one. It was lunchtime and it's possible no ne traded so far that day.
- charlesdm 9y ago
  Well played
  - brianwawok 9y ago
    The flip side to the story.. what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup? Would be epic fired story. I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.
    - megaman22 9y ago
      I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad driver update or dodgy install of some software. The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
      - FiveSquared 9y ago
        Well, I know I how to, and I am fairly young.

LoSboccacc 9y ago

Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on. Since then I’ve adopted custom prompt and color scheme for production. I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

gumby 9y ago
> Since then I’ve adopted custom prompt and color scheme for production. An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason. I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
- lovehashbrowns 9y ago
  I always have so many terminals open that this is a must for me. I have different colors for production and I used ansible to make them consistent. It's nice to not have to waste brain power trying to guess which terminal I'm on.
stephengillie 9y ago
> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management. These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
- dasil003 9y ago
  I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.

seanwilson 9y ago

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

3131s 9y ago
You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes. From man rm: -i prompt before every removal -I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
- nisse72 9y ago
  This alias seems to be set by default in many linux distributions these days (and has been that way for many years now). The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't. I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
  - chias 9y ago
    > This alias seems to be set by default in many linux distributions these days (and has been that way for many years now) Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
    - nisse72 9y ago
      I stand corrected, it doesn't seem to be the case anymore. I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.

dkersten 9y ago

I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)

mason55 9y ago
> I used to setup my shell so that the text on the production servers was red I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
- DigitalJack 9y ago
  I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point. So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
  - dkersten 9y ago
    That’s a pretty good idea. “Be careful” is a good default.

mfontani 9y ago

To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|

busterarm 9y ago
I have a different 'theme' for tmux on any production servers I set up than development/test servers and my local machine. It ends up that I nest my remote sessions in my local, but it's effective.
mmjaa 9y ago
Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..
- foobarchu 9y ago
  Maybe this is an obvious question but...how do you do that if you ssh into your remote machines through a bastion that normalizes everyone to the same end user? I'm unable to set a custom .bashrc on the remote server, because its a shared user, so how would you change the ps1 on the remote box?

humblebee 9y ago

One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no idea if anything was deleted, but he probably had ~ 2PB of mounted data attached to his workstation. Everything from build archives, customer logs dumps, test artifacts, dogfooding clusters (the product was a parallel distributed networked file system), and possibly a few random other things. No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.

kbenson 9y ago
This is why --one-file-system should be default, with a warning if another file system is encountered that's skipped so if it's what is desired you can re-run it (or umount and try again).
throwme211345 9y ago
Oh fuck moment.

cr0sh 9y ago

> Moral of the story - Don't let anyone give you access to production servers. In my case, it was "Don't let anyone have you do testing/debugging on the production machine." I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines. Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run. Well - at one point I cleared the wrong table - cleared the production and not the test table. Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money. Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened. Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method. Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in. Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).

thomastjeffery 9y ago

> Moral of the story - Don't let anyone give you access to production servers. If you are root, make another user. you can delete it later.