4 ms·

>The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Substitute r

>The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Substitute reboot with 'upgrade to win 7' and data center with university at get a story from a month? ago. rm ./*.* - delete all files in current dir rm /*.* - never do this Can you spot the difference? A colleague did the later on our testing server. No clients were disturbed, but us devs were left working on things that can run locally (much more pleasant stuff for sure, yet the schedule suffered a lot).

adamfeldman 12y ago
The incident was at Emory University: http://thenextweb.com/shareables/2014/05/16/emory-university-server-accidentally-sends-reformat-request-windows-pcs-including/ http://thenextweb.com/shareables/2014/05/16/emory-university...
jacquesm 12y ago
Your 'never do this' line would erase initrd.img on most machines, which you'd only find out about after the next reboot if nobody told you about it (and good luck getting that fixed). There are a large number of varieties of this particular error. Some with terrible results. rm -rf * .bak for instance (especially when executed in the root directory). That '#' prompt is there for a reason. The way to solve these sort of issues is to first get the files using 'find' until you're totally happy about the result and then to use 'rm' as the command passed to find. Of course, nobody does this ;)
- jethro_tell 12y ago
  >Your 'never do this' line Would also erase everything else on the box so you might not know for a few seconds, but then the database errors start and the lib and pid files start disappearing and now your the king of a mountain of shit. It won't take till next reboot to notice. If you ever get a chance, rm -rf / a box before you throw it out and just play around with it for a couple minutes while it eats itself. You can spare your self most errors like that with tab complete, the built in sanity checker. I tab complete everything since I'm dyslexic, and it saves me a shit ton of time cause I'm never far from the error when I notice it.
  - dbaupp 12y ago
    You have the same fun in an emulator (even http://bellard.org/jslinux/ http://bellard.org/jslinux/ is somewhat good enough).
  - jsmthrowaway 12y ago
    > Would also erase everything else on the box so you might not know for a few seconds, but then the database errors start and the lib and pid files start disappearing and now your the king of a mountain of shit. It won't take till next reboot to notice. Yes, it will, because the original post didn't include -r. So it's only deleting things that match the glob in the root directory. On many systems, that is nothing.
    - eurleif 12y ago
      Even with -r, /*.* only matches files and directories in / that have a period in their name. So it would skip over /usr, /home, etc.
      - jsmthrowaway 12y ago
        That's what "match the glob" means.
        eurleif 12y ago
        My point was that the lack of -r doesn't really change anything; even with -r, it would still only delete initrd.img and similar, which would take until the next reboot to notice.
- nl 12y ago
  >rm -rf * .bak Try chown -R user:group /var/local/some/data/dir / inside an init script. Of course it had been tested many times, and the space was only fat-fingered in when someone move the data directory.
- cordite 12y ago
  Another one with a space is from the bumblebee install script where it removed /usr Instead of /usr/some/path/here http://www.miltonbayer.com/blog/news/when-a-code-commit-goes-viral/ http://www.miltonbayer.com/blog/news/when-a-code-commit-goes...
aidos 12y ago
Ah yes. There are so many ways to bork a box. A colleague of mine once called me over because they were having trouble with their computer. Apparently while doing "a little tidying up" they decided to move the windows folder to somewhere else. I can't remember the details but for some reason it was impossible to get a command prompt (Windows NT maybe?) without a disk, which I didn't have. I think I had to rip the drive out and stick it in another machine to fix it.
Negitivefrags 12y ago
I once went to shutdown our staging realm to do some task. ~/realm_control_production$ ./realmctl.py stop --no-warn Oh shit, that is production, not staging. I got the surge of adrenaline and hit Ctrl+C within a couple of seconds. Thankfully it only completed the very first part of the process that prevents new users from logging in and puts up the maintenance page. No playing users were kicked off the game servers. All I had to do was run ./realmctl.py allow to get new logins to start working again. This is what taught us that we need to have an extra confirmation for actions on the production realm.
- will_hughes 12y ago
  So, it's now like this? ~/realm_control_production$ ./realmctl.py stop --no-warn --no-really --i-know-its-production
- protomyth 12y ago
  That is why my production terminal windows are red background with yellow text, because I sometimes "space out" the prompt.
  - mickeyp 12y ago
    This is good advice. I have always followed that through by insisting $PS1 has a bright red banner that says "PRODUCTION".
    - protomyth 12y ago
      I like that one, I will have to try that out. A damn shame you cannot add animation to the prompt.
susi22 12y ago
You know what's also fun. Try deleting all files that start with a . in your current directory: rm -rf .* Would you expect this to go UPWARD? I would have never. I.e. even if you're in /x/y/z it will traverse up the tree and start deleting /x and eventually /.
- vacri 12y ago
  '..' starts with '.', but I wouldn't have expected it to go more than one level up. I guess it saw ../.., then ../../.. ... These days, any time I do an rm with a wildcard, I always prepend with a directory, no matter how trivial, just to keep in the habit. 'rm ../mydir/.*'

adamfeldman 12y ago

The incident was at Emory University: http://thenextweb.com/shareables/2014/05/16/emory-university-server-accidentally-sends-reformat-request-windows-pcs-including/ http://thenextweb.com/shareables/2014/05/16/emory-university...

jacquesm 12y ago

Your 'never do this' line would erase initrd.img on most machines, which you'd only find out about after the next reboot if nobody told you about it (and good luck getting that fixed). There are a large number of varieties of this particular error. Some with terrible results. rm -rf * .bak for instance (especially when executed in the root directory). That '#' prompt is there for a reason. The way to solve these sort of issues is to first get the files using 'find' until you're totally happy about the result and then to use 'rm' as the command passed to find. Of course, nobody does this ;)

jethro_tell 12y ago
>Your 'never do this' line Would also erase everything else on the box so you might not know for a few seconds, but then the database errors start and the lib and pid files start disappearing and now your the king of a mountain of shit. It won't take till next reboot to notice. If you ever get a chance, rm -rf / a box before you throw it out and just play around with it for a couple minutes while it eats itself. You can spare your self most errors like that with tab complete, the built in sanity checker. I tab complete everything since I'm dyslexic, and it saves me a shit ton of time cause I'm never far from the error when I notice it.
- dbaupp 12y ago
  You have the same fun in an emulator (even http://bellard.org/jslinux/ http://bellard.org/jslinux/ is somewhat good enough).
- jsmthrowaway 12y ago
  > Would also erase everything else on the box so you might not know for a few seconds, but then the database errors start and the lib and pid files start disappearing and now your the king of a mountain of shit. It won't take till next reboot to notice. Yes, it will, because the original post didn't include -r. So it's only deleting things that match the glob in the root directory. On many systems, that is nothing.
  - eurleif 12y ago
    Even with -r, /*.* only matches files and directories in / that have a period in their name. So it would skip over /usr, /home, etc.
    - jsmthrowaway 12y ago
      That's what "match the glob" means.
      - eurleif 12y ago
        My point was that the lack of -r doesn't really change anything; even with -r, it would still only delete initrd.img and similar, which would take until the next reboot to notice.
nl 12y ago
>rm -rf * .bak Try chown -R user:group /var/local/some/data/dir / inside an init script. Of course it had been tested many times, and the space was only fat-fingered in when someone move the data directory.
cordite 12y ago
Another one with a space is from the bumblebee install script where it removed /usr Instead of /usr/some/path/here http://www.miltonbayer.com/blog/news/when-a-code-commit-goes-viral/ http://www.miltonbayer.com/blog/news/when-a-code-commit-goes...

aidos 12y ago

Ah yes. There are so many ways to bork a box. A colleague of mine once called me over because they were having trouble with their computer. Apparently while doing "a little tidying up" they decided to move the windows folder to somewhere else. I can't remember the details but for some reason it was impossible to get a command prompt (Windows NT maybe?) without a disk, which I didn't have. I think I had to rip the drive out and stick it in another machine to fix it.

Negitivefrags 12y ago

I once went to shutdown our staging realm to do some task. ~/realm_control_production$ ./realmctl.py stop --no-warn Oh shit, that is production, not staging. I got the surge of adrenaline and hit Ctrl+C within a couple of seconds. Thankfully it only completed the very first part of the process that prevents new users from logging in and puts up the maintenance page. No playing users were kicked off the game servers. All I had to do was run ./realmctl.py allow to get new logins to start working again. This is what taught us that we need to have an extra confirmation for actions on the production realm.

will_hughes 12y ago
So, it's now like this? ~/realm_control_production$ ./realmctl.py stop --no-warn --no-really --i-know-its-production
protomyth 12y ago
That is why my production terminal windows are red background with yellow text, because I sometimes "space out" the prompt.
- mickeyp 12y ago
  This is good advice. I have always followed that through by insisting $PS1 has a bright red banner that says "PRODUCTION".
  - protomyth 12y ago
    I like that one, I will have to try that out. A damn shame you cannot add animation to the prompt.

susi22 12y ago

You know what's also fun. Try deleting all files that start with a . in your current directory: rm -rf .* Would you expect this to go UPWARD? I would have never. I.e. even if you're in /x/y/z it will traverse up the tree and start deleting /x and eventually /.

vacri 12y ago
'..' starts with '.', but I wouldn't have expected it to go more than one level up. I guess it saw ../.., then ../../.. ... These days, any time I do an rm with a wildcard, I always prepend with a directory, no matter how trivial, just to keep in the habit. 'rm ../mydir/.*'