4 ms·

Another incredibly common cause for odd system behavior is disk being full. I wonder if there's any checklist with common system problems.

perlgeek 14y ago
I'm not aware of any. Maybe I'll start one. Another favorite of mine is: looking at the right file, but on the wrong host (for example when connected via ssh over a fast network, you simply forget that you're in a remote shell).
- freehunter 14y ago
  I was working in the web interface of one of our appliances at work just before lunch. One of my coworkers (with whom I share admin duties of this appliance) set his computer to shut down then left to pick up some food. Suddenly I noticed the appliance was unresponsive! I was getting site not found errors, and I couldn't even SSH into the machine. I look at my coworkers computer and realize it's still running, but the last command he ran was a shutdown in a remote shell...
  - waverider 14y ago
    Yes, that happens to the best of us: proper command, wrong machine :). Now, shutdown and rm are the worst to add to the mix.
    - freehunter 14y ago
      Don't get me started on how many times I've confused the rename command with the rm command... Hint: it was only once.
  - shabble 14y ago
    That's where molly-guard[1] comes in; stops (or at least confirms) shutdown on protected hosts. Named after the Big Red Button Cover of hacker-lore[2]. I've heard of people doing things like having a terminal background colour escape sequence in their login scripts as well, so it's immediately obvious if you're on a production host. [1] http://manpages.ubuntu.com/manpages/natty/man8/molly-guard.8.html http://manpages.ubuntu.com/manpages/natty/man8/molly-guard.8... [2] http://www.catb.org/jargon/html/M/molly-guard.html http://www.catb.org/jargon/html/M/molly-guard.html
    - jlgaddis 14y ago
      My $PS1 always contains the hostname of the machine I'm SSH'd into IN ALL CAPS so that I never do this (again).
      - shabble 14y ago
        Given the well-established treachery of my finger muscle-memory, I'm much more comfortable with a hard interlock than something that relies on me checking the prompt. Definitely better than nothing though.
scott_karana 14y ago
Similar, but harder to diagnose: running out of inodes.
- wtracy 14y ago
  Oh, man. At my last job, we had a MySQL instance that kept falling over. It turns out that someone was dividing a table up with tons of partitions, and MySQL creates a new data file for each partition. The MySQL instance created several hundred file descriptors while trying to open each one of these files, and "mysteriously" crashed.
jlgaddis 14y ago
In my kickstart scripts, I explicitly do two things: 1. Create a 1 GB file of "nothing" (e.g. dd if=/dev/zero of=/path/meh.bin bs=1000 count=1000000) on each filesystem (exception: /boot, which is typically 256 or 512 MB). 2. Configure each filesystem to reserve 5% of space for the root user. This is typically the default, but I do it explicitly just to make sure. When the first "low disk space" warning comes in, that 5% can be adjusted down to 1% to free up some space (on today's large filesystems, this 4% usually amounts to quite a bit of space). This buys you enough time (hopefully) to expand logical volumes, etc. If you manage to about run out of space after freeing up that 4%, you have one more chance: delete that 1 GB file.

perlgeek 14y ago

I'm not aware of any. Maybe I'll start one. Another favorite of mine is: looking at the right file, but on the wrong host (for example when connected via ssh over a fast network, you simply forget that you're in a remote shell).

freehunter 14y ago
I was working in the web interface of one of our appliances at work just before lunch. One of my coworkers (with whom I share admin duties of this appliance) set his computer to shut down then left to pick up some food. Suddenly I noticed the appliance was unresponsive! I was getting site not found errors, and I couldn't even SSH into the machine. I look at my coworkers computer and realize it's still running, but the last command he ran was a shutdown in a remote shell...
- waverider 14y ago
  Yes, that happens to the best of us: proper command, wrong machine :). Now, shutdown and rm are the worst to add to the mix.
  - freehunter 14y ago
    Don't get me started on how many times I've confused the rename command with the rm command... Hint: it was only once.
- shabble 14y ago
  That's where molly-guard[1] comes in; stops (or at least confirms) shutdown on protected hosts. Named after the Big Red Button Cover of hacker-lore[2]. I've heard of people doing things like having a terminal background colour escape sequence in their login scripts as well, so it's immediately obvious if you're on a production host. [1] http://manpages.ubuntu.com/manpages/natty/man8/molly-guard.8.html http://manpages.ubuntu.com/manpages/natty/man8/molly-guard.8... [2] http://www.catb.org/jargon/html/M/molly-guard.html http://www.catb.org/jargon/html/M/molly-guard.html
  - jlgaddis 14y ago
    My $PS1 always contains the hostname of the machine I'm SSH'd into IN ALL CAPS so that I never do this (again).
    - shabble 14y ago
      Given the well-established treachery of my finger muscle-memory, I'm much more comfortable with a hard interlock than something that relies on me checking the prompt. Definitely better than nothing though.

scott_karana 14y ago

Similar, but harder to diagnose: running out of inodes.

wtracy 14y ago
Oh, man. At my last job, we had a MySQL instance that kept falling over. It turns out that someone was dividing a table up with tons of partitions, and MySQL creates a new data file for each partition. The MySQL instance created several hundred file descriptors while trying to open each one of these files, and "mysteriously" crashed.

jlgaddis 14y ago

In my kickstart scripts, I explicitly do two things: 1. Create a 1 GB file of "nothing" (e.g. dd if=/dev/zero of=/path/meh.bin bs=1000 count=1000000) on each filesystem (exception: /boot, which is typically 256 or 512 MB). 2. Configure each filesystem to reserve 5% of space for the root user. This is typically the default, but I do it explicitly just to make sure. When the first "low disk space" warning comes in, that 5% can be adjusted down to 1% to free up some space (on today's large filesystems, this 4% usually amounts to quite a bit of space). This buys you enough time (hopefully) to expand logical volumes, etc. If you manage to about run out of space after freeing up that 4%, you have one more chance: delete that 1 GB file.