5 ms·

Over 15 years ago, as a teenager, I taught Linux / UNIX Admin courses, and worked as a consultant advising folks, and in the late 90s I was very adamant that yo

Over 15 years ago, as a teenager, I taught Linux / UNIX Admin courses, and worked as a consultant advising folks, and in the late 90s I was very adamant that you should never -9 anything unless you know exactly what you are doing. As infrastructures have grown, and I have managed large applications involving tens to hundreds, often over a thousand servers, and I have grown to accept that a power supply can fail and a node can disappear from the network and it's even possible that none of its' components, including its' drives, will ever work again. I've never _really_ experienced such a catastrophic failure, but it's a lot easier to sleep at night if you just assume that. kill -9 should never be worse than pulling the power plug, which is what netflix's chaos monkey always tries to simulate. we all have to live on a continuum of how much of that we can survive, but if you always assume abrubt failure, it'll be pretty tough to give you a bad day.

justizin 12y ago
To follow this up: test these situations, design yourself into resilience. obviously, you can't build perfection in a day, but for anything you rely on, be it mysql, redis, memcached, postgresql, rabbitmq, whatever, just kill -9 that shit in staging periodically while a few people hit the url, and see what happens, what logs, etc.. assume failure.
vacri 12y ago
kill -9 should never be worse than pulling the power plug No, it shouldn't, but just pulling the power plug isn't exactly recommended behaviour, either. There aren't many admins out there who will happily yank the power cord out of their desktops when they want to power it down. Last night I spent several hours getting a server back into gear after a 'pulled power plug' event. A friend's rack was affected by a (seven-hour!) substation power outage, and it wasn't on a UPS, so it didn't shut down cleanly. Eventually we were able to coax the server to boot again (had to remove all USB devices in the process, including internal ones), and the problem was a corrupted MBR. Make a rescue usb stick, boot into that, finally diagnose the problem, cat a new MBR onto it, and tada, fixed. Let's just say that I don't find the argument "shouldn't be worse than just pulling the plug" to be particularly comforting at the moment :)
- userbinator 12y ago
  Were you writing to the MBR somehow when it died? If not, that looks like really badly designed hardware to me. I've had abrupt shutdowns happen on laptops (one had a particularly loose battery...), desktops, servers and although have encountered corrupt files and filesystems, never had any of them corrupt the MBR.
  - vacri 12y ago
    It's unclear what caused the MBR problems - the power outage was a couple of days before, and the system seemed to come back okay. My friend was busy with work and a cursory check had it clear, but yesterday things started acting funny, he logged in and the load was 40 and rising before it became unresponsive to his diagnostics. This machine had been running happily for quite some time before the powerout event (it's basically just a kvm host, the fun stuff is on the guests), so it's particularly puzzling. Somehow the MBR was overwritten with a syslinux one, and he says that box had never had syslinux used on it (extlinux, yes). The root cause will become evident at some point, it just needs some head-scratching time.
    - userbinator 12y ago
      I almost read your last sentence as "The root kit will become evident at some point", because that's what came to mind with those symptoms. I'd check for an infection.
phireal 12y ago
<offtopic> I know that grammar comments are probably not welcome here on HN, but I think that since you seem to have an interest in it, I'd point this out: "Its" only has an apostrophe when it's a contraction of "it is" (or "it has"), the possessive is always "its" (without an apostrophe). </offtopic>
jeffdavis 12y ago
"kill -9 should never be worse than pulling the power plug" It's actually easy to misdesign a system such that it is safe against power failures but not kill -9. See my other comment: https://news.ycombinator.com/item?id=7793301 https://news.ycombinator.com/item?id=7793301

justizin 12y ago

To follow this up: test these situations, design yourself into resilience. obviously, you can't build perfection in a day, but for anything you rely on, be it mysql, redis, memcached, postgresql, rabbitmq, whatever, just kill -9 that shit in staging periodically while a few people hit the url, and see what happens, what logs, etc.. assume failure.

vacri 12y ago

kill -9 should never be worse than pulling the power plug No, it shouldn't, but just pulling the power plug isn't exactly recommended behaviour, either. There aren't many admins out there who will happily yank the power cord out of their desktops when they want to power it down. Last night I spent several hours getting a server back into gear after a 'pulled power plug' event. A friend's rack was affected by a (seven-hour!) substation power outage, and it wasn't on a UPS, so it didn't shut down cleanly. Eventually we were able to coax the server to boot again (had to remove all USB devices in the process, including internal ones), and the problem was a corrupted MBR. Make a rescue usb stick, boot into that, finally diagnose the problem, cat a new MBR onto it, and tada, fixed. Let's just say that I don't find the argument "shouldn't be worse than just pulling the plug" to be particularly comforting at the moment :)

userbinator 12y ago
Were you writing to the MBR somehow when it died? If not, that looks like really badly designed hardware to me. I've had abrupt shutdowns happen on laptops (one had a particularly loose battery...), desktops, servers and although have encountered corrupt files and filesystems, never had any of them corrupt the MBR.
- vacri 12y ago
  It's unclear what caused the MBR problems - the power outage was a couple of days before, and the system seemed to come back okay. My friend was busy with work and a cursory check had it clear, but yesterday things started acting funny, he logged in and the load was 40 and rising before it became unresponsive to his diagnostics. This machine had been running happily for quite some time before the powerout event (it's basically just a kvm host, the fun stuff is on the guests), so it's particularly puzzling. Somehow the MBR was overwritten with a syslinux one, and he says that box had never had syslinux used on it (extlinux, yes). The root cause will become evident at some point, it just needs some head-scratching time.
  - userbinator 12y ago
    I almost read your last sentence as "The root kit will become evident at some point", because that's what came to mind with those symptoms. I'd check for an infection.

phireal 12y ago

<offtopic> I know that grammar comments are probably not welcome here on HN, but I think that since you seem to have an interest in it, I'd point this out: "Its" only has an apostrophe when it's a contraction of "it is" (or "it has"), the possessive is always "its" (without an apostrophe). </offtopic>

jeffdavis 12y ago

"kill -9 should never be worse than pulling the power plug" It's actually easy to misdesign a system such that it is safe against power failures but not kill -9. See my other comment: https://news.ycombinator.com/item?id=7793301 https://news.ycombinator.com/item?id=7793301