3 ms·

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attent

by patio11 12y ago

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only discovered the next time the server was restarted. As you can imagine, inability to boot the database has unpleasant consequences for web apps connected to it.

jey 12y ago
Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.
- thaumaturgy 12y ago
  You would think so (and I won't say it isn't), but if you're just one guy handling a few servers for your hosted SaaS, it's easy to fall into the "don't disturb the dust" trap -- especially once you've had a routine maintenance reboot or upgrade completely bone you. Not too long ago I had a 36-hour straight marathon sysadmin session when a routine update happened to break my particular stack without advance notice on a web server used by a bunch of customers. I think my eye still twitches when I think of it. Usually the next thing people say is, "you should set up x, y, and z tools for better redundancy / failover / load distribution / backups / sysadmin management / etc." Again, they're not wrong, but none of those things directly makes you more money, and making time for them can be difficult, especially since so few of them can be set up as easily as they claim on the box.
  - rsync 12y ago
    Don't disturb the dust is actually a fairly good heuristic. It sure would have helped those guys working at Chernobyl.
- ownagefool 12y ago
  Rather than doing that, manage your server(s) with code. While there is a little overhead in learning something like ansible, being able to provision another machine to do a dry-run more than makes up for that cost.
Maxious 12y ago
Two things like this; my tomcat config got reset to max 128mb JVM stack and ubuntu seems to have switched from -virtual to -generic kernels but the latter isn't booting for me.
jleader 12y ago
I've never been a "real" sysadmin, but back in the mid 80s I learned that anything you don't practice doing regularly will inevitably be painful. I was working as a developer at Unisys (or it might have still been Burroughs), down the hall from the OS developers (for one of Burroughs' 3 distinct mainframe product lines, each with its own cpu architecture, operating system, system software, etc.). At that time, our mainframes could be patched while running, without re-booting; re-booting was considerably slower than patching. As a result, as each new OS patch was developed, it was applied on the fly to the machines in our dev environment, and they hadn't been rebooted for something like six months. It turned out that it was possible to have a sequence of patches, which when applied on the fly to a running system caused no problems, except that when it came time to reboot, it would crash. You can imagine the fun of trying to debug which change (or combination of several changes), out of many months of development work, had broken the boot process! These dev machines were shared with other teams (compiler developers, database engine developers, etc.), so the productivity hit waiting for our dev machines to be available again was substantial. After that, the policy was that the OS developers took turns coming in early one morning a week to reboot the dev machines, so in the worst case they'd only have to track down which of the last week's worth of patches might have broken the boot process.
tomjen3 12y ago
This isn't to handle the bash problem? What is it to fix then?
- notatoad 12y ago
  nobody is saying yet, but probably the xen issue that's scheduled for disclosure on october 1
- rmc 12y ago
  There is a bug in Xen, which hasn't been publically annouced yet.
mikegioia 12y ago
pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem I don't understand. apt-get update won't install new software on reboot. That just updates the index in apt. There's no other way to update bash without "apt-get update && apt-get install bash" and that won't impact anything on reboot. Your updated mysql config file sounds like you apt-get upgrade'd, installed a new version of mysql, or something like that and the changes didn't go into effect until the mysql process was restarted. I got bit in the ass with that same thing when I apt-get upgrade'd.
- patio11 12y ago
  Good thing I've I never totally hosed myself when picking the wrong word/letter in a command line as opposed to a non-executable HN comment. cries softly to self

by patio11 12y ago

jey 12y ago

Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.

thaumaturgy 12y ago
You would think so (and I won't say it isn't), but if you're just one guy handling a few servers for your hosted SaaS, it's easy to fall into the "don't disturb the dust" trap -- especially once you've had a routine maintenance reboot or upgrade completely bone you. Not too long ago I had a 36-hour straight marathon sysadmin session when a routine update happened to break my particular stack without advance notice on a web server used by a bunch of customers. I think my eye still twitches when I think of it. Usually the next thing people say is, "you should set up x, y, and z tools for better redundancy / failover / load distribution / backups / sysadmin management / etc." Again, they're not wrong, but none of those things directly makes you more money, and making time for them can be difficult, especially since so few of them can be set up as easily as they claim on the box.
- rsync 12y ago
  Don't disturb the dust is actually a fairly good heuristic. It sure would have helped those guys working at Chernobyl.
ownagefool 12y ago
Rather than doing that, manage your server(s) with code. While there is a little overhead in learning something like ansible, being able to provision another machine to do a dry-run more than makes up for that cost.

Maxious 12y ago

Two things like this; my tomcat config got reset to max 128mb JVM stack and ubuntu seems to have switched from -virtual to -generic kernels but the latter isn't booting for me.

jleader 12y ago

I've never been a "real" sysadmin, but back in the mid 80s I learned that anything you don't practice doing regularly will inevitably be painful. I was working as a developer at Unisys (or it might have still been Burroughs), down the hall from the OS developers (for one of Burroughs' 3 distinct mainframe product lines, each with its own cpu architecture, operating system, system software, etc.). At that time, our mainframes could be patched while running, without re-booting; re-booting was considerably slower than patching. As a result, as each new OS patch was developed, it was applied on the fly to the machines in our dev environment, and they hadn't been rebooted for something like six months. It turned out that it was possible to have a sequence of patches, which when applied on the fly to a running system caused no problems, except that when it came time to reboot, it would crash. You can imagine the fun of trying to debug which change (or combination of several changes), out of many months of development work, had broken the boot process! These dev machines were shared with other teams (compiler developers, database engine developers, etc.), so the productivity hit waiting for our dev machines to be available again was substantial. After that, the policy was that the OS developers took turns coming in early one morning a week to reboot the dev machines, so in the worst case they'd only have to track down which of the last week's worth of patches might have broken the boot process.

tomjen3 12y ago

This isn't to handle the bash problem? What is it to fix then?

notatoad 12y ago
nobody is saying yet, but probably the xen issue that's scheduled for disclosure on october 1
rmc 12y ago
There is a bug in Xen, which hasn't been publically annouced yet.

mikegioia 12y ago

pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem I don't understand. apt-get update won't install new software on reboot. That just updates the index in apt. There's no other way to update bash without "apt-get update && apt-get install bash" and that won't impact anything on reboot. Your updated mysql config file sounds like you apt-get upgrade'd, installed a new version of mysql, or something like that and the changes didn't go into effect until the mysql process was restarted. I got bit in the ass with that same thing when I apt-get upgrade'd.

patio11 12y ago
Good thing I've I never totally hosed myself when picking the wrong word/letter in a command line as opposed to a non-executable HN comment. cries softly to self