3 ms·

Top commenter on that article: First of all... If you're getting paged on disk alerts at 3 in the morning, you're doing it wrong. Write a script that check

by feld 11y ago

Top commenter on that article: First of all... If you're getting paged on disk alerts at 3 in the morning, you're doing it wrong. Write a script that checks every file system and makes sure that X amount of free space is always present, adding free space as-needed, and emailing you on the backend during business hours whenever the volume group is nearing depletion. I'm sorry, but no -- I'm not going to automate adding more disk to a VM, extending a volume group, and growing a filesystem. Not now, not ever. No way in hell. That needs to be done by a human after a backup has been tested and writes have been quiesced. This is what grown-ups do. Storage is cheap. Downtime is not. Then I don't want to grow up like you. Have fun recovering your corrupted filesystem.

digi_owl 11y ago
Welcome to the devops world, where admin is boring and coding is fun...
- trcollinson 11y ago
  That's a falacy of an argument really. It's not as if admins can't properly script environments and changes. Monitoring, updating, and scripting of changes happened long before the term devops came into existence.
  - digi_owl 11y ago
    http://www.commitstrip.com/en/2015/07/08/true-story-fixing-a-self-ddos/ http://www.commitstrip.com/en/2015/07/08/true-story-fixing-a... The "just add more to the pool" solution seems akin to the "solution" in the strip above. And i have seen similar arguments for when a daemon or server goes down. Hook it up to a script that reboots it automatically (or fires up a new instance over and over and over) and go back to coding.
- toomuchtodo 11y ago
  Eh, sort of. I do Devops/Infrastructure exclusively, and while the tedious stuff is automated away, I've seen too much shit break on the other side of the fence to trust it fully (AWS API throwing errors when you absolutely need it to work, etc). If something is broken, you should have enough automation to get it into a known good state without a human involved, while maintaining data consistency. Anything else should be automated, but you should still be around to babysit it while its going through the motions. If you think you can automate everything and always trust it to work flawlessly, you haven't been around long enough for the edge cases.
  - digi_owl 11y ago
    > If you think you can automate everything and always trust it to work flawlessly, you haven't been around long enough for the edge cases. < And then you have some "clever" coder coming along with a fix for said edge case (that invariably create some new edge cases just outside the domain of the fix).
trcollinson 11y ago
I'm not the person who posted that comment, but let's take a step back. I have scripts which do this for me. However, I also have automation that takes back ups of the systems, restores the systems on other virtual hosts, and verifies that the restores are up and running properly. In fact if you can script one, you can generally script all of those scenarios. For expanding database volumes, for example, I have a system which first monitors the size of my current database systems. If they get too large, it will create a new virtual which is bigger, restores a current backup of the running system to it, verify that, put it into rotation, make sure that all current data is replicated, then remove the old system. Frankly, doing this by hand would be highly error prone and dangerous. I say, script everything.
- feld 11y ago
  Of course I have scripts to do that stuff, but considering the potential danger involved and being burned so many times in the past from bad filesystems/tools/kernels it is something I would only initiate manually -- never as a knee-jerk reaction to "low disk space" alerts.
dang 11y ago
Please don't copy-paste a comment from elsewhere in order to flame it on Hacker News. The local supply is excessive as it is. The substantive point you're making is fine, but would be better and clearer if you expanded on it a bit and dropped the indignation.
brazzledazzle 11y ago
I know it's risky to expand an existing disk online but isn't adding additional disks to the volume group pretty straightforward?

by feld 11y ago

digi_owl 11y ago

Welcome to the devops world, where admin is boring and coding is fun...

trcollinson 11y ago
That's a falacy of an argument really. It's not as if admins can't properly script environments and changes. Monitoring, updating, and scripting of changes happened long before the term devops came into existence.
- digi_owl 11y ago
  http://www.commitstrip.com/en/2015/07/08/true-story-fixing-a-self-ddos/ http://www.commitstrip.com/en/2015/07/08/true-story-fixing-a... The "just add more to the pool" solution seems akin to the "solution" in the strip above. And i have seen similar arguments for when a daemon or server goes down. Hook it up to a script that reboots it automatically (or fires up a new instance over and over and over) and go back to coding.
toomuchtodo 11y ago
Eh, sort of. I do Devops/Infrastructure exclusively, and while the tedious stuff is automated away, I've seen too much shit break on the other side of the fence to trust it fully (AWS API throwing errors when you absolutely need it to work, etc). If something is broken, you should have enough automation to get it into a known good state without a human involved, while maintaining data consistency. Anything else should be automated, but you should still be around to babysit it while its going through the motions. If you think you can automate everything and always trust it to work flawlessly, you haven't been around long enough for the edge cases.
- digi_owl 11y ago
  > If you think you can automate everything and always trust it to work flawlessly, you haven't been around long enough for the edge cases. < And then you have some "clever" coder coming along with a fix for said edge case (that invariably create some new edge cases just outside the domain of the fix).

trcollinson 11y ago

I'm not the person who posted that comment, but let's take a step back. I have scripts which do this for me. However, I also have automation that takes back ups of the systems, restores the systems on other virtual hosts, and verifies that the restores are up and running properly. In fact if you can script one, you can generally script all of those scenarios. For expanding database volumes, for example, I have a system which first monitors the size of my current database systems. If they get too large, it will create a new virtual which is bigger, restores a current backup of the running system to it, verify that, put it into rotation, make sure that all current data is replicated, then remove the old system. Frankly, doing this by hand would be highly error prone and dangerous. I say, script everything.

feld 11y ago
Of course I have scripts to do that stuff, but considering the potential danger involved and being burned so many times in the past from bad filesystems/tools/kernels it is something I would only initiate manually -- never as a knee-jerk reaction to "low disk space" alerts.

dang 11y ago

Please don't copy-paste a comment from elsewhere in order to flame it on Hacker News. The local supply is excessive as it is. The substantive point you're making is fine, but would be better and clearer if you expanded on it a bit and dropped the indignation.

brazzledazzle 11y ago

I know it's risky to expand an existing disk online but isn't adding additional disks to the volume group pretty straightforward?