4 ms·

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

by dilyevsky 6y ago

geocrasher 6y ago
In a perfect world, this is true. But we don't have one of those.
- dilyevsky 6y ago
  That’s not perfect world this is standard industry practice to have host level monitoring
  - akdma 6y ago
    THANK YOU. How are so many people in this thread content with saying “monitoring isn’t perfect, this solution is ingenious”. Ofc nothing is perfect and even when you do everything right things can still go wrong, but if you don’t have a ROBUST monitoring/alert system in place then you’re not even doing the bare minimum. They’re acting like it’s rocket science to set thresholds, and have meaningful alerts and checks in place. Not to mention if you wait until disk full you risk issues like block corruption among others and your 8GB of space doesn’t do anything. It’s why people in this industry are on call, it’s why they have monitoring on their monitoring systems. The bare minimum
    - dilyevsky 6y ago
      Yeah it’s crazy. If someone does this on their homelab server it’s probably fine but if they run it in production I really want to know because Im not buying jack from them.
  - geocrasher 6y ago
    Of course! But do you put all your trust in your monitoring, 100%? You've never had monitoring fail for any reason at all? You've never had a server fill up before you can respond to the alert? This 8gb file idea isn't to replace monitoring. It's to offer a quick stopgap solution so you can do things in a hurry and give yourself a little extra "out" when things go awry. Because believe me, they WILL go awry. And if you're not prepared for that eventuality, then I don't know what else to say.
    - luckylion 6y ago
      > But do you put all your trust in your monitoring, 100%? Yes. If I didn't feel that I can trust it, I would get another solution. > You've never had a server fill up before you can respond to the alert? I have. With the proposed hack in this article: it would fill up even faster: by that amount of time it would take the problem to write 8gb of data. > Because believe me, they WILL go awry. In my experience: not in any way that this would help. If your disk fills up, it's either slow (and your monitoring alerts you days or at least hours before it's a problem) or it's really, really fast. In the latter case, it's much faster than you can jump on your computer, ssh into the machine and delete your spacer file. Invest in better monitoring, that's much, much, much, much better than adding spacer files to fill up your disk or changing the wall clock to give you more time.
      - geocrasher 6y ago
        Ah I see where you are coming from. You see the spacer as a way to prevent a problem that should be prevented by better monitoring. But that's not what it is for. It's for quickly providing a stopgap so that you have time to solve the root cause without enduring more downtime. If you've had a disk go full on you, what's the first thing you do? For me, I log in and start looking for a log file to truncate to buy me a few megs of space, at least. This spacer file is just a guaranteed way to find the space you need without having to hunt for it. Also it doesn't HAVE to be 8GB. On most systems I think a 500mb file would be every bit as effective.
deleted 6y ago
[deleted]
berkes 6y ago
"proper monitoring" is extremely broad. And, I would say, almost unreachable goal. You have it mail you when it goes over 80% disk usage (and what if you are on holiday)? Does it mail all colleagues? Who picks it up (I thought Bob picked it up, but Bob thought Anne picked it up. So no one did)? Does it come and wake you in person when it reaches 92%? Will this catch this async job that fails (but should never) in an endless loop but keeps creating 20MB json files as fast as the disk allows it to? Is it an alerting that finds anomalies in trends? Will it be fast enough for you to come online before that job has filled the disk? I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.
- geocrasher 6y ago
  > I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen. I work in hosting too, and have been for a long time. I feel ya.
- dilyevsky 6y ago
  Slack warning/ticket at 75%, page at 85% (to oncall obviously). Don’t let user workload crap into your root partition. I’ve been doing this for over 10 years and managed many thousands of nodes and literally don’t recall full disk problem unless it was in staging somewhere where monitoring was deliberately disabled.
  - berkes 6y ago
    Your requirements for "proper monitoring" are not everyones requirements. On a current gig, we host at heroku. Our monitoring is all about 95th percentile response-times, secondary services, backlogs, slow-queries and whatnots. For another job, "disk space filling up" is important. Again another job will need to monitor email-delivery-rates and so on and so forth.
jabroni_salad 6y ago
Keep in mind that sysadmins are essentially babysitting software that they do not develop. The hacks that we come up with are to work around responsible party and help us get a good night's sleep instead of a 2am wakeup call. I try to cut you guys some slack, usually this proliferates when management decides they are willing to accept some inefficiency in favor of getting new features out the door. I get it, really. My org is in the middle of a SRE introduction and for some reason I'm getting a lot of pushback on the topic of 'error budgets' and what to do with alerts when they are exceeded. Can't imagine why.
- dilyevsky 6y ago
  How does using proposed solution prevents a 2am wake up call? Your monitoring/alerting does, this just makes it easier to recover already broken software. And btw I’ve been carrying pagers for more than a decade so well aware of all the organizational dynamics here. Best way to prevent this is have devs carrying pager too (amazons “you built it you run it”) - and magically your nighttime oncall is much more pleasant ;-)

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

geocrasher 6y ago

In a perfect world, this is true. But we don't have one of those.

dilyevsky 6y ago
That’s not perfect world this is standard industry practice to have host level monitoring
- akdma 6y ago
  THANK YOU. How are so many people in this thread content with saying “monitoring isn’t perfect, this solution is ingenious”. Ofc nothing is perfect and even when you do everything right things can still go wrong, but if you don’t have a ROBUST monitoring/alert system in place then you’re not even doing the bare minimum. They’re acting like it’s rocket science to set thresholds, and have meaningful alerts and checks in place. Not to mention if you wait until disk full you risk issues like block corruption among others and your 8GB of space doesn’t do anything. It’s why people in this industry are on call, it’s why they have monitoring on their monitoring systems. The bare minimum
  - dilyevsky 6y ago
    Yeah it’s crazy. If someone does this on their homelab server it’s probably fine but if they run it in production I really want to know because Im not buying jack from them.
- geocrasher 6y ago
  Of course! But do you put all your trust in your monitoring, 100%? You've never had monitoring fail for any reason at all? You've never had a server fill up before you can respond to the alert? This 8gb file idea isn't to replace monitoring. It's to offer a quick stopgap solution so you can do things in a hurry and give yourself a little extra "out" when things go awry. Because believe me, they WILL go awry. And if you're not prepared for that eventuality, then I don't know what else to say.
  - luckylion 6y ago
    > But do you put all your trust in your monitoring, 100%? Yes. If I didn't feel that I can trust it, I would get another solution. > You've never had a server fill up before you can respond to the alert? I have. With the proposed hack in this article: it would fill up even faster: by that amount of time it would take the problem to write 8gb of data. > Because believe me, they WILL go awry. In my experience: not in any way that this would help. If your disk fills up, it's either slow (and your monitoring alerts you days or at least hours before it's a problem) or it's really, really fast. In the latter case, it's much faster than you can jump on your computer, ssh into the machine and delete your spacer file. Invest in better monitoring, that's much, much, much, much better than adding spacer files to fill up your disk or changing the wall clock to give you more time.
    - geocrasher 6y ago
      Ah I see where you are coming from. You see the spacer as a way to prevent a problem that should be prevented by better monitoring. But that's not what it is for. It's for quickly providing a stopgap so that you have time to solve the root cause without enduring more downtime. If you've had a disk go full on you, what's the first thing you do? For me, I log in and start looking for a log file to truncate to buy me a few megs of space, at least. This spacer file is just a guaranteed way to find the space you need without having to hunt for it. Also it doesn't HAVE to be 8GB. On most systems I think a 500mb file would be every bit as effective.

deleted 6y ago

[deleted]

berkes 6y ago

"proper monitoring" is extremely broad. And, I would say, almost unreachable goal. You have it mail you when it goes over 80% disk usage (and what if you are on holiday)? Does it mail all colleagues? Who picks it up (I thought Bob picked it up, but Bob thought Anne picked it up. So no one did)? Does it come and wake you in person when it reaches 92%? Will this catch this async job that fails (but should never) in an endless loop but keeps creating 20MB json files as fast as the disk allows it to? Is it an alerting that finds anomalies in trends? Will it be fast enough for you to come online before that job has filled the disk? I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.

geocrasher 6y ago
> I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen. I work in hosting too, and have been for a long time. I feel ya.
dilyevsky 6y ago
Slack warning/ticket at 75%, page at 85% (to oncall obviously). Don’t let user workload crap into your root partition. I’ve been doing this for over 10 years and managed many thousands of nodes and literally don’t recall full disk problem unless it was in staging somewhere where monitoring was deliberately disabled.
- berkes 6y ago
  Your requirements for "proper monitoring" are not everyones requirements. On a current gig, we host at heroku. Our monitoring is all about 95th percentile response-times, secondary services, backlogs, slow-queries and whatnots. For another job, "disk space filling up" is important. Again another job will need to monitor email-delivery-rates and so on and so forth.

jabroni_salad 6y ago

Keep in mind that sysadmins are essentially babysitting software that they do not develop. The hacks that we come up with are to work around responsible party and help us get a good night's sleep instead of a 2am wakeup call. I try to cut you guys some slack, usually this proliferates when management decides they are willing to accept some inefficiency in favor of getting new features out the door. I get it, really. My org is in the middle of a SRE introduction and for some reason I'm getting a lot of pushback on the topic of 'error budgets' and what to do with alerts when they are exceeded. Can't imagine why.

dilyevsky 6y ago
How does using proposed solution prevents a 2am wake up call? Your monitoring/alerting does, this just makes it easier to recover already broken software. And btw I’ve been carrying pagers for more than a decade so well aware of all the organizational dynamics here. Best way to prevent this is have devs carrying pager too (amazons “you built it you run it”) - and magically your nighttime oncall is much more pleasant ;-)