4 ms·
Ignoring disk full condition ! really. Handling an out of space condition should be part of your test suite - it certainly was back when I looked after a Map r
by C1sc0cat 8y ago
Ignoring disk full condition ! really.
Handling an out of space condition should be part of your test suite - it certainly was back when I looked after a Map reduce based Billing system at BT and that was back in the day when a cluster of 17 systems was a really big thing.
- mrmuagi 8y agoIMO that's too late of a condition check. You should keep a margin of free space for OS/Background applications just so you have enough life left in the application to detect and log it.
- time0ut 8y agoIn this case, the application itself does not use the disk in question, so the old version kept chugging along obliviously. We actually did have monitoring on the capacity of the disk in question. We discovered during the analysis that it had been alerting the responsible team all day. They had just been ignoring it.
- lifeisstillgood 8y agoI think the parent did well openly and honestly raising a personal example where missing a "basic" check caused near career changing problems - I applaud them for sharing a difficult situation. I was concerned that it's possible to read your comment as if it was critical of the parent - was that your intention?
- time0ut 8y agoTo clarify a little, I was neither responsible for monitoring the underlying hardware or the deployment systems in this case. I also didn't have access to fix it myself. It took me a couple hours to go from "random weird test results" to "full disk broke the deploy".
- C1sc0cat 8y agoIt was more a criticism of the OP's Sysad's - Though for the system I mentioned one of the reasons they employed me as a developer was that I was also a sysad on PR1MES - an early example of devops maybe.