3 ms·
... and then six months later your server runs out of disk space and collapses because you hadn't configured proper graphs, and you spend several hours frantica
by simonw 11y ago
... and then six months later your server runs out of disk space and collapses because you hadn't configured proper graphs, and you spend several hours frantically trying to fix it. And then two months later your RAID controller fails and you realise that your backups stopped working when disk space ran out you never fixed them again. And then when you rebuild the server you realise you documented/automated 95% of the setup of the original machine, but that last 5% has you totally screwed because a package you were depending on has been deprecated by your Linux distribution and you need to figure out how to build it from scratch.
Running a server in a production environment across multiple years is hard. One of the hardest part is when everyone forgets about it because it's fully automated... and then it breaks.
- Bromlife 11y agoRaid... controller ? Your servers are virtualised with redundant hardware (if they're not IaaS in the first place). There's no "rebuilding" of the server. Have you ever actually managed a production environment in the last 10 years? We're not even talking about a lot of money here. A raid controller dying should either a) not bring down production or b) not be your problem anyway.