3 ms·

Yup. If the argument is "don't fall in love with your servers", I'm in agreement. However, the idea that whenever anything weird happens you should just kill y

Yup. If the argument is "don't fall in love with your servers", I'm in agreement. However, the idea that whenever anything weird happens you should just kill you server/cluster and move on without doing any sort of investigation seems like a recipe for disaster. That's a great way to mask bugs that may in-fact be systemic in nature, where they are or will eventually cause service degradation for your customers. I would hate to work in an environment where bugs are ignored and worked around instead of understood and fixed.

lamontcg 5y ago
I like the idea of killing it and moving on without paging someone at 2AM in the middle of the night. Ideally that goes into an async queue of issues though and someone finds the root cause and that goes into a queue of issues to fix, which is actually burned down. I suspect what is happening more often is that the whole stack has so many levels and the SREs responsible for it all don't have the visibility into the stack they need to debug it all, so they use their SLOs as a club to ignore issues as long as they're meeting their metrics until it becomes a firefighting drill. A pile of cargo culted best practices and SLOs replacing hands on debugging.
tene 5y ago
What you do with the server/cluster after you take it out of service is up to you. Having automation like this to take things out of service means that you can immediately restore production workloads to full functionality. I'm far more likely to ignore and work around a bug instead of doing a proper investigation when I've got pressure to get production back up because this server/cluster is a Special Snowflake that must be fixed in-place. Hardware fails, and bugs happen. There's no getting around it. Automation to handle this case is a good part of any strategy for identifying, understanding, and fixing bugs.

lamontcg 5y ago

I like the idea of killing it and moving on without paging someone at 2AM in the middle of the night. Ideally that goes into an async queue of issues though and someone finds the root cause and that goes into a queue of issues to fix, which is actually burned down. I suspect what is happening more often is that the whole stack has so many levels and the SREs responsible for it all don't have the visibility into the stack they need to debug it all, so they use their SLOs as a club to ignore issues as long as they're meeting their metrics until it becomes a firefighting drill. A pile of cargo culted best practices and SLOs replacing hands on debugging.

tene 5y ago

What you do with the server/cluster after you take it out of service is up to you. Having automation like this to take things out of service means that you can immediately restore production workloads to full functionality. I'm far more likely to ignore and work around a bug instead of doing a proper investigation when I've got pressure to get production back up because this server/cluster is a Special Snowflake that must be fixed in-place. Hardware fails, and bugs happen. There's no getting around it. Automation to handle this case is a good part of any strategy for identifying, understanding, and fixing bugs.