4 ms·

> It means that we’d rather just replace a “sick” instance by a new healthy one than taking it to a doctor. This analogy really bothers me. Cattle are expensiv

> It means that we’d rather just replace a “sick” instance by a new healthy one than taking it to a doctor. This analogy really bothers me. Cattle are expensive. They are an investment. You don't put down an investment just because it got sick. If you have a sick cow you will in-fact call your local large animal vet to come and treat it.

ska 5y ago
Yes that statement is wrong. I guess the real difference in the pet/cow calculus is for cattle you probably won't pay more for treatment than the cow is worth; with pets people do this all the time.
- psanford 5y ago
  Yup. If the argument is "don't fall in love with your servers", I'm in agreement. However, the idea that whenever anything weird happens you should just kill you server/cluster and move on without doing any sort of investigation seems like a recipe for disaster. That's a great way to mask bugs that may in-fact be systemic in nature, where they are or will eventually cause service degradation for your customers. I would hate to work in an environment where bugs are ignored and worked around instead of understood and fixed.
  - lamontcg 5y ago
    I like the idea of killing it and moving on without paging someone at 2AM in the middle of the night. Ideally that goes into an async queue of issues though and someone finds the root cause and that goes into a queue of issues to fix, which is actually burned down. I suspect what is happening more often is that the whole stack has so many levels and the SREs responsible for it all don't have the visibility into the stack they need to debug it all, so they use their SLOs as a club to ignore issues as long as they're meeting their metrics until it becomes a firefighting drill. A pile of cargo culted best practices and SLOs replacing hands on debugging.
  - tene 5y ago
    What you do with the server/cluster after you take it out of service is up to you. Having automation like this to take things out of service means that you can immediately restore production workloads to full functionality. I'm far more likely to ignore and work around a bug instead of doing a proper investigation when I've got pressure to get production back up because this server/cluster is a Special Snowflake that must be fixed in-place. Hardware fails, and bugs happen. There's no getting around it. Automation to handle this case is a good part of any strategy for identifying, understanding, and fixing bugs.
pyuser583 5y ago
Don’t let facts get in the way of a good metaphor. There’s nothing remotely disgusting about making sausages, but people say “Laws are like sausages - it’s better not to see them being made.” I stopped correcting them long ago.
- astrange 5y ago
  The process for hot dogs does look strangely artificial. https://www.youtube.com/watch?v=2NzUm7UEEIY https://www.youtube.com/watch?v=2NzUm7UEEIY
LAC-Tech 5y ago
Are you in Agtech too? :)

ska 5y ago

Yes that statement is wrong. I guess the real difference in the pet/cow calculus is for cattle you probably won't pay more for treatment than the cow is worth; with pets people do this all the time.

psanford 5y ago
Yup. If the argument is "don't fall in love with your servers", I'm in agreement. However, the idea that whenever anything weird happens you should just kill you server/cluster and move on without doing any sort of investigation seems like a recipe for disaster. That's a great way to mask bugs that may in-fact be systemic in nature, where they are or will eventually cause service degradation for your customers. I would hate to work in an environment where bugs are ignored and worked around instead of understood and fixed.
- lamontcg 5y ago
  I like the idea of killing it and moving on without paging someone at 2AM in the middle of the night. Ideally that goes into an async queue of issues though and someone finds the root cause and that goes into a queue of issues to fix, which is actually burned down. I suspect what is happening more often is that the whole stack has so many levels and the SREs responsible for it all don't have the visibility into the stack they need to debug it all, so they use their SLOs as a club to ignore issues as long as they're meeting their metrics until it becomes a firefighting drill. A pile of cargo culted best practices and SLOs replacing hands on debugging.
- tene 5y ago
  What you do with the server/cluster after you take it out of service is up to you. Having automation like this to take things out of service means that you can immediately restore production workloads to full functionality. I'm far more likely to ignore and work around a bug instead of doing a proper investigation when I've got pressure to get production back up because this server/cluster is a Special Snowflake that must be fixed in-place. Hardware fails, and bugs happen. There's no getting around it. Automation to handle this case is a good part of any strategy for identifying, understanding, and fixing bugs.

pyuser583 5y ago

Don’t let facts get in the way of a good metaphor. There’s nothing remotely disgusting about making sausages, but people say “Laws are like sausages - it’s better not to see them being made.” I stopped correcting them long ago.

astrange 5y ago
The process for hot dogs does look strangely artificial. https://www.youtube.com/watch?v=2NzUm7UEEIY https://www.youtube.com/watch?v=2NzUm7UEEIY

LAC-Tech 5y ago

Are you in Agtech too? :)