25 ms·

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still

by luhn 6y ago

This is a fun debugging story, but is a great example why servers should be cattle not pets. Having trouble with a VM? Blow it up and get a fresh one. Still having trouble? The provisioning steps are codified, you can walk through them and find the one that causes the issue.

daenz 6y ago
The end result was a kernel patch to LKML, so I for one am happy they solved this problem at the source.
atombender 6y ago
If a freshly provisioned VM doesn't have the same issue, then they're not using automated configuration management (Puppet, Chef, or similar) for these settings, and then they have a more serious problem, as nothing in their runtime environment is "codified" or predictable.
- throwaway2048 6y ago
  plenty of issues are not deterministic, even with 100% of everything managed by configuration management software.
  - atombender 6y ago
    But /etc/sysctl.conf is deterministic.
    - jeffbee 6y ago
      What is a reason why that file would correspond to the actual sysctls in effect?
      - atombender 6y ago
        If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system. sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it's possible to do.
        jeffbee 6y ago
        OK, so a reason why this file might reflect reality is that some automatic system wrote the file and subsequently successfully ran sysctl -p, but there are dozens of reasons why the file and reality would differ. The only source of truth is sysctl(8) or reading files in /proc/sys, and these are the values that need to propagate to observability systems and decision-making.
        atombender 6y ago
        The configuration management does whatever you tell it to. In this case, it's your responsibility to ensure that sysctl.conf is exhaustive, and there are ways to ensure that it is. If anyone applies changes on the side, they will be reverted on the next pass. Not saying it's easy, but it's not non-trivial. Making this exhaustive with /proc is another story. Unfortunately, this is the only way until kernel folks start agreeing that random mutability is a good thing. Right now, the kernel has way too many mutation points, and it's not (as far as I know) possible to ask it for a "diff" against the defaults.
        lanstin 6y ago
        I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to minimize the expected time; the puppet config was migrated from cfengine and was based on a lot of host-name based regular expressions and very dangerous to debug/refactor).
    - throwaway2048 6y ago
      Not all issues stem from sysctl.conf
      - atombender 6y ago
        No, but that was the context of my comment.
josefresco 6y ago
> servers should be cattle not pets I'm stealing this.
- cameronbrown 6y ago
  It's a well known phrase: https://devops.stackexchange.com/questions/653/what-is-the-definition-of-cattle-not-pets https://devops.stackexchange.com/questions/653/what-is-the-d...
  - josefresco 6y ago
    Hey thanks! I've never heard it but given that I'm old and it (the phrase) was coined in 2011-2012 I'm not surprised.
    - cameronbrown 6y ago
      Awesome! Everyone learns something new every day :)
      - karatestomp 6y ago
        We just pray we don't learn the "obvious thing everyone who's any good knows" in an interview.
- fossuser 6y ago
  If you're interested there are a couple of great books that dig more into this kind of thing. The Phoenix Project: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine... Its related Dev Ops Handbook: https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations-ebook/dp/B01M9ASFQ3/ https://www.amazon.com/DevOps-Handbook-World-Class-Reliabili...
dralley 6y ago
That's awfully convenient, and I can't deny having done this, but it's also a great way to never understand what went wrong.
- cmeacham98 6y ago
  Re-provisioning a failed server to solve the problem and taking a deep dive to find the root cause are not mutually exclusive. Essentially all VM software will allow you to snapshot/backup/clone the VM for later analysis, while also fixing your production environment _now_.
  - londons_explore 6y ago
    When you have a service which is acting weirdly, ideally I'd like to snapshot the VM, then do whatever I need to do to repair the service urgently. That might involve making a new VM from scratch, but it also might twiddling some settings, or other emergency changes. Afterwards, I want to be able to restore the VM state, probably in a firewalled off environment, so I can debug exactly what was wrong. Sometimes I'd like to do it to a set of VM's - for example, if there is some DNS wierdness, I might want to snapshot an application server and a DNS server. So far, no cloud provider seems to offer functionality to make that easy, which is a bit disappointing.
storyinmemo 6y ago
My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually guessed at as bad. Cattle is a threshold, but when the same problem keeps coming up it's time to call the vet. http://rachelbythebay.com/w/ http://rachelbythebay.com/w/ has many good examples of this, some submitted and voted up here.
- jeffbee 6y ago
  It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones at the tops of the racks, or the ends of the rows, are hotter (or colder); there's some difference between the same model of hard disk made in Hungary compared to the ones made in Mexico; at some date the BIOS vendor made an undocumented firmware revision that changes an obscure energy/performance register in your CPU; you have a machine with a dead CMOS battery that worked normally until it was rebooted. Cattle is a good philosophy but it takes a huge amount of work to approach perfection.
  - sitkack 6y ago
    There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is designed against the empirical contract, not the stated one.
    - jeffbee 6y ago
zimbatm 6y ago
Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.
- brightball 6y ago
  I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes. The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.
  - anonymousisme 6y ago
    Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.
    - indigodaddy 6y ago
      Was thinking the exact same thought as I finished up the article...
    - lalos 6y ago
      Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.
      - mcguire 6y ago
        "Professional responsibility", if nothing else.
      - calvinmorrison 6y ago
fred_is_fred 6y ago
We really need a new analogy - I assure you that ranchers care when they lose a cow.
mcguire 6y ago
Once upon a time I got pulled off a project in the middle; the project was given to a new, but experienced, developer to finish. I was talking to a friend who is one of our cloud guys the other day and he tells me "my" app is a bit of a problem child. It seems it crashes regularly but the problem is mitigated by the server restarting, so no one has any urge to fix it. (As a professional, I'm offended.) Servers as cattle can cover a variety of sins.
raziel2p 6y ago
I can't agree with this. In the case that was debugged, the configuration change was probably legitimately changed for a valid business reason and was codified. If you have traditional VM infrastructure your provisioning may consist of 100s of small configuration changes. Are you really going to step through every single one of them manually? Configuration management tools don't exactly have a `git bisect` equivalent, and even if you did, you'd have to re-image the VM every time because VMs are stateful. And even if you could somehow bisect every single configuration change in your configuration management, there's the added complexity of how many configuration changes are actually needed to test if the problem is still present. In this case it would probably be fairly easy because DNS is such a core "feature", but if this is something more application-level, you're really going to be lost.
totaldex 6y ago
While I agree with your premise, its important to stress that finding solutions by 'blowing up your server' and starting fresh are rarely sustainable. It's often a useful exercise to dig into the root cause - a lot of times the problem you're seeing is just the tip of the iceberg.
foobiekr 6y ago
the cattle-vs-pets thing is not really an excuse to not root cause a persistent issue while you have one.
xi- 6y ago
They had a perfectly valid configuration and it uncovered a bug in the Linux kernel. That wouldn't have happened had they ignored the issue and tried again. Of course, anything necessary to get your production box producing, but a well-engineered server is worth the debugging time.
skywhopper 6y ago
You don't have any information about where the customer was on the cattle/pet divide, nor if the general advice to "treat servers as cattle" even makes sense in their case. Regardless, the whole point of the exercise was to find the root cause and prevent it from happening in the future. Sometimes you gotta dig in and do the work.

by luhn 6y ago

daenz 6y ago

The end result was a kernel patch to LKML, so I for one am happy they solved this problem at the source.

atombender 6y ago

If a freshly provisioned VM doesn't have the same issue, then they're not using automated configuration management (Puppet, Chef, or similar) for these settings, and then they have a more serious problem, as nothing in their runtime environment is "codified" or predictable.

throwaway2048 6y ago
plenty of issues are not deterministic, even with 100% of everything managed by configuration management software.
- atombender 6y ago
  But /etc/sysctl.conf is deterministic.
  - jeffbee 6y ago
    What is a reason why that file would correspond to the actual sysctls in effect?
    - atombender 6y ago
      If you use an automated configuration management system such as Puppet, you don't ever run sysctl manually in a shell. Instead, everything is controlled by the configuration management system. sysctl is a bit problematic in terms of exhaustiveness. That is, how do you ensure that the kernel only has its original values plus whatever you put in sysctl.conf, and nobody actually ran sysctl manually at some point? But it's possible to do.
      - jeffbee 6y ago
        OK, so a reason why this file might reflect reality is that some automatic system wrote the file and subsequently successfully ran sysctl -p, but there are dozens of reasons why the file and reality would differ. The only source of truth is sysctl(8) or reading files in /proc/sys, and these are the values that need to propagate to observability systems and decision-making.
        atombender 6y ago
        The configuration management does whatever you tell it to. In this case, it's your responsibility to ensure that sysctl.conf is exhaustive, and there are ways to ensure that it is. If anyone applies changes on the side, they will be reverted on the next pass. Not saying it's easy, but it's not non-trivial. Making this exhaustive with /proc is another story. Unfortunately, this is the only way until kernel folks start agreeing that random mutability is a good thing. Right now, the kernel has way too many mutation points, and it's not (as far as I know) possible to ask it for a "diff" against the defaults.
      - lanstin 6y ago
        I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to minimize the expected time; the puppet config was migrated from cfengine and was based on a lot of host-name based regular expressions and very dangerous to debug/refactor).
  - throwaway2048 6y ago
    Not all issues stem from sysctl.conf
    - atombender 6y ago
      No, but that was the context of my comment.

josefresco 6y ago

> servers should be cattle not pets I'm stealing this.

cameronbrown 6y ago
It's a well known phrase: https://devops.stackexchange.com/questions/653/what-is-the-definition-of-cattle-not-pets https://devops.stackexchange.com/questions/653/what-is-the-d...
- josefresco 6y ago
  Hey thanks! I've never heard it but given that I'm old and it (the phrase) was coined in 2011-2012 I'm not surprised.
  - cameronbrown 6y ago
    Awesome! Everyone learns something new every day :)
    - karatestomp 6y ago
      We just pray we don't learn the "obvious thing everyone who's any good knows" in an interview.
fossuser 6y ago
If you're interested there are a couple of great books that dig more into this kind of thing. The Phoenix Project: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine... Its related Dev Ops Handbook: https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations-ebook/dp/B01M9ASFQ3/ https://www.amazon.com/DevOps-Handbook-World-Class-Reliabili...

dralley 6y ago

That's awfully convenient, and I can't deny having done this, but it's also a great way to never understand what went wrong.

cmeacham98 6y ago
Re-provisioning a failed server to solve the problem and taking a deep dive to find the root cause are not mutually exclusive. Essentially all VM software will allow you to snapshot/backup/clone the VM for later analysis, while also fixing your production environment _now_.
- londons_explore 6y ago
  When you have a service which is acting weirdly, ideally I'd like to snapshot the VM, then do whatever I need to do to repair the service urgently. That might involve making a new VM from scratch, but it also might twiddling some settings, or other emergency changes. Afterwards, I want to be able to restore the VM state, probably in a firewalled off environment, so I can debug exactly what was wrong. Sometimes I'd like to do it to a set of VM's - for example, if there is some DNS wierdness, I might want to snapshot an application server and a DNS server. So far, no cloud provider seems to offer functionality to make that easy, which is a bit disappointing.

storyinmemo 6y ago

My fleet of machines I was the owner of at Facebook was around 10,000. I still remember the odd JVM crash that prompted me to reimage a machine. I wouldn't have remembered it except there were a few that month and it was the 3rd time I reimaged the same machine that I thought, "That's odd... I think I know that machine name." Checked history, saw the 3 repair jobs I had submitted... RAM was reset, CPU was eventually guessed at as bad. Cattle is a threshold, but when the same problem keeps coming up it's time to call the vet. http://rachelbythebay.com/w/ http://rachelbythebay.com/w/ has many good examples of this, some submitted and voted up here.

jeffbee 6y ago
It is indeed folly to assume that cattle have no identity. There's an internally famous video inside Google in which a particular well-known machine was unracked, dragged out into a field, and ceremonially smashed to pieces by some hardware techs. Sometimes a machine just takes an arrow to the knee and it's never the same again. Then there are all the uncontrolled or unrecorded differences between machines: the ones at the tops of the racks, or the ends of the rows, are hotter (or colder); there's some difference between the same model of hard disk made in Hungary compared to the ones made in Mexico; at some date the BIOS vendor made an undocumented firmware revision that changes an obscure energy/performance register in your CPU; you have a machine with a dead CMOS battery that worked normally until it was rebooted. Cattle is a good philosophy but it takes a huge amount of work to approach perfection.
- sitkack 6y ago
  There needs to be some level of conformity between instances, they stop being a heard and more of a zoo if the skew is too large. The workloads running on the instances shouldn't be able to tell which instance type they are running on, or your workloads should be written such that it doesn't matter (but at some point it will). Things that grow and move together, wear together, so you will end up with a system that is designed against the empirical contract, not the stated one.
  - jeffbee 6y ago

zimbatm 6y ago

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

brightball 6y ago
I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes. The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.
- anonymousisme 6y ago
  Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.
  - indigodaddy 6y ago
    Was thinking the exact same thought as I finished up the article...
  - lalos 6y ago
    Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.
    - mcguire 6y ago
      "Professional responsibility", if nothing else.
    - calvinmorrison 6y ago

fred_is_fred 6y ago

We really need a new analogy - I assure you that ranchers care when they lose a cow.

mcguire 6y ago

Once upon a time I got pulled off a project in the middle; the project was given to a new, but experienced, developer to finish. I was talking to a friend who is one of our cloud guys the other day and he tells me "my" app is a bit of a problem child. It seems it crashes regularly but the problem is mitigated by the server restarting, so no one has any urge to fix it. (As a professional, I'm offended.) Servers as cattle can cover a variety of sins.

raziel2p 6y ago

I can't agree with this. In the case that was debugged, the configuration change was probably legitimately changed for a valid business reason and was codified. If you have traditional VM infrastructure your provisioning may consist of 100s of small configuration changes. Are you really going to step through every single one of them manually? Configuration management tools don't exactly have a `git bisect` equivalent, and even if you did, you'd have to re-image the VM every time because VMs are stateful. And even if you could somehow bisect every single configuration change in your configuration management, there's the added complexity of how many configuration changes are actually needed to test if the problem is still present. In this case it would probably be fairly easy because DNS is such a core "feature", but if this is something more application-level, you're really going to be lost.

totaldex 6y ago

While I agree with your premise, its important to stress that finding solutions by 'blowing up your server' and starting fresh are rarely sustainable. It's often a useful exercise to dig into the root cause - a lot of times the problem you're seeing is just the tip of the iceberg.

foobiekr 6y ago

the cattle-vs-pets thing is not really an excuse to not root cause a persistent issue while you have one.

xi- 6y ago

They had a perfectly valid configuration and it uncovered a bug in the Linux kernel. That wouldn't have happened had they ignored the issue and tried again. Of course, anything necessary to get your production box producing, but a well-engineered server is worth the debugging time.

skywhopper 6y ago

You don't have any information about where the customer was on the cattle/pet divide, nor if the general advice to "treat servers as cattle" even makes sense in their case. Regardless, the whole point of the exercise was to find the root cause and prevent it from happening in the future. Sometimes you gotta dig in and do the work.