6 ms·

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do wit

by yalooze 10y ago

I wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...

jsingleton 10y ago
I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.
- moondev 10y ago
  Everything goes through Spinnaker now, which in turn supports all clouddriver provides including aws, gcp, azure and kubernetes. Resource limits should be set by instance type. It's more of an application level thing than infrastructure which is what chaos monkey is supposed to simulate
- benhoyt 10y ago
  What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)
  - devonkim 10y ago
    "Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.
    - dberg 10y ago
      So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ? Edit: Sorry responded to wrong parent, sigh.
      - sametmax 10y ago
        It's just a matter of scale. If you are at the scale of netflix, vm are probably too complex black boxes, and logs output somewhere else anyway. Plus the problem may involve the interraction of several vm, or the network, or other composants together.
      - saryant 10y ago
        At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical. Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.
        vacri 10y ago
        Parsing logs isn't the only way to troubleshoot.
        flukus 10y ago
        It's not even the best way, just sometimes it's the only way.
  - RBerenguel 10y ago
    If deployment is automated and "clean", images get baked into machines and they just start. For instance, we use Ansible against the machine itself on boot, so we don't really need ssh access to it: everything is automated (but we keep it open to troubleshoot anything that may happen)
  - jaffa214525 10y ago
    Immutable infrastructure [1] is preferred. If SSH is detected, you must assume something on the server changed, and has deviated from the baseline. [1] https://www.oreilly.com/ideas/an-introduction-to-immutable-infrastructure https://www.oreilly.com/ideas/an-introduction-to-immutable-i...
- FireBeyond 10y ago
  I actually agree with you about immutable infrastructure, as I work at implementing it. But that's dangerously close to the "One True Way". Which is certainly not the case - so much of this is evolving, and a wide variety of situations and circumstances.
cortesoft 10y ago
Was wondering the same thing... I know in our environment, unexpected events on a box cause more problems than entire server failures. We design around servers coming in and out; specific processes failing in random ways is harder to design around.
saulrh 10y ago
If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.
- fyshotshot 10y ago
  Seems to me those kind of issues would be a good way to test your monitoring. For instance verifying that the appropriate people are notified or automatic action is taken when within a reasonable amount of time after the CPU on the box starts spiking.
- drdrey 10y ago
  We have another system for outlier detection that can kill instances that start behaving badly in terms of CPU, response time, error rates, etc. http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html http://techblog.netflix.com/2015/07/tracking-down-villains-o...
aaronblohowiak 10y ago
See my other reply in this thread for our reasoning. https://news.ycombinator.com/item?id=12744567 https://news.ycombinator.com/item?id=12744567
bbarn 10y ago
I think it could be because it doesn't provide the value to the development teams. Why chase a memory leak or a CPU bug if it's just being caused by your fault testing app? To prevent the negative effect of random machines dissapearing though.. that's a challenge that involves good ops, devs, even UI/UX I would imagine, and closer to something that users experience negatives because of in real life.
Nitramp 10y ago
To test a distributed system, there's not a lot of value to simulate all the many different conditions that can happen on a machine. From the system's perspective, you don't really care what happens on a single computer. The conditions you need to simulate are (a) the machine being abruptly gone or (b) the machine still accepting requests, but being very slow in returning them and maybe (c) machine returning incorrect results. Seen from the outside, everything that can happen is usually (a) or (b), and with ECC memory and reasonable software (?) hopefully never (c).

by yalooze 10y ago

jsingleton 10y ago

I would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.

moondev 10y ago
Everything goes through Spinnaker now, which in turn supports all clouddriver provides including aws, gcp, azure and kubernetes. Resource limits should be set by instance type. It's more of an application level thing than infrastructure which is what chaos monkey is supposed to simulate
benhoyt 10y ago
What do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)
- devonkim 10y ago
  "Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.
  - dberg 10y ago
    So logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ? Edit: Sorry responded to wrong parent, sigh.
    - sametmax 10y ago
      It's just a matter of scale. If you are at the scale of netflix, vm are probably too complex black boxes, and logs output somewhere else anyway. Plus the problem may involve the interraction of several vm, or the network, or other composants together.
    - saryant 10y ago
      At this scale, you basically have to have centralized logging. When you have thousands of parallel instances of a single application, searching logs box-by-box just isn't practical. Consider also that if you're elastically scaling EC2 instances and you need logs off an instance that's since been terminated, too late! That disk is gone. So again, you need a central log service.
      - vacri 10y ago
        Parsing logs isn't the only way to troubleshoot.
        flukus 10y ago
        It's not even the best way, just sometimes it's the only way.
- RBerenguel 10y ago
  If deployment is automated and "clean", images get baked into machines and they just start. For instance, we use Ansible against the machine itself on boot, so we don't really need ssh access to it: everything is automated (but we keep it open to troubleshoot anything that may happen)
- jaffa214525 10y ago
  Immutable infrastructure [1] is preferred. If SSH is detected, you must assume something on the server changed, and has deviated from the baseline. [1] https://www.oreilly.com/ideas/an-introduction-to-immutable-infrastructure https://www.oreilly.com/ideas/an-introduction-to-immutable-i...
FireBeyond 10y ago
I actually agree with you about immutable infrastructure, as I work at implementing it. But that's dangerously close to the "One True Way". Which is certainly not the case - so much of this is evolving, and a wide variety of situations and circumstances.

cortesoft 10y ago

Was wondering the same thing... I know in our environment, unexpected events on a box cause more problems than entire server failures. We design around servers coming in and out; specific processes failing in random ways is harder to design around.

saulrh 10y ago

If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.

fyshotshot 10y ago
Seems to me those kind of issues would be a good way to test your monitoring. For instance verifying that the appropriate people are notified or automatic action is taken when within a reasonable amount of time after the CPU on the box starts spiking.
drdrey 10y ago
We have another system for outlier detection that can kill instances that start behaving badly in terms of CPU, response time, error rates, etc. http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html http://techblog.netflix.com/2015/07/tracking-down-villains-o...

aaronblohowiak 10y ago

See my other reply in this thread for our reasoning. https://news.ycombinator.com/item?id=12744567 https://news.ycombinator.com/item?id=12744567

bbarn 10y ago

I think it could be because it doesn't provide the value to the development teams. Why chase a memory leak or a CPU bug if it's just being caused by your fault testing app? To prevent the negative effect of random machines dissapearing though.. that's a challenge that involves good ops, devs, even UI/UX I would imagine, and closer to something that users experience negatives because of in real life.

Nitramp 10y ago

To test a distributed system, there's not a lot of value to simulate all the many different conditions that can happen on a machine. From the system's perspective, you don't really care what happens on a single computer. The conditions you need to simulate are (a) the machine being abruptly gone or (b) the machine still accepting requests, but being very slow in returning them and maybe (c) machine returning incorrect results. Seen from the outside, everything that can happen is usually (a) or (b), and with ECC memory and reasonable software (?) hopefully never (c).