5 ms·

I wonder if their chaos system causes network partitions as well as node failures.

by jshen 12y ago

darkr 12y ago
Not AFAIK - last time I looked it requires one or more AWS autoscaling groups, and works by semi-randomly terminating instances that match configured criteria.
sargun 12y ago
The total toolkit they have is called the Simian Army -- https://github.com/Netflix/SimianArmy https://github.com/Netflix/SimianArmy They wrote a blog post about it here: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html I don't think they have one that introduces network partitions, but inside a datacenter, network partitions are rare.
- jon-wood 12y ago
  Inside a single data center they're rare, but when running on AWS in multiple availability zones and regions they're a fact of life which you should be designing your systems to deal with. Given that it would be great to have automated tooling to simulate them and keep everyone on their toes.
- derek 12y ago
  > ...but inside a datacenter, network partitions are rare It's ... complicated. http://aphyr.com/posts/288-the-network-is-reliable http://aphyr.com/posts/288-the-network-is-reliable
  - sargun 12y ago
    Yeah, there was an excellent ACM article with him, and Bailis. I think if the network starts to partition, or fail in a datacenter, that's time to evacuate the datacenter / AZ. If a handful of machines fail, they should disengage. If more than say, 5% of the machines in the DC are having reachability issues at any given point (in a modern DC that's like ~2000 machines), it's time to shut it down.
- SEJeff 12y ago
  Ever lost a switch? Rare, but happens on a regular enough case to prepare against it. Even if you use bonding/teaming, with dual power supply network kit + servers, it can still happen.
- jshen 12y ago
  I've seen them happen with the systems I work on a few times over the past five years. It isn't something you should ignore.

I wonder if their chaos system causes network partitions as well as node failures.

by jshen 12y ago

I wonder if their chaos system causes network partitions as well as node failures.

darkr 12y ago

Not AFAIK - last time I looked it requires one or more AWS autoscaling groups, and works by semi-randomly terminating instances that match configured criteria.

sargun 12y ago

The total toolkit they have is called the Simian Army -- https://github.com/Netflix/SimianArmy https://github.com/Netflix/SimianArmy They wrote a blog post about it here: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html I don't think they have one that introduces network partitions, but inside a datacenter, network partitions are rare.

jon-wood 12y ago
Inside a single data center they're rare, but when running on AWS in multiple availability zones and regions they're a fact of life which you should be designing your systems to deal with. Given that it would be great to have automated tooling to simulate them and keep everyone on their toes.
derek 12y ago
> ...but inside a datacenter, network partitions are rare It's ... complicated. http://aphyr.com/posts/288-the-network-is-reliable http://aphyr.com/posts/288-the-network-is-reliable
- sargun 12y ago
  Yeah, there was an excellent ACM article with him, and Bailis. I think if the network starts to partition, or fail in a datacenter, that's time to evacuate the datacenter / AZ. If a handful of machines fail, they should disengage. If more than say, 5% of the machines in the DC are having reachability issues at any given point (in a modern DC that's like ~2000 machines), it's time to shut it down.
SEJeff 12y ago
Ever lost a switch? Rare, but happens on a regular enough case to prepare against it. Even if you use bonding/teaming, with dual power supply network kit + servers, it can still happen.
jshen 12y ago
I've seen them happen with the systems I work on a few times over the past five years. It isn't something you should ignore.