6 ms·

Which company deployed a chaos monkey deamon on their systems? Seemed to improve resiliency when I read about it.

theolivenbaum 2y ago
Netflix did that many years ago, interesting idea even if a bit disruptive in the beginning https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/
- e28eta 2y ago
  See also, the rest of the simian army: https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116 https://netflixtechblog.com/the-netflix-simian-army-16e57fba...
amelius 2y ago
"Your flight has been delayed due to Chaos Monkey."
- deleted 2y ago
  [deleted]
- bigiain 2y ago
  "My name is Susie and I'll be the purser on your flight today, and on behalf of the Captain Chaos Monkey and the First Officer Chaos Monkey... Oh. Shit..."
- nine_k 2y ago
  This means a major system problem. The point of the Chaos Monkey is that the system should function without interruptions or problems despite the activity of the Chaos Monkey. That is, it is to keep the system in such a shape that it could swallow and overcome some rate of failure, higher than the "naturally occurring" rate.
bitwize 2y ago
The chaos monkey is there to remind you to always mount a scratch monkey.
RcouF1uZ4gsC 2y ago
The same company that was in the news recently for screwing up a livestream of a boxing match.
- chrisweekly 2y ago
  True, but it's the exception that proves the rule; it's also the same company responsible for delivering a staggeringly high percentage of internet video, typically without a hitch.
  - eru 2y ago
    Yes, though serving static files is easier than streaming live.
  - tovej 2y ago
    That's not what an exception proving a rule means. It has a technical meaning: a sign that says "free parking on sundays" implies parking is not free as a rule. When used like this it just confuses a reader with rethoric. In this case netflix is just bad at live streaming, they clearly haven't done the necessary engineering work on it.
    - lelanthran 2y ago
      > That's not what an exception proving a rule means. It has a technical meaning: a sign that says "free parking on sundays" implies parking is not free as a rule. So the rule is "Free parking on Sundays", and the exception that proves it is "Free parking on Sundays"? That's a post-hoc (circular) argument that does not convince me at all. I read a different explanation of this phrase on HN recently: the "prove" in "exception proves the rule" has the same meaning as the "prove" (or "proof") in "50% proof alcohol". AIUI, in this context "proof" means "tests". The exception that tests the rule simply shows where the limits of the rules actually are. Well, that's how I understood it, anyway. Made sense to me at the time I read the explanation, but I'm open to being convinced otherwise with sufficiently persuasive logic :-)
      - 2y ago
philsnow 2y ago
At Google, the global Chubby cell had gone so long without any downtime that people were starting to assume that’s it was just always available, leading to some kind of outage or other when the global cell finally did have some organic downtime. Chubby-SRE added quarterly synthetic downtime of the global cell (iff the downtime SLA had not already been exceeded).
- kelnos 2y ago
  For those of us who haven't worked at Google, what's "Chubby" and what's a "cell"?
  - chili6426 2y ago
    I don't work at Google but I do live in a country where I have access to google.com. Chubby is a lock service that is used internally at Google and a cell is referring to an instance of Chubby. You can read more here: https://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf https://static.googleusercontent.com/media/research.google.c...
  - nine_k 2y ago
    Replace this with "API gateway cluster", or basically any simple enough, very widely used service.
  - philsnow 2y ago
    Ah, chubby is a distributed lock service. Think “zookeeper” and you won’t be far off. https://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf https://static.googleusercontent.com/media/research.google.c... [pdf] Some random blog post: https://medium.com/coinmonks/chubby-a-centralized-lock-service-for-distributed-applications-390571273052 https://medium.com/coinmonks/chubby-a-centralized-lock-servi... You can run multiple copies/instances of chubby at the same time (like you could run two separate zookeepers). You usually run an odd number of them, typically 5. A group of chubby processes all managing the same namespace is a “cell”. A while ago, nearly everything at Google had at least an indirect dependency on chubby being available (for service discovery etc), so part of the standard bringup for a datacenter was setting up a dc-specific chubby cell. You could have multiple SRE-managed chubby cells per datacenter/cluster if there was some reason for it. Anybody could run their own, but chubby-sre wasn’t responsible for anybody else’s, I think. Finally, there was a global cell. It was both distributed across multiple datacenters and also contained endpoint information for the per-dc chubby cells, so if a brand new process woke up somewhere and all it knew how to access was the global chubby cell, it could bootstrap from that to talking to chubby in any datacenter and thence to any other process anywhere, more or less. ^ there’s a lot in there that I’m fuzzy about, maybe processes wake up and only know how to access local chubby, but that cell has endpoint info for the global one? I don’t think any part of this process used dns; service discovery (including how to discover the service discovery service) was done through chubby.

Netflix did that many years ago, interesting idea even if a bit disruptive in the beginning https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/

e28eta 2y ago
See also, the rest of the simian army: https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116 https://netflixtechblog.com/the-netflix-simian-army-16e57fba...

amelius 2y ago

"Your flight has been delayed due to Chaos Monkey."

deleted 2y ago
[deleted]
bigiain 2y ago
"My name is Susie and I'll be the purser on your flight today, and on behalf of the Captain Chaos Monkey and the First Officer Chaos Monkey... Oh. Shit..."
nine_k 2y ago
This means a major system problem. The point of the Chaos Monkey is that the system should function without interruptions or problems despite the activity of the Chaos Monkey. That is, it is to keep the system in such a shape that it could swallow and overcome some rate of failure, higher than the "naturally occurring" rate.

bitwize 2y ago

The chaos monkey is there to remind you to always mount a scratch monkey.

RcouF1uZ4gsC 2y ago

The same company that was in the news recently for screwing up a livestream of a boxing match.

chrisweekly 2y ago
True, but it's the exception that proves the rule; it's also the same company responsible for delivering a staggeringly high percentage of internet video, typically without a hitch.
- eru 2y ago
  Yes, though serving static files is easier than streaming live.
- tovej 2y ago
  That's not what an exception proving a rule means. It has a technical meaning: a sign that says "free parking on sundays" implies parking is not free as a rule. When used like this it just confuses a reader with rethoric. In this case netflix is just bad at live streaming, they clearly haven't done the necessary engineering work on it.
  - lelanthran 2y ago
    > That's not what an exception proving a rule means. It has a technical meaning: a sign that says "free parking on sundays" implies parking is not free as a rule. So the rule is "Free parking on Sundays", and the exception that proves it is "Free parking on Sundays"? That's a post-hoc (circular) argument that does not convince me at all. I read a different explanation of this phrase on HN recently: the "prove" in "exception proves the rule" has the same meaning as the "prove" (or "proof") in "50% proof alcohol". AIUI, in this context "proof" means "tests". The exception that tests the rule simply shows where the limits of the rules actually are. Well, that's how I understood it, anyway. Made sense to me at the time I read the explanation, but I'm open to being convinced otherwise with sufficiently persuasive logic :-)
    - 2y ago

philsnow 2y ago

At Google, the global Chubby cell had gone so long without any downtime that people were starting to assume that’s it was just always available, leading to some kind of outage or other when the global cell finally did have some organic downtime. Chubby-SRE added quarterly synthetic downtime of the global cell (iff the downtime SLA had not already been exceeded).

kelnos 2y ago
For those of us who haven't worked at Google, what's "Chubby" and what's a "cell"?
- chili6426 2y ago
  I don't work at Google but I do live in a country where I have access to google.com. Chubby is a lock service that is used internally at Google and a cell is referring to an instance of Chubby. You can read more here: https://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf https://static.googleusercontent.com/media/research.google.c...
- nine_k 2y ago
  Replace this with "API gateway cluster", or basically any simple enough, very widely used service.
- philsnow 2y ago
  Ah, chubby is a distributed lock service. Think “zookeeper” and you won’t be far off. https://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf https://static.googleusercontent.com/media/research.google.c... [pdf] Some random blog post: https://medium.com/coinmonks/chubby-a-centralized-lock-service-for-distributed-applications-390571273052 https://medium.com/coinmonks/chubby-a-centralized-lock-servi... You can run multiple copies/instances of chubby at the same time (like you could run two separate zookeepers). You usually run an odd number of them, typically 5. A group of chubby processes all managing the same namespace is a “cell”. A while ago, nearly everything at Google had at least an indirect dependency on chubby being available (for service discovery etc), so part of the standard bringup for a datacenter was setting up a dc-specific chubby cell. You could have multiple SRE-managed chubby cells per datacenter/cluster if there was some reason for it. Anybody could run their own, but chubby-sre wasn’t responsible for anybody else’s, I think. Finally, there was a global cell. It was both distributed across multiple datacenters and also contained endpoint information for the per-dc chubby cells, so if a brand new process woke up somewhere and all it knew how to access was the global chubby cell, it could bootstrap from that to talking to chubby in any datacenter and thence to any other process anywhere, more or less. ^ there’s a lot in there that I’m fuzzy about, maybe processes wake up and only know how to access local chubby, but that cell has endpoint info for the global one? I don’t think any part of this process used dns; service discovery (including how to discover the service discovery service) was done through chubby.