5 ms·

I suspect that the issue isn't the engineers. It's the changes. A system running just fine will usually continue to run just fine. Until it's changed.

by adekok 10y ago

fail2fail2ban 10y ago
Or until the power is cycled. That's why it's best to do the crazy monkey or whatever it's called, walk into the production offline datacenter, online for testing (you do have geographically redundant datacenters right?) and pull out a half dozen random NIC cables. You can simulate with software tools but nothing beats the real test.
- softawre 10y ago
  > crazy monkey Literally lol'd. It's "chaos monkey".
  - jsprogrammer 10y ago
    Chaos monkey is a simulation, I believe.
    - bch 10y ago
      It's not a simulation, it's real-world production technology. https://en.wikipedia.org/wiki/Technical_details_of_Netflix https://en.wikipedia.org/wiki/Technical_details_of_Netflix
      - rrdharan 10y ago
        And it's OSS: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
      - s_kilk 10y ago
        I think they meant that Netflix Chaos Monkey is a "simulation" of the real-deal, which is sending a shaved-ape into a physical server room and pulling out physical network cables. I didn't read it as derogatory.
        bch 10y ago
        Oh perhaps they did mean shaved-ape simulator. I read none of it as derogatory, and hopefully my response didn't suggest otherwise.
Tenhundfeld 10y ago
I hear what you're saying, in theory. In practice, a system is always changing – regardless of whether developers are deploying code changes. Some events (or "changes") I've seen in last year that caused a system to stop running fine: - analyst ran query that locked reporting DB, which had ripple effects - integration partner "tweaked" schema of data uploaded to us - NIC started dropping packets - growing data made backup software exceed expected window, locking files - critical security patch released (didn't cause a problem exactly, but did change the system from having no known security holes to having a big one) On and on. So, yeah, I'm not disputing the idea that changes are often the source of problems. I'm just saying that any moderately complex system is constantly encountering new events whether or not the developers are making changes. This isn't a rebuttal of your statement so much, as it is rebuttal of a common attitude I see in business folks. A lot of non-technical executives seem to have the mentality that software is "done" after it's built, which is naive IMO. Active systems require active maintenance. You can avoid a lot of problems with intelligent architecture and robust instrumentation/monitoring, but at the end of the day systems rot and will eventually stop running fine. And even if your perfectly planned and architected system runs in total isolation on a private server where security isn't a major concern, you'll still build up technical debt if you aren't routinely upgrading major libraries, etc. You'll get to a point where you want to implement some feature, and while there are 3 excellent OSS libraries for your platform to accomplish it, none of them is compatible with your 5 year old version. You probably know all of this and might even agree with most of it. I just had a visceral reaction to the idea that "a system running just fine will usually continue to run just fine," and felt compelled to respond. :) Note: most of my experience is with public or enterprise web applications. I imagine other types of systems have other problems.

I suspect that the issue isn't the engineers. It's the changes. A system running just fine will usually continue to run just fine. Until it's changed.

by adekok 10y ago

I suspect that the issue isn't the engineers. It's the changes. A system running just fine will usually continue to run just fine. Until it's changed.

fail2fail2ban 10y ago

Or until the power is cycled. That's why it's best to do the crazy monkey or whatever it's called, walk into the production offline datacenter, online for testing (you do have geographically redundant datacenters right?) and pull out a half dozen random NIC cables. You can simulate with software tools but nothing beats the real test.

softawre 10y ago
> crazy monkey Literally lol'd. It's "chaos monkey".
- jsprogrammer 10y ago
  Chaos monkey is a simulation, I believe.
  - bch 10y ago
    It's not a simulation, it's real-world production technology. https://en.wikipedia.org/wiki/Technical_details_of_Netflix https://en.wikipedia.org/wiki/Technical_details_of_Netflix
    - rrdharan 10y ago
      And it's OSS: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
    - s_kilk 10y ago
      I think they meant that Netflix Chaos Monkey is a "simulation" of the real-deal, which is sending a shaved-ape into a physical server room and pulling out physical network cables. I didn't read it as derogatory.
      - bch 10y ago
        Oh perhaps they did mean shaved-ape simulator. I read none of it as derogatory, and hopefully my response didn't suggest otherwise.

Tenhundfeld 10y ago

I hear what you're saying, in theory. In practice, a system is always changing – regardless of whether developers are deploying code changes. Some events (or "changes") I've seen in last year that caused a system to stop running fine: - analyst ran query that locked reporting DB, which had ripple effects - integration partner "tweaked" schema of data uploaded to us - NIC started dropping packets - growing data made backup software exceed expected window, locking files - critical security patch released (didn't cause a problem exactly, but did change the system from having no known security holes to having a big one) On and on. So, yeah, I'm not disputing the idea that changes are often the source of problems. I'm just saying that any moderately complex system is constantly encountering new events whether or not the developers are making changes. This isn't a rebuttal of your statement so much, as it is rebuttal of a common attitude I see in business folks. A lot of non-technical executives seem to have the mentality that software is "done" after it's built, which is naive IMO. Active systems require active maintenance. You can avoid a lot of problems with intelligent architecture and robust instrumentation/monitoring, but at the end of the day systems rot and will eventually stop running fine. And even if your perfectly planned and architected system runs in total isolation on a private server where security isn't a major concern, you'll still build up technical debt if you aren't routinely upgrading major libraries, etc. You'll get to a point where you want to implement some feature, and while there are 3 excellent OSS libraries for your platform to accomplish it, none of them is compatible with your 5 year old version. You probably know all of this and might even agree with most of it. I just had a visceral reaction to the idea that "a system running just fine will usually continue to run just fine," and felt compelled to respond. :) Note: most of my experience is with public or enterprise web applications. I imagine other types of systems have other problems.