5 ms·

One reply tweet notes "a complex system always operates in a failure state", but to find more discussion on this point it's worth noting that this is a restatem

One reply tweet notes "a complex system always operates in a failure state", but to find more discussion on this point it's worth noting that this is a restatement of: > "The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode." -- John Gall, General Systemantics (aka Systemantics, aka The Systems Bible), 1977 < https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics > Searching for "fundamental failure mode theorem" will provide interesting further insights on this.

TeeMassive 4y ago
The number of interactions of a system with n components increases according to O(n^2). The probability of an interaction not going wrong decreases exponentially with each interaction.
rdoherty 4y ago
Also related: How Complex Systems Fail https://how.complexsystems.fail/ https://how.complexsystems.fail/
msluyter 4y ago
Thank you for introducing me to this rabbit hole. Here's a NYTimes article from 1976 -- note, many issues with the OCR, but still interesting and entertainingly written: https://www.nytimes.com/1976/12/26/archives/why-nothing-works-the-way-its-supposed-to-systemanantics.html https://www.nytimes.com/1976/12/26/archives/why-nothing-work...
- selestify 4y ago
  Same author wrote an entire book, The Systems Bible, on the topic.
MichaelZuo 4y ago
'complex systems usually operate in a failure mode' has a somewhat different meaning from "complex system always operates in a failure state". They usually operate in a failure mode because it's easy to be deceived by the superficial appearances of complex systems, thus it seems like a big bother to drill down the layers, until a failure has occurred. As excellently demonstrated by this 'Factorio' player.
pengstrom 4y ago
The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.
- kbenson 4y ago
  Experienced sysadmins see the occasional chatter of problems in logs and emails as the pulse of the system, the EKG hooked up to the patient letting you know how things are going. The right level of logging is usually enough to be annoying but not so annoying you need to reduce it. That means you're hopefully thorough enough in the logging that you're reporting what you care about and the things you see are the the things you can manually fix or ignore, and fit a frequency level you can live with. Until the stuff you're managing becomes too large for that to make sense anymore, because that only scales so far (but surprisingly far for diligent admins). Then you need to move to something more complex, which is a big job and requires an entirely different way of reviewing (like Prometheus). You're probably better doing that from the beginning, but it's time and effort but everyone can take, or may not have existed at that time.
  - outworlder 4y ago
    > because that only scales so far (but surprisingly far for diligent admins) Surprisingly far. We log terabytes of data a day, most of it is unnecessary. I've been trying to tell people that it is not a good idea but so far I've had limited success. It's easy for people to just "printf" everything (we are lucky if logging libraries are used with proper levels). Right now, we have the most expensive 'add' operation in history. K8s pod logs some data. That is eventually noticed by fluentd, which is watching the filesystem. Fluentd sends to our logging servers over the network. That goes to kafka, then is shipped to the system that actually does the indexing. And then, there's a query running, which will then count the number of times that string appeared and updates a counter. All of which could have been avoided if the apps just added +1 to a counter, and exposed that as /metrics so Prometheus could scrape.
    - renlo 4y ago
AndrewKemendo 4y ago
This is so spot on it's universally true
AceJohnny2 4y ago
Thanks! I read this years ago and have been sharing the concept with colleagues, but couldn't remember where it came from. It's a really useful concept to keep in mind when designing or maintaining complex systems.
deleted 4y ago
[deleted]

TeeMassive 4y ago

The number of interactions of a system with n components increases according to O(n^2). The probability of an interaction not going wrong decreases exponentially with each interaction.

rdoherty 4y ago

Also related: How Complex Systems Fail https://how.complexsystems.fail/ https://how.complexsystems.fail/

msluyter 4y ago

Thank you for introducing me to this rabbit hole. Here's a NYTimes article from 1976 -- note, many issues with the OCR, but still interesting and entertainingly written: https://www.nytimes.com/1976/12/26/archives/why-nothing-works-the-way-its-supposed-to-systemanantics.html https://www.nytimes.com/1976/12/26/archives/why-nothing-work...

selestify 4y ago
Same author wrote an entire book, The Systems Bible, on the topic.

MichaelZuo 4y ago

'complex systems usually operate in a failure mode' has a somewhat different meaning from "complex system always operates in a failure state". They usually operate in a failure mode because it's easy to be deceived by the superficial appearances of complex systems, thus it seems like a big bother to drill down the layers, until a failure has occurred. As excellently demonstrated by this 'Factorio' player.

pengstrom 4y ago

The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.

kbenson 4y ago
Experienced sysadmins see the occasional chatter of problems in logs and emails as the pulse of the system, the EKG hooked up to the patient letting you know how things are going. The right level of logging is usually enough to be annoying but not so annoying you need to reduce it. That means you're hopefully thorough enough in the logging that you're reporting what you care about and the things you see are the the things you can manually fix or ignore, and fit a frequency level you can live with. Until the stuff you're managing becomes too large for that to make sense anymore, because that only scales so far (but surprisingly far for diligent admins). Then you need to move to something more complex, which is a big job and requires an entirely different way of reviewing (like Prometheus). You're probably better doing that from the beginning, but it's time and effort but everyone can take, or may not have existed at that time.
- outworlder 4y ago
  > because that only scales so far (but surprisingly far for diligent admins) Surprisingly far. We log terabytes of data a day, most of it is unnecessary. I've been trying to tell people that it is not a good idea but so far I've had limited success. It's easy for people to just "printf" everything (we are lucky if logging libraries are used with proper levels). Right now, we have the most expensive 'add' operation in history. K8s pod logs some data. That is eventually noticed by fluentd, which is watching the filesystem. Fluentd sends to our logging servers over the network. That goes to kafka, then is shipped to the system that actually does the indexing. And then, there's a query running, which will then count the number of times that string appeared and updates a counter. All of which could have been avoided if the apps just added +1 to a counter, and exposed that as /metrics so Prometheus could scrape.
  - renlo 4y ago

AndrewKemendo 4y ago

This is so spot on it's universally true

AceJohnny2 4y ago

Thanks! I read this years ago and have been sharing the concept with colleagues, but couldn't remember where it came from. It's a really useful concept to keep in mind when designing or maintaining complex systems.

deleted 4y ago

[deleted]