8 ms·

Everything is broken, and it’s okay

62 points by kiyanwang 6y ago

lmm 6y ago
This is true of current systems, which tend to simply automate human behaviour with human failings. It doesn't have to be that way though. It's possible to produce 100% correct components, and a system made of that kind of component can be dramatically simpler to understand.
- daliusd 6y ago
  Could you give example of component that is 100% correct? I feel that even components can be 100% correct only within context, e.g. within ideal automaton under expected limitations and we don't have that. While idea to construct everything from components overall seem to work better than to build monoliths quite often (Space X vs NASA, Linux vs many other OSes) but not always (e.g. SQLite).
  - bezout 6y ago
    Well, your argument would only apply to software which operating contexts are not known to the developers ahead of its release. Is it the case for every piece of software? I’d argue not. Most of the times, you know what to expect . And you can always put up some guards to ensure it fails gracefully.
    - daliusd 6y ago
      So expected limitations in your cases is "known to the developers ahead of its release", but what happens with the software after release, e.g. after 10 iterations or 2 years or 10 years? What happens if your product from 10s of users grows to 1000s of users, millions of users? What happens when new functionality must be added and it collides with the existing one in multiple points? You can and you must do your best, but you can't do everything - you must choose what is your sacrifice.
  - lmm 6y ago
    > Could you give example of component that is 100% correct? I feel that even components can be 100% correct only within context, e.g. within ideal automaton under expected limitations and we don't have that. Verified* instances in safe Idris. Of course GIGO is still valid and failures in lower-level components (such as a processor just not behaving as specified) can cause failures in higher-level components, but it's simply impossible for there to be a failure in that component itself - any observed failure will necessarily be the result of a lower-level failure.
    - azornathogron 6y ago
      I think you are ignoring both failures to correctly specify what was intended (the design was right conceptually, but you specified it incorrectly when formalising it so now you have a perfectly verified incorrect implementation), and failures in the design (the thing works as you designed it, but it turns out your design isn't perfect). More likely, you're not unaware of these possibilities, you're just using a meaning of "failure" that excludes them. But regardless of whether you call these failures or call them something else, they prevent you from guaranteeing perfection.
      - lmm 6y ago
        On the contrary, there is guaranteed perfection: I can guarantee that they perfectly implement the given formal specification. I think we're in agreement that one can't produce a guaranteed-perfect system (since the system will always have a part that must interact with humans), but I don't think that actually supports the approach of treating every component as fallible.
        marcus_holmes 6y ago
        And don't ignore the fact that needs change over time - a perfectly designed and functioning component will become imperfectly specified over time as the need for it changes. And there is the cost/benefit calculation - the effort required to get something perfect is often greater than the benefit of having a perfect component. The purpose of commercial code is to make money, and if technically perfect code makes less money than technically imperfect code, then is it fulfilling its purpose perfectly?
- di4na 6y ago
  [Citation needed] If anything, every research on complexity and correctness shows that it does not compose and does not survive in a dynamic system. I would love to see the research that support this claim. It would be a revolution in these fields.
  - v-erne 6y ago
    Could You point me towards research papers about complexity and corectness ? I have this unconfirmed intuition about the way we manage complexity and correctness (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality) I would like to broaden it a bit and find out if there possibly is some truth to it.
    - fsflover 6y ago
      https://blog.invisiblethings.org/2008/09/02/three-approaches-to-computer-security.html https://blog.invisiblethings.org/2008/09/02/three-approaches...
      - stan_rogers 6y ago
        Blogs are not research papers. No, not even in computing.
        fsflover 6y ago
        The author is a well known security researcher. She probably has corresponding publications, too.
    - di4na 6y ago
      The good news, yes. The bad news: it depends a lot of what you mean, it may not be the answer you search and i do not have a "perfect" reference, because it is considered mostly obvious in these fields... But before i get there, a few things. > (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality) Nearly all the references i point to will tell you that this leakiness and this constant changing reality is exactly what makes them work in the real world. Now onto references. A lot of them can be found at http://resiliencepapers.club/ http://resiliencepapers.club/ I particularly advise to start at https://www.youtube.com/watch?v=Pb_zYs8G6Co https://www.youtube.com/watch?v=Pb_zYs8G6Co https://www.youtube.com/watch?v=PGLYEDpNu60 https://www.youtube.com/watch?v=PGLYEDpNu60 For more in depth, i particularly like https://www.ida.liu.se/~729A15/mtrl/CSEnew.pdf https://www.ida.liu.se/~729A15/mtrl/CSEnew.pdf And this is basically a whole meta study of all the research and ideas that came to this field since WW2. Beware it is long https://www.sintef.no/globalassets/upload/teknologi_og_samfunn/sikkerhet-og-palitelighet/rapporter/sintef-a17034-organisational-accidents-and-resilience-organisations-six-perspectives.-revision-2.pdf https://www.sintef.no/globalassets/upload/teknologi_og_samfu... For an example of how things go wrong when things are correct but interact well together, i advise these two papers on space systems. (I link to Adrian blog on them because they may be easier to read, but there is a link to the papers in there) https://blog.acolyer.org/2017/11/30/the-role-of-software-in-spacecraft-accidents/ https://blog.acolyer.org/2017/11/30/the-role-of-software-in-... https://blog.acolyer.org/2017/12/01/analyzing-software-requirements-errors-in-safety-critical-embedded-systems/ https://blog.acolyer.org/2017/12/01/analyzing-software-requi... If these things make you curious for more, there is a lot. Or if you want to talk about/have help navigating what is in the http://resiliencepapers.club/ http://resiliencepapers.club/ link, feel free to reach to me on twitter or by email. I will be happy to talk more.
bezout 6y ago
Just because a piece of software won’t kill people if it fails, it doesn’t mean that we should drop the ball on trying to make it as reliable as possible. And I’m not talking about OSS. I’m talking about the software you pay for. You are giving away your money and you expect something as valuable in return.
- CaptArmchair 6y ago
  From the article: > Once you accept failure is inevitable and commit to avoiding the concatenation of failures that causes catastrophes, you—and your organization—need to decide what risk looks like and what to prioritize. If everything is the most important thing, then nothing is. [...] > The more complex the system, the more likely it is that some part of it is broken. As engineers, we’re at the apex of literally millions of hours of design and engineering time to help us, and our users, with everything from finding our phone to continuously integrating our code. We have to assume that some of that infrastructure, and some of our work, is broken. > If we accept these imperfections, we can work toward building resilient systems that can handle a little static and deal with flawed foundations without falling over. We don’t stop working toward perfection just because it’s impossible. The author's point isn't to throw up arms in defeat. Her point is to "courageously change the things you can change, accept the things you can't change, and find the wisdom to know the difference between the two". There will always be tension between striving for the ideal world where everything is always written to perfection using the latest innovative technology, and the real world where legacy code and technical debt are par for the course. It's all too easy to compare software on which lives depend (avionics, healthcare,...) with video games, short-lived marketing apps or business suites. "value for your money" means very different things to different people in different contexts.
  - MattPalmer1086 6y ago
    I agree with your main point that the article is not suggesting giving up. It does amuse me though that an ideal of perfection is having no legacy code, all written with the latest technologies. I once spent some time refactoring some really old gnarly code to be beautiful and well designed. The rest of the team looked on in horror and asked why I was replacing highly reliable, battle tested code that had been in successful operation for nearly ten years!
JackPoach 6y ago
It's OK, if you are going fix it. Otherwise try living in a house where things are broken.
- durnygbur 6y ago
  So a rented place? Where the owner has no incentive to fix it, and the tenant oftentimes doesn't have the authorization to do so. Especially when the owning entity is focused exclusively on extracting the profits. You accept it, move out, or turn insane.
- bloak 6y ago
  Every house I've lived in has had several things - doors, window blinds, appliances - that only I can operate because it takes practice to work around their brokenness. It's only when I see someone else trying and failing to use them that I remember that they're broken. But it doesn't convince me to repair them. I get used to things and don't like unnecessary change.
- afarrell 6y ago
  Its much harder to fix things if people are unwilling to hear the idea that things are broken.
MattPalmer1086 6y ago
This article makes some very good points. I've been thinking about resilience for a while. Nothing is perfect, and if your system can only work with that assumption, you are bound for failure. I always loved the Erlang philosophy of let it crash. The idea you build a highly resilient system out of error prone parts rocked my world when I first encountered it. None of this is to say we should accept broken things or not try to make them better. We should instead accept some parts will always fail and design accordingly.
ArcMex 6y ago
I take this philosophy conversely when developing software. I expect my first version to be imperfect. I then chip out the imperfections. Before, I wanted to build the perfect system right away. Now failure is planned and part of the process. Some people call it TDD.
SunlightEdge 6y ago
This could have been written by ProjectRed a la cyberpunk. The company knew the game was full of bugs still but were under immense pressure to ship out the game way before it was ready.
- rob74 6y ago
  Well, most AAA title are released full of bugs - those are the "failures" the article mentions. But most of them manage to avoid the "catastrophe" of being deemed completely unplayable...
doggodaddo78 6y ago
Shrugging apathy is the enemy of the excellence (not perfection). Btw, with very simple programs, it is possible to have zero bugs but it is decreasingly possible as programs increase in complexity. As an example, Android is Humpty Dumpty. Manufacturers just try to glue as many pieces together as possible.
moocowtruck 6y ago
this is where as a user of software that is often times broken, I wish I had better protections on when/how to get my money back. Software is not obviously broken, and I feel like buyers have very little recourse
mshaler 6y ago
...that's how the light gets in.

62 points by kiyanwang 6y ago

lmm 6y ago

This is true of current systems, which tend to simply automate human behaviour with human failings. It doesn't have to be that way though. It's possible to produce 100% correct components, and a system made of that kind of component can be dramatically simpler to understand.

daliusd 6y ago
Could you give example of component that is 100% correct? I feel that even components can be 100% correct only within context, e.g. within ideal automaton under expected limitations and we don't have that. While idea to construct everything from components overall seem to work better than to build monoliths quite often (Space X vs NASA, Linux vs many other OSes) but not always (e.g. SQLite).
- bezout 6y ago
  Well, your argument would only apply to software which operating contexts are not known to the developers ahead of its release. Is it the case for every piece of software? I’d argue not. Most of the times, you know what to expect . And you can always put up some guards to ensure it fails gracefully.
  - daliusd 6y ago
    So expected limitations in your cases is "known to the developers ahead of its release", but what happens with the software after release, e.g. after 10 iterations or 2 years or 10 years? What happens if your product from 10s of users grows to 1000s of users, millions of users? What happens when new functionality must be added and it collides with the existing one in multiple points? You can and you must do your best, but you can't do everything - you must choose what is your sacrifice.
- lmm 6y ago
  > Could you give example of component that is 100% correct? I feel that even components can be 100% correct only within context, e.g. within ideal automaton under expected limitations and we don't have that. Verified* instances in safe Idris. Of course GIGO is still valid and failures in lower-level components (such as a processor just not behaving as specified) can cause failures in higher-level components, but it's simply impossible for there to be a failure in that component itself - any observed failure will necessarily be the result of a lower-level failure.
  - azornathogron 6y ago
    I think you are ignoring both failures to correctly specify what was intended (the design was right conceptually, but you specified it incorrectly when formalising it so now you have a perfectly verified incorrect implementation), and failures in the design (the thing works as you designed it, but it turns out your design isn't perfect). More likely, you're not unaware of these possibilities, you're just using a meaning of "failure" that excludes them. But regardless of whether you call these failures or call them something else, they prevent you from guaranteeing perfection.
    - lmm 6y ago
      On the contrary, there is guaranteed perfection: I can guarantee that they perfectly implement the given formal specification. I think we're in agreement that one can't produce a guaranteed-perfect system (since the system will always have a part that must interact with humans), but I don't think that actually supports the approach of treating every component as fallible.
      - marcus_holmes 6y ago
        And don't ignore the fact that needs change over time - a perfectly designed and functioning component will become imperfectly specified over time as the need for it changes. And there is the cost/benefit calculation - the effort required to get something perfect is often greater than the benefit of having a perfect component. The purpose of commercial code is to make money, and if technically perfect code makes less money than technically imperfect code, then is it fulfilling its purpose perfectly?
di4na 6y ago
[Citation needed] If anything, every research on complexity and correctness shows that it does not compose and does not survive in a dynamic system. I would love to see the research that support this claim. It would be a revolution in these fields.
- v-erne 6y ago
  Could You point me towards research papers about complexity and corectness ? I have this unconfirmed intuition about the way we manage complexity and correctness (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality) I would like to broaden it a bit and find out if there possibly is some truth to it.
  - fsflover 6y ago
    https://blog.invisiblethings.org/2008/09/02/three-approaches-to-computer-security.html https://blog.invisiblethings.org/2008/09/02/three-approaches...
    - stan_rogers 6y ago
      Blogs are not research papers. No, not even in computing.
      - fsflover 6y ago
        The author is a well known security researcher. She probably has corresponding publications, too.
  - di4na 6y ago
    The good news, yes. The bad news: it depends a lot of what you mean, it may not be the answer you search and i do not have a "perfect" reference, because it is considered mostly obvious in these fields... But before i get there, a few things. > (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality) Nearly all the references i point to will tell you that this leakiness and this constant changing reality is exactly what makes them work in the real world. Now onto references. A lot of them can be found at http://resiliencepapers.club/ http://resiliencepapers.club/ I particularly advise to start at https://www.youtube.com/watch?v=Pb_zYs8G6Co https://www.youtube.com/watch?v=Pb_zYs8G6Co https://www.youtube.com/watch?v=PGLYEDpNu60 https://www.youtube.com/watch?v=PGLYEDpNu60 For more in depth, i particularly like https://www.ida.liu.se/~729A15/mtrl/CSEnew.pdf https://www.ida.liu.se/~729A15/mtrl/CSEnew.pdf And this is basically a whole meta study of all the research and ideas that came to this field since WW2. Beware it is long https://www.sintef.no/globalassets/upload/teknologi_og_samfunn/sikkerhet-og-palitelighet/rapporter/sintef-a17034-organisational-accidents-and-resilience-organisations-six-perspectives.-revision-2.pdf https://www.sintef.no/globalassets/upload/teknologi_og_samfu... For an example of how things go wrong when things are correct but interact well together, i advise these two papers on space systems. (I link to Adrian blog on them because they may be easier to read, but there is a link to the papers in there) https://blog.acolyer.org/2017/11/30/the-role-of-software-in-spacecraft-accidents/ https://blog.acolyer.org/2017/11/30/the-role-of-software-in-... https://blog.acolyer.org/2017/12/01/analyzing-software-requirements-errors-in-safety-critical-embedded-systems/ https://blog.acolyer.org/2017/12/01/analyzing-software-requi... If these things make you curious for more, there is a lot. Or if you want to talk about/have help navigating what is in the http://resiliencepapers.club/ http://resiliencepapers.club/ link, feel free to reach to me on twitter or by email. I will be happy to talk more.

bezout 6y ago

Just because a piece of software won’t kill people if it fails, it doesn’t mean that we should drop the ball on trying to make it as reliable as possible. And I’m not talking about OSS. I’m talking about the software you pay for. You are giving away your money and you expect something as valuable in return.

CaptArmchair 6y ago
From the article: > Once you accept failure is inevitable and commit to avoiding the concatenation of failures that causes catastrophes, you—and your organization—need to decide what risk looks like and what to prioritize. If everything is the most important thing, then nothing is. [...] > The more complex the system, the more likely it is that some part of it is broken. As engineers, we’re at the apex of literally millions of hours of design and engineering time to help us, and our users, with everything from finding our phone to continuously integrating our code. We have to assume that some of that infrastructure, and some of our work, is broken. > If we accept these imperfections, we can work toward building resilient systems that can handle a little static and deal with flawed foundations without falling over. We don’t stop working toward perfection just because it’s impossible. The author's point isn't to throw up arms in defeat. Her point is to "courageously change the things you can change, accept the things you can't change, and find the wisdom to know the difference between the two". There will always be tension between striving for the ideal world where everything is always written to perfection using the latest innovative technology, and the real world where legacy code and technical debt are par for the course. It's all too easy to compare software on which lives depend (avionics, healthcare,...) with video games, short-lived marketing apps or business suites. "value for your money" means very different things to different people in different contexts.
- MattPalmer1086 6y ago
  I agree with your main point that the article is not suggesting giving up. It does amuse me though that an ideal of perfection is having no legacy code, all written with the latest technologies. I once spent some time refactoring some really old gnarly code to be beautiful and well designed. The rest of the team looked on in horror and asked why I was replacing highly reliable, battle tested code that had been in successful operation for nearly ten years!

JackPoach 6y ago

It's OK, if you are going fix it. Otherwise try living in a house where things are broken.

durnygbur 6y ago
So a rented place? Where the owner has no incentive to fix it, and the tenant oftentimes doesn't have the authorization to do so. Especially when the owning entity is focused exclusively on extracting the profits. You accept it, move out, or turn insane.
bloak 6y ago
Every house I've lived in has had several things - doors, window blinds, appliances - that only I can operate because it takes practice to work around their brokenness. It's only when I see someone else trying and failing to use them that I remember that they're broken. But it doesn't convince me to repair them. I get used to things and don't like unnecessary change.
afarrell 6y ago
Its much harder to fix things if people are unwilling to hear the idea that things are broken.

MattPalmer1086 6y ago

This article makes some very good points. I've been thinking about resilience for a while. Nothing is perfect, and if your system can only work with that assumption, you are bound for failure. I always loved the Erlang philosophy of let it crash. The idea you build a highly resilient system out of error prone parts rocked my world when I first encountered it. None of this is to say we should accept broken things or not try to make them better. We should instead accept some parts will always fail and design accordingly.

ArcMex 6y ago

I take this philosophy conversely when developing software. I expect my first version to be imperfect. I then chip out the imperfections. Before, I wanted to build the perfect system right away. Now failure is planned and part of the process. Some people call it TDD.

SunlightEdge 6y ago

This could have been written by ProjectRed a la cyberpunk. The company knew the game was full of bugs still but were under immense pressure to ship out the game way before it was ready.

rob74 6y ago
Well, most AAA title are released full of bugs - those are the "failures" the article mentions. But most of them manage to avoid the "catastrophe" of being deemed completely unplayable...

doggodaddo78 6y ago

Shrugging apathy is the enemy of the excellence (not perfection). Btw, with very simple programs, it is possible to have zero bugs but it is decreasingly possible as programs increase in complexity. As an example, Android is Humpty Dumpty. Manufacturers just try to glue as many pieces together as possible.

moocowtruck 6y ago

this is where as a user of software that is often times broken, I wish I had better protections on when/how to get my money back. Software is not obviously broken, and I feel like buyers have very little recourse

mshaler 6y ago

...that's how the light gets in.