7 ms·

After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me person

After a lifetime of fixing bugs I’m often surprised that computer systems work at all. I had something similar to the failed post office system affect me personally, and everyone just rejects that the computer system could be at fault, even when presented with evidence, as it too too much work for them to try to understand.

HPsquared 3y ago
Occam's razor can be double-edged sometimes.
SanderNL 3y ago
It is too much work. It’s way too complex.
bmitc 3y ago
> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.
- zubairq 3y ago
  That makes sense, yes we have trained ourselves to see things that don’t work as working , like frogs in boiling water!
  - psychoslave 3y ago
    While the metaphore is great, it's actually stand on a myth. That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits. https://en.m.wikipedia.org/wiki/Boiling_frog https://en.m.wikipedia.org/wiki/Boiling_frog
- vasco 3y ago
  Any sufficiently complex system operates constantly in failure mode
  - uoaei 3y ago
    This is an incomplete view of complex systems. It is more accurate to say they operate constantly in recovery mode.
  - bmacho 3y ago
    > The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode. https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics now this is depressing. I hate everything and I live in a constant 'everything is shit' state of mind, but this book hates things way more than me.
  - simonh 3y ago
    Which of course also applies to human cognitive processes. We’ve had millions of years of evolution to prune out problems, but in many cases the problems were papered over by a kludge that introduced other problems, which were then kludged, etc. No wonder the complex systems we build are such a mess. They’re a reflection of ourselves.
    - taneq 3y ago
      They’re not yet a reflection of ourselves, only of our thought processes. They’re flawed, as are we, but we’ve been ruthlessly battle-tested in the only arena that matters.
- hgomersall 3y ago
  With medical devices you have to act on the assumption that the software will fail and that mitigations should be in place to deal with the consequences of failure.
- bostik 3y ago
  People outsource their critical thinking all the time, but with computers the failures are more egregious. There is less recourse to correct things, and information propagating through systems can make the error more persistent. Also: people take the verdict from a computer as a given even when reality and computer disagree. I've used this story a couple of times before on HN, and if it hadn't happened to mother's coworker, I would have thought it a preposterous joke. Mom's colleague was buying something during her lunch break, and had her card rejected. Got the same result in another store, so obviously thought her card had just broken. Went to her local bank branch to ask what's wrong. The clerk took the card, looked up from her screen and against the evidence in front of eyes declared: "You are dead." Once that kind of flag starts to make its way through our computerised systems, it is very difficult to reverse. It took several weeks to get the error corrected, and in the meanwhile other systems that trusted the bank as a source of truth had picked up the information and started to apply it to their accounts. Eventually it came to light that a full namesake's estate had had their accounts finally closed, and the clerk doing that had done only a name search when applying the "account terminated, owner deceased" flag to the accounts. Bank never took responsibility for their process failure. They fought for a good time to even have to accept that there had been an error in the first place. Which incidentally is not entirely unlike the situation in this perversion of justice.
  - RetroTechie 3y ago
    "Which incidentally is not entirely unlike the situation in this perversion of justice." So true. I went through the Wikipedia article, and couldn't escape this feeling: 1) Some errors were made (faulty IT system). 2) Handling of said errors expands enormously into various reports, IT vendor denying their system is faulty, legal proceedings, criminal convictions "tainted" & overturned, etc. To the point where original errors almost seem irrelevant within the bigger frame of subsequent events. Imho all the mistakes were made in the processes concerning 2), and 1) is a tiny side-show at most. IT systems can fail, period. What matters is how that's handled.
  - citizenpaul 3y ago
    I had a similar experience as your moms colleague. I transferred 20k from savings to checking. During the process I think what happened is the web banking page froze and allowed me to hold the enter key on confirm that repeated several times. I ended up with 200k in my account that came as a negative 180k balance to my saving account sent to my checking account. (values rounded for story simplicity) This meant that not only did their online portal not have controls to make sure that internal money transfers are confirmed before processing. They also had no controls to make sure that when an internal transfer is done that the money is there to be transferred. I waited a couple days expecting a call. Nope. At the end of week I call them. Get transferred around a bunch. The whole time everyone is saying everything is good nothing is wrong, there is no problem. I 100% certain every single one of them was lying through their teeth while they go the FBI on the phone because of the automatic assumption that the computers are right and I am automatically a criminal. There is no way all those people saw a -200k balance in a saving account and thought it was fine. (BTW that means it is standard bank policy to lie to a customer when something is wrong) So what happened. They immediately zeroed my accounts and froze all my bank activity for three months while they "investigated" Then they only returned 15K of the original 20k that I moved so they effectively stole 5k from me. I spend another 6 months trying to get that money back and did not succeed. I would have been screwed if I had not been born poor and was familiar with the banking system corruption. I have my income,spending,saving and investing split among different banks and banks accounts in case of something like this ever happened. The bank initials are WF.
- PartiallyTyped 3y ago
  > How can it possibly all work? What must not to fail has very rigorous standards, e.g. Nasa, aerospace, and medical devices. Everything is built like an onion of failsafes.
  - tpmoney 3y ago
    Even the stuff that's built to rigorous standards is mostly built with a serials of failsafes because the same issues of being able to trust the thing apply. But it's fine because like you note, failsafes limit the effect of any individual failure. Beyond that too, I think it's just important for people to realize that most things just don't matter that much. Everything we do eventually has to reach human scale and at human scale, there's so much room for slop and correction that most failures don't matter. Missing a paycheck because the payroll computer dropped your row is awful, but society can be built around allowing people to recover from that. Late payment forgiveness, forbearance, procedures around cutting a new check early. I think the biggest danger about the "presumption that computers are working correctly" is in it's ability to erode our understand of the fact that "to err is human". It's too easy if you assume the machines are perfect to assume that any errors are therefore malicious, and so it's easy to go from a system where you have "oops my tax software messed up, no big deal I'll file an amended return" to "the only way for a mistake like this to happen is to intentionally put in bad data, so you must be guilty of tax fraud". Or even worse "the computer won't let me do it". I understand you want computers to have check in place that prevent the overriding of procedure and you don't want to give your low level employees the ability to override procedure at will. I also think that EVERY system should be built such that it is possible with a certain combination of people on site at the point of contact to override ANY check in the system to make the state of the system match what it should be. My specific example in this case was a problem with a prescription at the pharmacy. The script had been partially filled because they had stocking problems, but instead of going into the "partial" state, it was closed as filled completely. The tech knew what had happened and knew we still were owed the rest of the script. The pharmacist knew what had happened and agreed we were owed the rest of the script. The store manager agreed. And yet no one could change the state of the system. The only option was to get a new script from the doctor for a new fill entirely. That's the danger of assuming everything is working correctly all the time.
- K0balt 3y ago
  To me, modern computers, in particular, are an enigma, and I am an electronic engineer specializing in digital systems. Imagine something with 200 billion individual parts with individual functions (and that’s just the cpu) that is presumed to work perfectly flawlessly billions of times a second for a decade. By contrast, the crankshaft on your car goes through a bout 1 billion revolutions before it is worn out. Your heart might beat 4 billion times before you die, if you are really lucky. The precision and reliability we take for granted from modern computing hardware is dumbfounding.
- sifar 3y ago
  >> It boggles my mind, because I have seen how software gets built. How can it possibly all work? Wait till you uncover what the SW runs on :). From math, processor/memory, rtl design, verification, tools that generate the circuits to the fabs that create the ICs. And when I submit this comment it appears for others to view. It is a miracle this entire structure works. It never ceases to amaze me.
bboygravity 3y ago
I would add some nuiance and state that a lot of popular software is broken to the point that it's main function (the reason that the software exists) is not working. Some examples: -- The websites of various airlines (TAP, Vueling, KLM) have often had bugs in them (in the past 2 years when I used them) to the point that they don't allow you to book tickets or log-in with your account. I would say they where broken more often than not. -- Instagram and Google have been deactivating people's accounts (automatically) for no apparent reason. I've had this happen with Instagram 3x in the space of 2 months after which I just gave up and stopped using it. -- Skype, no explanation needed for those who use it. -- Teams fails in many amazing ways no matter how often they "update" it. I've heard stories about the mute button not actually muting, Teams suddenly reversing the order of the letters in your sentence as you're typing in a chat, it just shuts down in the middle of a call, unresponsive, etc etc etc. The amount of productivity these bugs in this software burn worldwide must be significant and the real cost to companies in the form of wasted man-hours is likely higher than whatever they pay in licensing fees to Microsoft. -- Telegram can't handle an external monitor (the scaling gets all messed up causing you to have to restart Telegram to fix it). -- iCloud takes between seconds and an hour or so to sync files even when all devices are on a good connection.
- hulitu 3y ago
  >Teams suddenly reversing the order of the letters in your sentence as you're typing in a chat I think they had a button for this on the right click menu. TBH, Teams is the quintesence of today's SW development state of the art.
- bboygravity 3y ago
  To add some meta irony to my previous post: as you can see in my previous post the formatting got all mangled. The new-lines (enters) that I typed got ignored upon posting. I cannot think of any good reason for that "functionality" from a user perspective? Also I can't edit that post anymore to fix it. And to think that HN is an example of relatively good software that seems un-bloated, super stable, fast and seems to always work :P
  - 3y ago
vladms 3y ago
Most people have a tendency to take the easy path and think stuff is perfect rather than accept everything works "approximately", although we see again and again proofs of the contrary. But as an engineer I find this natural and efficient! You should prioritize and some bugs are less important than others. If anything more effort should be put in education, health and fundamental research rather than fixing all bugs in all software.
d_tr 3y ago
The rapid growth of the software industry is both the reason for the unsalvageable mess and for companies being able to spend obscene amounts of money to somehow keep shit together. At least that's how I feel about it.
almatabata 3y ago
Sadly this often happens because often times the machine works 99% of the time. So when the thing starts failing in 1% of the cases people will dismiss it because it worked in the previous 99% of the time. People have trained themselves to trust the results so questioning it does not even enter their minds. It does remind me though of the book Systemantics: https://www.amazon.com/Systemantics-Systems-Work-Especially-They/dp/0812906748 https://www.amazon.com/Systemantics-Systems-Work-Especially-... One of the core thesis in the book is : The system itself does not actually do what it says it is doing (The Operational Fallacy). This people and engineers as well often forget. Just because the system says it does X does not mean it actually does it. People would do well to cultivate a healthy skepticism towards those complex IT systems. They might not fail most of the time but they inevitably will.

HPsquared 3y ago

Occam's razor can be double-edged sometimes.

SanderNL 3y ago

It is too much work. It’s way too complex.

bmitc 3y ago

> I’m often surprised that computer systems work at all. I also remain contunually amazed that anything works. It boggles my mind, because I have seen how software gets built. How can it possibly all work? I think my answer to this conundrum and seeming paradox is that it doesn't all work. We have trained ourselves and been trained to accept things that aren't working as something else. And like you say, people treat the computer system either as all knowing and infallible or as a scapegoat.

zubairq 3y ago
That makes sense, yes we have trained ourselves to see things that don’t work as working , like frogs in boiling water!
- psychoslave 3y ago
  While the metaphore is great, it's actually stand on a myth. That's all the more interesting in such a context to evocate it. We often take for granted what feels like aptly grounded easy to grab information. I think that is where LLM feels the most as a terrible threat to human societies, because they can at large scale provide customized very convincing utter bullshits. https://en.m.wikipedia.org/wiki/Boiling_frog https://en.m.wikipedia.org/wiki/Boiling_frog
vasco 3y ago
Any sufficiently complex system operates constantly in failure mode
- uoaei 3y ago
  This is an incomplete view of complex systems. It is more accurate to say they operate constantly in recovery mode.
- bmacho 3y ago
  > The Fundamental Failure-Mode Theorem (F.F.T.): complex systems usually operate in a failure mode. https://en.wikipedia.org/wiki/Systemantics https://en.wikipedia.org/wiki/Systemantics now this is depressing. I hate everything and I live in a constant 'everything is shit' state of mind, but this book hates things way more than me.
- simonh 3y ago
  Which of course also applies to human cognitive processes. We’ve had millions of years of evolution to prune out problems, but in many cases the problems were papered over by a kludge that introduced other problems, which were then kludged, etc. No wonder the complex systems we build are such a mess. They’re a reflection of ourselves.
  - taneq 3y ago
    They’re not yet a reflection of ourselves, only of our thought processes. They’re flawed, as are we, but we’ve been ruthlessly battle-tested in the only arena that matters.
hgomersall 3y ago
With medical devices you have to act on the assumption that the software will fail and that mitigations should be in place to deal with the consequences of failure.
bostik 3y ago
People outsource their critical thinking all the time, but with computers the failures are more egregious. There is less recourse to correct things, and information propagating through systems can make the error more persistent. Also: people take the verdict from a computer as a given even when reality and computer disagree. I've used this story a couple of times before on HN, and if it hadn't happened to mother's coworker, I would have thought it a preposterous joke. Mom's colleague was buying something during her lunch break, and had her card rejected. Got the same result in another store, so obviously thought her card had just broken. Went to her local bank branch to ask what's wrong. The clerk took the card, looked up from her screen and against the evidence in front of eyes declared: "You are dead." Once that kind of flag starts to make its way through our computerised systems, it is very difficult to reverse. It took several weeks to get the error corrected, and in the meanwhile other systems that trusted the bank as a source of truth had picked up the information and started to apply it to their accounts. Eventually it came to light that a full namesake's estate had had their accounts finally closed, and the clerk doing that had done only a name search when applying the "account terminated, owner deceased" flag to the accounts. Bank never took responsibility for their process failure. They fought for a good time to even have to accept that there had been an error in the first place. Which incidentally is not entirely unlike the situation in this perversion of justice.
- RetroTechie 3y ago
  "Which incidentally is not entirely unlike the situation in this perversion of justice." So true. I went through the Wikipedia article, and couldn't escape this feeling: 1) Some errors were made (faulty IT system). 2) Handling of said errors expands enormously into various reports, IT vendor denying their system is faulty, legal proceedings, criminal convictions "tainted" & overturned, etc. To the point where original errors almost seem irrelevant within the bigger frame of subsequent events. Imho all the mistakes were made in the processes concerning 2), and 1) is a tiny side-show at most. IT systems can fail, period. What matters is how that's handled.
- citizenpaul 3y ago
  I had a similar experience as your moms colleague. I transferred 20k from savings to checking. During the process I think what happened is the web banking page froze and allowed me to hold the enter key on confirm that repeated several times. I ended up with 200k in my account that came as a negative 180k balance to my saving account sent to my checking account. (values rounded for story simplicity) This meant that not only did their online portal not have controls to make sure that internal money transfers are confirmed before processing. They also had no controls to make sure that when an internal transfer is done that the money is there to be transferred. I waited a couple days expecting a call. Nope. At the end of week I call them. Get transferred around a bunch. The whole time everyone is saying everything is good nothing is wrong, there is no problem. I 100% certain every single one of them was lying through their teeth while they go the FBI on the phone because of the automatic assumption that the computers are right and I am automatically a criminal. There is no way all those people saw a -200k balance in a saving account and thought it was fine. (BTW that means it is standard bank policy to lie to a customer when something is wrong) So what happened. They immediately zeroed my accounts and froze all my bank activity for three months while they "investigated" Then they only returned 15K of the original 20k that I moved so they effectively stole 5k from me. I spend another 6 months trying to get that money back and did not succeed. I would have been screwed if I had not been born poor and was familiar with the banking system corruption. I have my income,spending,saving and investing split among different banks and banks accounts in case of something like this ever happened. The bank initials are WF.
PartiallyTyped 3y ago
> How can it possibly all work? What must not to fail has very rigorous standards, e.g. Nasa, aerospace, and medical devices. Everything is built like an onion of failsafes.
- tpmoney 3y ago
  Even the stuff that's built to rigorous standards is mostly built with a serials of failsafes because the same issues of being able to trust the thing apply. But it's fine because like you note, failsafes limit the effect of any individual failure. Beyond that too, I think it's just important for people to realize that most things just don't matter that much. Everything we do eventually has to reach human scale and at human scale, there's so much room for slop and correction that most failures don't matter. Missing a paycheck because the payroll computer dropped your row is awful, but society can be built around allowing people to recover from that. Late payment forgiveness, forbearance, procedures around cutting a new check early. I think the biggest danger about the "presumption that computers are working correctly" is in it's ability to erode our understand of the fact that "to err is human". It's too easy if you assume the machines are perfect to assume that any errors are therefore malicious, and so it's easy to go from a system where you have "oops my tax software messed up, no big deal I'll file an amended return" to "the only way for a mistake like this to happen is to intentionally put in bad data, so you must be guilty of tax fraud". Or even worse "the computer won't let me do it". I understand you want computers to have check in place that prevent the overriding of procedure and you don't want to give your low level employees the ability to override procedure at will. I also think that EVERY system should be built such that it is possible with a certain combination of people on site at the point of contact to override ANY check in the system to make the state of the system match what it should be. My specific example in this case was a problem with a prescription at the pharmacy. The script had been partially filled because they had stocking problems, but instead of going into the "partial" state, it was closed as filled completely. The tech knew what had happened and knew we still were owed the rest of the script. The pharmacist knew what had happened and agreed we were owed the rest of the script. The store manager agreed. And yet no one could change the state of the system. The only option was to get a new script from the doctor for a new fill entirely. That's the danger of assuming everything is working correctly all the time.
K0balt 3y ago
To me, modern computers, in particular, are an enigma, and I am an electronic engineer specializing in digital systems. Imagine something with 200 billion individual parts with individual functions (and that’s just the cpu) that is presumed to work perfectly flawlessly billions of times a second for a decade. By contrast, the crankshaft on your car goes through a bout 1 billion revolutions before it is worn out. Your heart might beat 4 billion times before you die, if you are really lucky. The precision and reliability we take for granted from modern computing hardware is dumbfounding.
sifar 3y ago
>> It boggles my mind, because I have seen how software gets built. How can it possibly all work? Wait till you uncover what the SW runs on :). From math, processor/memory, rtl design, verification, tools that generate the circuits to the fabs that create the ICs. And when I submit this comment it appears for others to view. It is a miracle this entire structure works. It never ceases to amaze me.

bboygravity 3y ago

I would add some nuiance and state that a lot of popular software is broken to the point that it's main function (the reason that the software exists) is not working. Some examples: -- The websites of various airlines (TAP, Vueling, KLM) have often had bugs in them (in the past 2 years when I used them) to the point that they don't allow you to book tickets or log-in with your account. I would say they where broken more often than not. -- Instagram and Google have been deactivating people's accounts (automatically) for no apparent reason. I've had this happen with Instagram 3x in the space of 2 months after which I just gave up and stopped using it. -- Skype, no explanation needed for those who use it. -- Teams fails in many amazing ways no matter how often they "update" it. I've heard stories about the mute button not actually muting, Teams suddenly reversing the order of the letters in your sentence as you're typing in a chat, it just shuts down in the middle of a call, unresponsive, etc etc etc. The amount of productivity these bugs in this software burn worldwide must be significant and the real cost to companies in the form of wasted man-hours is likely higher than whatever they pay in licensing fees to Microsoft. -- Telegram can't handle an external monitor (the scaling gets all messed up causing you to have to restart Telegram to fix it). -- iCloud takes between seconds and an hour or so to sync files even when all devices are on a good connection.

hulitu 3y ago
>Teams suddenly reversing the order of the letters in your sentence as you're typing in a chat I think they had a button for this on the right click menu. TBH, Teams is the quintesence of today's SW development state of the art.
bboygravity 3y ago
To add some meta irony to my previous post: as you can see in my previous post the formatting got all mangled. The new-lines (enters) that I typed got ignored upon posting. I cannot think of any good reason for that "functionality" from a user perspective? Also I can't edit that post anymore to fix it. And to think that HN is an example of relatively good software that seems un-bloated, super stable, fast and seems to always work :P
- 3y ago

vladms 3y ago

Most people have a tendency to take the easy path and think stuff is perfect rather than accept everything works "approximately", although we see again and again proofs of the contrary. But as an engineer I find this natural and efficient! You should prioritize and some bugs are less important than others. If anything more effort should be put in education, health and fundamental research rather than fixing all bugs in all software.

d_tr 3y ago

The rapid growth of the software industry is both the reason for the unsalvageable mess and for companies being able to spend obscene amounts of money to somehow keep shit together. At least that's how I feel about it.

almatabata 3y ago

Sadly this often happens because often times the machine works 99% of the time. So when the thing starts failing in 1% of the cases people will dismiss it because it worked in the previous 99% of the time. People have trained themselves to trust the results so questioning it does not even enter their minds. It does remind me though of the book Systemantics: https://www.amazon.com/Systemantics-Systems-Work-Especially-They/dp/0812906748 https://www.amazon.com/Systemantics-Systems-Work-Especially-... One of the core thesis in the book is : The system itself does not actually do what it says it is doing (The Operational Fallacy). This people and engineers as well often forget. Just because the system says it does X does not mean it actually does it. People would do well to cultivate a healthy skepticism towards those complex IT systems. They might not fail most of the time but they inevitably will.