5 ms·

The industry is annoyingly conservative but whizzkids pretending to have a better opinion about how to cure insert desease from isolated, mechanistic, non-stand

by DrNuke 8y ago

The industry is annoyingly conservative but whizzkids pretending to have a better opinion about how to cure insert desease from isolated, mechanistic, non-standard observations of few markers are dangerous.

Fomite 8y ago
Yeah - I work a lot in healthcare associated infections, and a good 80% of the stuff I read about "We're from Silicon Valley and we're here to help..." just ends with me going "They're going to kill people."
- cf498 8y ago
  I think this is something that isnt that present in our profession as it should be. If you are an electrical engineer, it is clear and present, that your products might kill someone if you screw up. The same goes for a mechanical engineer. And it will be your responsibility. If you screw up and should have known better, you might even go to jail for negligence. We on the other hand have the Therac-25. We claim to be on a level of engineering fields. We should also act that way.
  - Balgair 8y ago
    Agreed. The Order of the Engineer is a good place to start and bringing that tradition down into comp-eng/sci would be a nice start. https://en.wikipedia.org/wiki/Order_of_the_Engineer https://en.wikipedia.org/wiki/Order_of_the_Engineer
    - Latteland 8y ago
      Interesting, had not heard of that. What should we computer programmers, we flexible always trying new things without enough tried and true practices think of real engineers, what should they think of us?
      - ionised 8y ago
        Real engineers should probably look down on us from a great height.
        Balgair 8y ago
        Here's a more humorous take: https://imgs.xkcd.com/comics/voting_software.png https://imgs.xkcd.com/comics/voting_software.png
  - Someone 8y ago
    Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–present https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–p... or https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#2010–present https://en.wikipedia.org/wiki/List_of_structural_failures_an... with https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs, software doesn’t look that deadly.
    - jcranmer 8y ago
      People generally don't attribute failures to software bugs if there is anything else that contributes to failure. The list in particular is missing the loads of accidents caused by bad UI; incidents of "I was just following the GPS's/autopilot's directions" that result in death are missing (such as the KAL 007 airliner incident).
    - cgdcraig 8y ago
      I'd imagine a lot of places where software is part of a safety critical system also come with physical fail safes. An example being burst plates on pressure vessels, if the software that controls pressurization spasms and causes an over pressure the physical fail safe will prevent a catastrophic accident.
    - gruturo 8y ago
      Software doesn't have many opportunities to be deadly, and when said opportunities do exist, this is frequently in regulated fields where safeties are mandated by law (interlocks on a laser or a welding robot arm etc).
    - tremon 8y ago
      The Therac-25 is the go-to example because it is the best documented and least-contested case. Also, software usually isn't in ultimate control, most systems have additional (mechanical) safety controls. Mars Climate Orbiter, 1999: http://articles.latimes.com/1999/oct/01/news/mn-17288 http://articles.latimes.com/1999/oct/01/news/mn-17288. Undetected metric conversion error (though unclear if this was a manual process or software). Toyota unintended acceleration, 2010: https://www.edmunds.com/car-safety/for-toyota-owners-unintended-acceleration-lawsuit-settlement.html https://www.edmunds.com/car-safety/for-toyota-owners-uninten.... Toyota maintains that it was not caused by software error. Now let's list some cases where the software did have ultimate control: Schiaparelli EDM lander, 2016: https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/49696/ https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/496.... Faulty decision-making in the automated descent system due to input saturation. Tesla auto-pilot crash, 2016: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut.... Faulty decision-making due to incorrect image analysis. In this case, the additional safety controls (the driver himself) failed too. Uber car crash, 2018: https://money.cnn.com/2018/03/20/news/companies/self-driving-uber-death/index.html https://money.cnn.com/2018/03/20/news/companies/self-driving.... Faulty decision-making due to incorrect image analysis, though the investigation is still ongoing, so no definite cause determined (FAFAIK).
  - conanbatt 8y ago
    Software engineering is more like witch-doctoring than to civil engineering. It is not remotely price-efficient to prove completeness on programs. If you expect your doctor's software to be perfect, be prepared to pay for 2 software engineers salaries per doctor.
  - pimmen 8y ago
    I agree that we act recklessly sometimes, but I hope I'm not alone among software engineers in believing that this finding from the committee was just plain stupidity which should not be done by any engineer worth his or her salt: "AECL had never tested the Therac-25 with the combination of software and hardware until it was assembled at the hospital."
- Balgair 8y ago
  I've second hand experience with this. Though not a lethal situation (thank the heavens!) the mindset was dead on. A grad student in the BioEng lab was doing her PhD on a fall detection sensor for the elderly. Kinda like what the new Apple watch is supposed to do, but, you know, something that actually works. The lab was working with a device manufacturer that was looking to branch into medical devices as a new market and was partnered with the lab to do the initial PoC. Well, one day she came into the lab to discover that the company had shut down and remote bricked the devices overnight. They were still in early testing, so no people were harmed, which was really lucky. But if they had been under clinical trial already, there was a real possibility that real people out there would have been harmed severely (a 'bad' geriatric fall can become lethal). All emails to the company were unanswered. All phone calls went to voicemail. Nothing could be done. She was 5 years into her PhD on this and had to restart the whole project (she wasn't much of a coder, more a clinical trials person). In medicine, the SV mindset of 'move fast, break things' means that granny is breaking her hip and isn't going to last much longer. You CANNOT 'break things' in medicine. However, if you do get though clinical trials, hoooo baby! You essentially have a monopoly and will be rolling in it. MedTech investing is a lot like SV's VC investing, just a lot more careful.
  - cf498 8y ago
    What most people dont realize, depending on the country, those press for emergency buttons for the elderly are not considered an emergency call. Including the advertisement you see on buses targeting people fearing a heart attack. Meaning, it is not guaranteed that the message will go through. It is an app like any else with the same reliability. That was fortunately the point where my former boss declined a contract.
  - megy 8y ago
    > Kinda like what the new Apple watch is supposed to do, but, you know, something that actually works. Does the Apple watch not work? > But if they had been under clinical trial already, there was a real possibility that real people out there would have been harmed severely (a 'bad' geriatric fall can become lethal). How would that be different if they didn't have a device? They would have died anyway, right?

by DrNuke 8y ago

Fomite 8y ago

Yeah - I work a lot in healthcare associated infections, and a good 80% of the stuff I read about "We're from Silicon Valley and we're here to help..." just ends with me going "They're going to kill people."

cf498 8y ago
I think this is something that isnt that present in our profession as it should be. If you are an electrical engineer, it is clear and present, that your products might kill someone if you screw up. The same goes for a mechanical engineer. And it will be your responsibility. If you screw up and should have known better, you might even go to jail for negligence. We on the other hand have the Therac-25. We claim to be on a level of engineering fields. We should also act that way.
- Balgair 8y ago
  Agreed. The Order of the Engineer is a good place to start and bringing that tradition down into comp-eng/sci would be a nice start. https://en.wikipedia.org/wiki/Order_of_the_Engineer https://en.wikipedia.org/wiki/Order_of_the_Engineer
  - Latteland 8y ago
    Interesting, had not heard of that. What should we computer programmers, we flexible always trying new things without enough tried and true practices think of real engineers, what should they think of us?
    - ionised 8y ago
      Real engineers should probably look down on us from a great height.
      - Balgair 8y ago
        Here's a more humorous take: https://imgs.xkcd.com/comics/voting_software.png https://imgs.xkcd.com/comics/voting_software.png
- Someone 8y ago
  Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–present https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–p... or https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#2010–present https://en.wikipedia.org/wiki/List_of_structural_failures_an... with https://en.wikipedia.org/wiki/List_of_software_bugs https://en.wikipedia.org/wiki/List_of_software_bugs, software doesn’t look that deadly.
  - jcranmer 8y ago
    People generally don't attribute failures to software bugs if there is anything else that contributes to failure. The list in particular is missing the loads of accidents caused by bad UI; incidents of "I was just following the GPS's/autopilot's directions" that result in death are missing (such as the KAL 007 airliner incident).
  - cgdcraig 8y ago
    I'd imagine a lot of places where software is part of a safety critical system also come with physical fail safes. An example being burst plates on pressure vessels, if the software that controls pressurization spasms and causes an over pressure the physical fail safe will prevent a catastrophic accident.
  - gruturo 8y ago
    Software doesn't have many opportunities to be deadly, and when said opportunities do exist, this is frequently in regulated fields where safeties are mandated by law (interlocks on a laser or a welding robot arm etc).
  - tremon 8y ago
    The Therac-25 is the go-to example because it is the best documented and least-contested case. Also, software usually isn't in ultimate control, most systems have additional (mechanical) safety controls. Mars Climate Orbiter, 1999: http://articles.latimes.com/1999/oct/01/news/mn-17288 http://articles.latimes.com/1999/oct/01/news/mn-17288. Undetected metric conversion error (though unclear if this was a manual process or software). Toyota unintended acceleration, 2010: https://www.edmunds.com/car-safety/for-toyota-owners-unintended-acceleration-lawsuit-settlement.html https://www.edmunds.com/car-safety/for-toyota-owners-uninten.... Toyota maintains that it was not caused by software error. Now let's list some cases where the software did have ultimate control: Schiaparelli EDM lander, 2016: https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/49696/ https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/496.... Faulty decision-making in the automated descent system due to input saturation. Tesla auto-pilot crash, 2016: https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk https://www.theguardian.com/technology/2016/jun/30/tesla-aut.... Faulty decision-making due to incorrect image analysis. In this case, the additional safety controls (the driver himself) failed too. Uber car crash, 2018: https://money.cnn.com/2018/03/20/news/companies/self-driving-uber-death/index.html https://money.cnn.com/2018/03/20/news/companies/self-driving.... Faulty decision-making due to incorrect image analysis, though the investigation is still ongoing, so no definite cause determined (FAFAIK).
- conanbatt 8y ago
  Software engineering is more like witch-doctoring than to civil engineering. It is not remotely price-efficient to prove completeness on programs. If you expect your doctor's software to be perfect, be prepared to pay for 2 software engineers salaries per doctor.
- pimmen 8y ago
  I agree that we act recklessly sometimes, but I hope I'm not alone among software engineers in believing that this finding from the committee was just plain stupidity which should not be done by any engineer worth his or her salt: "AECL had never tested the Therac-25 with the combination of software and hardware until it was assembled at the hospital."
Balgair 8y ago
I've second hand experience with this. Though not a lethal situation (thank the heavens!) the mindset was dead on. A grad student in the BioEng lab was doing her PhD on a fall detection sensor for the elderly. Kinda like what the new Apple watch is supposed to do, but, you know, something that actually works. The lab was working with a device manufacturer that was looking to branch into medical devices as a new market and was partnered with the lab to do the initial PoC. Well, one day she came into the lab to discover that the company had shut down and remote bricked the devices overnight. They were still in early testing, so no people were harmed, which was really lucky. But if they had been under clinical trial already, there was a real possibility that real people out there would have been harmed severely (a 'bad' geriatric fall can become lethal). All emails to the company were unanswered. All phone calls went to voicemail. Nothing could be done. She was 5 years into her PhD on this and had to restart the whole project (she wasn't much of a coder, more a clinical trials person). In medicine, the SV mindset of 'move fast, break things' means that granny is breaking her hip and isn't going to last much longer. You CANNOT 'break things' in medicine. However, if you do get though clinical trials, hoooo baby! You essentially have a monopoly and will be rolling in it. MedTech investing is a lot like SV's VC investing, just a lot more careful.
- cf498 8y ago
  What most people dont realize, depending on the country, those press for emergency buttons for the elderly are not considered an emergency call. Including the advertisement you see on buses targeting people fearing a heart attack. Meaning, it is not guaranteed that the message will go through. It is an app like any else with the same reliability. That was fortunately the point where my former boss declined a contract.
- megy 8y ago
  > Kinda like what the new Apple watch is supposed to do, but, you know, something that actually works. Does the Apple watch not work? > But if they had been under clinical trial already, there was a real possibility that real people out there would have been harmed severely (a 'bad' geriatric fall can become lethal). How would that be different if they didn't have a device? They would have died anyway, right?