4 ms·

Mine also focused on Therac-25, but I'm not from Berkeley. It's likely that it was used throughout several campuses. Though I feel that's not so much about eth

Mine also focused on Therac-25, but I'm not from Berkeley. It's likely that it was used throughout several campuses. Though I feel that's not so much about ethics as it was appropriate safeguards/testing given the problem space.

jrockway 9y ago
I think it's a pretty common ethics case study. Therac-25 almost seems like a meme to me these days: "What are the ethical implications in the field of software engineering?" "Therac-25." I don't think we've gotten any better, either. I'm surprised there aren't thousands of Therac-25s killing people every day, actually. I guess the hardware engineers got better at turning off software, because I don't think software engineers got better at software engineering.
- Gibbon1 9y ago
  > I guess the hardware engineers got better at turning off software, because I don't think software engineers got better at software engineering. Years ago (like 25 years) I read an author discussing safety critical system. He flat out said, safety must be a primary system level design goal. Attempts to band aid it in will fail. Properly designed systems have safeties, interlocks, and monitoring subsystems to prevent software bugs from propagating to the point where people are harmed. People designing safety critical systems tend to completely distrust software.
- bipson 9y ago
  Huh, I think I never heard of the Therac-25. We instead talked about flight Lufthansa 2904 [1] (Airbus A320-211 Warsaw accident), also as a case of overconfident engineers (although Wikipedia mostly blames the weather report). [1] https://en.wikipedia.org/wiki/Lufthansa_Flight_2904 https://en.wikipedia.org/wiki/Lufthansa_Flight_2904
noir_lord 9y ago
Therac-25 featured on our syllabus here in the north of England. Its a pretty infamous case.
DaiPlusPlus 9y ago
The "ethics" aspect came from the conduct of the engineers behind Therac that did not take the reports of malfunctions seriously, including a fatality - they were too overconfident and arrogant - as a result it took 2 more people to die, and many more seriously injured, before the problem was identified and fixed. We also covered other ethics-related case-studies that covered project-mismanagement leading to massive cost-overruns, especially of public-service projects - particularly when those overruns are caused by malfeasance instead of incompetence or ever-changing client requirements (e.g. embezzlement, fraudulent conduct, etc). Another one was the ethics of personal data collection.
TuringNYC 9y ago
To loop others in: Summary: https://en.wikipedia.org/wiki/Therac-25#Problem_description https://en.wikipedia.org/wiki/Therac-25#Problem_description Full Reading from Berkeley course: https://www-inst.cs.berkeley.edu/~cs61a/reader/Therac-25.pdf https://www-inst.cs.berkeley.edu/~cs61a/reader/Therac-25.pdf

jrockway 9y ago

I think it's a pretty common ethics case study. Therac-25 almost seems like a meme to me these days: "What are the ethical implications in the field of software engineering?" "Therac-25." I don't think we've gotten any better, either. I'm surprised there aren't thousands of Therac-25s killing people every day, actually. I guess the hardware engineers got better at turning off software, because I don't think software engineers got better at software engineering.

Gibbon1 9y ago
> I guess the hardware engineers got better at turning off software, because I don't think software engineers got better at software engineering. Years ago (like 25 years) I read an author discussing safety critical system. He flat out said, safety must be a primary system level design goal. Attempts to band aid it in will fail. Properly designed systems have safeties, interlocks, and monitoring subsystems to prevent software bugs from propagating to the point where people are harmed. People designing safety critical systems tend to completely distrust software.
bipson 9y ago
Huh, I think I never heard of the Therac-25. We instead talked about flight Lufthansa 2904 [1] (Airbus A320-211 Warsaw accident), also as a case of overconfident engineers (although Wikipedia mostly blames the weather report). [1] https://en.wikipedia.org/wiki/Lufthansa_Flight_2904 https://en.wikipedia.org/wiki/Lufthansa_Flight_2904

noir_lord 9y ago

Therac-25 featured on our syllabus here in the north of England. Its a pretty infamous case.

DaiPlusPlus 9y ago

The "ethics" aspect came from the conduct of the engineers behind Therac that did not take the reports of malfunctions seriously, including a fatality - they were too overconfident and arrogant - as a result it took 2 more people to die, and many more seriously injured, before the problem was identified and fixed. We also covered other ethics-related case-studies that covered project-mismanagement leading to massive cost-overruns, especially of public-service projects - particularly when those overruns are caused by malfeasance instead of incompetence or ever-changing client requirements (e.g. embezzlement, fraudulent conduct, etc). Another one was the ethics of personal data collection.

TuringNYC 9y ago

To loop others in: Summary: https://en.wikipedia.org/wiki/Therac-25#Problem_description https://en.wikipedia.org/wiki/Therac-25#Problem_description Full Reading from Berkeley course: https://www-inst.cs.berkeley.edu/~cs61a/reader/Therac-25.pdf https://www-inst.cs.berkeley.edu/~cs61a/reader/Therac-25.pdf