5 ms·

This is why I think that we're getting to the point where software developers need to be trained in ethics and licensed in the same way as professional engineer

by bcrl 1y ago

This is why I think that we're getting to the point where software developers need to be trained in ethics and licensed in the same way as professional engineers are around the world. The people building the tools need to be held accountable when design choices kill people.

WalterBright 1y ago
Doesn't the FDA already regulate medical software?
- nradov 1y ago
  To an extent, but only if the software meets the criteria for being legally classified as a "medical device". And at the lower classification levels the regulations are very light. You don't necessarily have to prove any level of reliability or usability. https://www.fda.gov/medical-devices/classify-your-medical-device/how-determine-if-your-product-medical-device https://www.fda.gov/medical-devices/classify-your-medical-de...
  - Neywiny 1y ago
    Ex employee of the division of imagine devices and software reliability here. I have nothing to add, and I'm not qualified to confirm or deny your comment. Just interesting to see a mention of my past life
- codeulike 1y ago
  But thats part of the problem, as well as part of the solution. The FDA regulations say the software must do this and must do that, and you end up with thousands of pop up alerts. I think the regulations are well meaning but they way they translate into software is not realistic. In the article an FDA-regulated pop-up is part of the story.
  - nradov 1y ago
    There are no FDA regulations that require EHR software to display pop-up alerts for possible sepsis.
    - deleted 1y ago
      [deleted]
Zak 1y ago
Most software development is not much like engineering. That's probably the right choice for low-stakes consumer products, but wouldn't be accepted in avionics and shouldn't be accepted in medical software that impacts life safety.
- bcrl 1y ago
  Ethics is applicable to many situations outside of life critical systems. Engineers take ethics very seriously because it is important, and I think it is important for software developers as well. The first time I worked closely with a Professional Engineer on a project, it was eye opening as to how the concern of doing things the ethical way shaped the interaction... Everything from identifying potential business conflicts to ensuring data collection for inputs to the design process was done correctly and traceable. Corporations already push enough random HR requirements on tech employees. I can't see how having a semester or 2 of ethics courses is particularly onerous.
  - Zak 1y ago
    I think including ethics in the curriculum at both the high school and university level is a great idea. I do not think requiring software developers to be licensed is a good idea at all. Safety-critical software projects should have a licensed engineer in a supervisory role.
    - bcrl 1y ago
      Licensing takes many forms. Some are onerous, some are not. Working At Heights training takes 0.5-1 day of training every 3 years where I live, and it's mandated because it saves lives. Yes, it's a form of licensing, but it's an easy box to check because it's accessible. For example: running experiments on humans without their prior consent is considered unethical. However, large tech firms routinely run A / B tests on their users without providing Informed Consent. If software developers were trained like engineers, they would be ethically obligated to obtain Informed Consent prior to engaging in this kind of conduct. More importantly, when software developers realize they were being experimented upon (as happened in the Linux kernel community a while back), they were justifiably outraged. I know this isn't going to be a popular position to hold here, but there's a lot of harm being done by unethical practices that are currently widespread in big tech. Ephemeral ads that prey on the elderly would be considered Not Good by anyone that has seen a parent fall prey to them, yet there doesn't seem to be any concern whatsoever amongst the industry giants themselves to prevent this practice. So long as an ad brings in money, it's good to run seems to be the bar for advertisers at present, and I don't think that's good for society. When there are no ethical considerations given to the consequences of an action, unethical outcomes are inevitable. Fixing that starts with learning, which is something every great developer already does.
terminalshort 1y ago
Bureaucracy won't help anything. You can't train people to be more ethical.
- s5300 1y ago
  [dead]
- justin66 1y ago
  Bureaucracy and accountability are two separate concepts.
ozgrakkurt 1y ago
It is doctor’s responsibility to know the patient might have such a condition, software system is just for helping

by bcrl 1y ago

WalterBright 1y ago

Doesn't the FDA already regulate medical software?

nradov 1y ago
To an extent, but only if the software meets the criteria for being legally classified as a "medical device". And at the lower classification levels the regulations are very light. You don't necessarily have to prove any level of reliability or usability. https://www.fda.gov/medical-devices/classify-your-medical-device/how-determine-if-your-product-medical-device https://www.fda.gov/medical-devices/classify-your-medical-de...
- Neywiny 1y ago
  Ex employee of the division of imagine devices and software reliability here. I have nothing to add, and I'm not qualified to confirm or deny your comment. Just interesting to see a mention of my past life
codeulike 1y ago
But thats part of the problem, as well as part of the solution. The FDA regulations say the software must do this and must do that, and you end up with thousands of pop up alerts. I think the regulations are well meaning but they way they translate into software is not realistic. In the article an FDA-regulated pop-up is part of the story.
- nradov 1y ago
  There are no FDA regulations that require EHR software to display pop-up alerts for possible sepsis.
  - deleted 1y ago
    [deleted]

Zak 1y ago

Most software development is not much like engineering. That's probably the right choice for low-stakes consumer products, but wouldn't be accepted in avionics and shouldn't be accepted in medical software that impacts life safety.

bcrl 1y ago
Ethics is applicable to many situations outside of life critical systems. Engineers take ethics very seriously because it is important, and I think it is important for software developers as well. The first time I worked closely with a Professional Engineer on a project, it was eye opening as to how the concern of doing things the ethical way shaped the interaction... Everything from identifying potential business conflicts to ensuring data collection for inputs to the design process was done correctly and traceable. Corporations already push enough random HR requirements on tech employees. I can't see how having a semester or 2 of ethics courses is particularly onerous.
- Zak 1y ago
  I think including ethics in the curriculum at both the high school and university level is a great idea. I do not think requiring software developers to be licensed is a good idea at all. Safety-critical software projects should have a licensed engineer in a supervisory role.
  - bcrl 1y ago
    Licensing takes many forms. Some are onerous, some are not. Working At Heights training takes 0.5-1 day of training every 3 years where I live, and it's mandated because it saves lives. Yes, it's a form of licensing, but it's an easy box to check because it's accessible. For example: running experiments on humans without their prior consent is considered unethical. However, large tech firms routinely run A / B tests on their users without providing Informed Consent. If software developers were trained like engineers, they would be ethically obligated to obtain Informed Consent prior to engaging in this kind of conduct. More importantly, when software developers realize they were being experimented upon (as happened in the Linux kernel community a while back), they were justifiably outraged. I know this isn't going to be a popular position to hold here, but there's a lot of harm being done by unethical practices that are currently widespread in big tech. Ephemeral ads that prey on the elderly would be considered Not Good by anyone that has seen a parent fall prey to them, yet there doesn't seem to be any concern whatsoever amongst the industry giants themselves to prevent this practice. So long as an ad brings in money, it's good to run seems to be the bar for advertisers at present, and I don't think that's good for society. When there are no ethical considerations given to the consequences of an action, unethical outcomes are inevitable. Fixing that starts with learning, which is something every great developer already does.

terminalshort 1y ago

Bureaucracy won't help anything. You can't train people to be more ethical.

s5300 1y ago
[dead]
justin66 1y ago
Bureaucracy and accountability are two separate concepts.

ozgrakkurt 1y ago

It is doctor’s responsibility to know the patient might have such a condition, software system is just for helping