4 ms·

I know the department of defense used to use Ada because it was a “safety critical” language. I feel like when it comes to this devices there should be more str

I know the department of defense used to use Ada because it was a “safety critical” language. I feel like when it comes to this devices there should be more stringent guidelines around coding practices. Is that not the case?

kragen 3y ago
there are, that's why insulin pump firmware is developed using obsolete and error-prone methods by substandard engineers and why you can't get the source for your insulin pump firmware and reflash it with a bug-fixed version
- chpatrick 3y ago
  I see where you're coming from, but I don't think letting anyone mess with the equipment keeping them alive is a good idea either. I really love open source IoT stuff for example and I think it's a great that people are trying to take real ownership of their devices. However it's different if the penalty for failure is your lights not working or immediate death. I think a better solution would be if the source was required to be public so people can identify bugs, but without giving users the ability to flash.
  - noir_lord 3y ago
    > I think a better solution would be if the source was required to be public so people can identify bugs, but without giving users the ability to flash. That would be a solution, still has the problem of "how do you confirm the bug if you can't run the code". Finding those kinds of subtle bugs pretty much requires you be able to execute it.
  - kragen 3y ago
    i do, because not 'letting anyone mess with the equipment keeping them alive' is, obviously, condemning some of them to deaths that they would avoid if they were allowed to i think you would need an overwhelmingly large number of prevented suicides and avoided foolish accidents to justify such a monstrous evil the balance of deaths is very likely to work out the other way around, in fact, because people reflashing their own insulin pumps have much stronger incentives to maximize their survival chances than any possible regulatory regime would, much less the one that actually exists the policy of 'let us make laws and set up regulatory agencies to prevent people from harming themselves' has such an astoundingly bad track record that it always surprises me to hear someone advocating it in apparent seriousness
- numpad0 3y ago
  Rewriting firmware in Rust is not a solution when your beautiful piece of Rust is to be micromanaged using an Android app as the controller.
  - kragen 3y ago
    i'm not sure if rust would help much for most bugs like this, though i guess it would have prevented the therac-25 deaths
- selimthegrim 3y ago
  I will say that my company did have an engineer that could write proper lock-free algorithms. Unfortunately, they were completely unable to retain him and he didn’t help matters.
rkangel 3y ago
There are: ISO 13485 and 62304. And different levels of design and testing for different classes of medical device. For example, I worked on a new ventilator at the start of the pandemic (https://www.cambridgeconsultants.com/press-releases/building-life-saving-ventilator-lightning-speed https://www.cambridgeconsultants.com/press-releases/building...). As a medical device that was "in the loop" of keeping the patient alive, some of the strictest restrictions applied. There were 2 MCUs - one doing the control work, and another monitoring that first one, checking it agreed with the decisions and sounding an alarm if not. These two processors had to be different manufacturers, and had to use separate development teams (all to reduce the chance of a common mode failure). The regulators are just checking you do the process though: does it look like you have done your design rigorously, and tested thoroughly. There's no way to check that you've done your design well though.
- seren 3y ago
  As someone working on Class III medical devices, it is true that the FDA does not check if your design is "safe", only that the documentation and process seems to have been followed, but on the other hand you are also required to monitor your install base and report any hazardous event, and in case no adequate remediation/plan to your adverse events, they have the real power to stop the commercialization of the product. But I agree this is coming a bit late in the process if people have already been harmed.
HumblyTossed 3y ago
The is the point where the Rust brigade swoops into the conversation.
- galangalalgol 3y ago
  I smell unoxidized metal! I love rust, use it almost exclusively. And ferrocene just certified it for industrial and automotive. And it doesn't sound to me like using rust or ada would have prevented this. It sounds like the android ui, which almost certainly is not in any certified toolchain, had a string handling error. The ui for a safety critical device is safety critical code, it sounds like that isn't how the rules actually work though, or you wouldn't be running it on a phone.
ajdude 3y ago
I had the exact same thoughts and I was about to bring up this very same point. Ada is still use heavily in medical devices and even has a decimal type.
moss2 3y ago
Why? Safety Critical Software engineers are expensive. If you write a webstack with Node.js and React to run the insulin pump you can hire web developers. They are much cheaper. And who would be able to tell the difference? Boomer lawyers and judges working on the lawsuit targeted against us won't understand.
- adrianN 3y ago
  Having worked on safety critical software, yes, lawyers understand very well.
- galangalalgol 3y ago
  Actually, critical software engineers often get less than half the TC of a full stack and still much less than a front end.
- rayiner 3y ago
  Judges have the benefit of experts spoon feeding them information, and no deadlines other than the ones they impose on themselves. The judge I worked for wrote the panel decision throwing out the Communications Decency Act, at age 64. In addition to covering packet routing and caching, it did a pretty good job of capturing the decentralized ethos of the Internet: https://archive.nytimes.com/www.nytimes.com/library/cyber/week/0612decision.html https://archive.nytimes.com/www.nytimes.com/library/cyber/we... > 11. No single entity -- academic, corporate, governmental, or non-profit -- administers the Internet. It exists and functions as a result of the fact that hundreds of thousands of separate operators of computers and computer networks independently decided to use common data transfer protocols to exchange communications and information with other computers (which in turn exchange communications and information with still other computers). There is no centralized storage location, control point, or communications channel for the Internet, and it would not be technically feasible for a single entity to control all of the information conveyed on the Internet. Ironically, in this day and age of Facebook and Twitter, that’s probably not even true anymore in a practical sense.
  - tptacek 3y ago
    That's the opposite of the conventional wisdom on the strengths of the judiciary, so much so that there's a norm that courts defer to legislatures on findings of fact, since they're so comparatively hamstrung at generating facts.
hennell 3y ago
I would have assumed that medical devices would have strict guidelines and heavy testing. A bit worrying if it's just suppliers discretion.

kragen 3y ago

there are, that's why insulin pump firmware is developed using obsolete and error-prone methods by substandard engineers and why you can't get the source for your insulin pump firmware and reflash it with a bug-fixed version

chpatrick 3y ago
I see where you're coming from, but I don't think letting anyone mess with the equipment keeping them alive is a good idea either. I really love open source IoT stuff for example and I think it's a great that people are trying to take real ownership of their devices. However it's different if the penalty for failure is your lights not working or immediate death. I think a better solution would be if the source was required to be public so people can identify bugs, but without giving users the ability to flash.
- noir_lord 3y ago
  > I think a better solution would be if the source was required to be public so people can identify bugs, but without giving users the ability to flash. That would be a solution, still has the problem of "how do you confirm the bug if you can't run the code". Finding those kinds of subtle bugs pretty much requires you be able to execute it.
- kragen 3y ago
  i do, because not 'letting anyone mess with the equipment keeping them alive' is, obviously, condemning some of them to deaths that they would avoid if they were allowed to i think you would need an overwhelmingly large number of prevented suicides and avoided foolish accidents to justify such a monstrous evil the balance of deaths is very likely to work out the other way around, in fact, because people reflashing their own insulin pumps have much stronger incentives to maximize their survival chances than any possible regulatory regime would, much less the one that actually exists the policy of 'let us make laws and set up regulatory agencies to prevent people from harming themselves' has such an astoundingly bad track record that it always surprises me to hear someone advocating it in apparent seriousness
numpad0 3y ago
Rewriting firmware in Rust is not a solution when your beautiful piece of Rust is to be micromanaged using an Android app as the controller.
- kragen 3y ago
  i'm not sure if rust would help much for most bugs like this, though i guess it would have prevented the therac-25 deaths
selimthegrim 3y ago
I will say that my company did have an engineer that could write proper lock-free algorithms. Unfortunately, they were completely unable to retain him and he didn’t help matters.

rkangel 3y ago

There are: ISO 13485 and 62304. And different levels of design and testing for different classes of medical device. For example, I worked on a new ventilator at the start of the pandemic (https://www.cambridgeconsultants.com/press-releases/building-life-saving-ventilator-lightning-speed https://www.cambridgeconsultants.com/press-releases/building...). As a medical device that was "in the loop" of keeping the patient alive, some of the strictest restrictions applied. There were 2 MCUs - one doing the control work, and another monitoring that first one, checking it agreed with the decisions and sounding an alarm if not. These two processors had to be different manufacturers, and had to use separate development teams (all to reduce the chance of a common mode failure). The regulators are just checking you do the process though: does it look like you have done your design rigorously, and tested thoroughly. There's no way to check that you've done your design well though.

seren 3y ago
As someone working on Class III medical devices, it is true that the FDA does not check if your design is "safe", only that the documentation and process seems to have been followed, but on the other hand you are also required to monitor your install base and report any hazardous event, and in case no adequate remediation/plan to your adverse events, they have the real power to stop the commercialization of the product. But I agree this is coming a bit late in the process if people have already been harmed.

HumblyTossed 3y ago

The is the point where the Rust brigade swoops into the conversation.

galangalalgol 3y ago
I smell unoxidized metal! I love rust, use it almost exclusively. And ferrocene just certified it for industrial and automotive. And it doesn't sound to me like using rust or ada would have prevented this. It sounds like the android ui, which almost certainly is not in any certified toolchain, had a string handling error. The ui for a safety critical device is safety critical code, it sounds like that isn't how the rules actually work though, or you wouldn't be running it on a phone.

ajdude 3y ago

I had the exact same thoughts and I was about to bring up this very same point. Ada is still use heavily in medical devices and even has a decimal type.

moss2 3y ago

Why? Safety Critical Software engineers are expensive. If you write a webstack with Node.js and React to run the insulin pump you can hire web developers. They are much cheaper. And who would be able to tell the difference? Boomer lawyers and judges working on the lawsuit targeted against us won't understand.

adrianN 3y ago
Having worked on safety critical software, yes, lawyers understand very well.
galangalalgol 3y ago
Actually, critical software engineers often get less than half the TC of a full stack and still much less than a front end.
rayiner 3y ago
Judges have the benefit of experts spoon feeding them information, and no deadlines other than the ones they impose on themselves. The judge I worked for wrote the panel decision throwing out the Communications Decency Act, at age 64. In addition to covering packet routing and caching, it did a pretty good job of capturing the decentralized ethos of the Internet: https://archive.nytimes.com/www.nytimes.com/library/cyber/week/0612decision.html https://archive.nytimes.com/www.nytimes.com/library/cyber/we... > 11. No single entity -- academic, corporate, governmental, or non-profit -- administers the Internet. It exists and functions as a result of the fact that hundreds of thousands of separate operators of computers and computer networks independently decided to use common data transfer protocols to exchange communications and information with other computers (which in turn exchange communications and information with still other computers). There is no centralized storage location, control point, or communications channel for the Internet, and it would not be technically feasible for a single entity to control all of the information conveyed on the Internet. Ironically, in this day and age of Facebook and Twitter, that’s probably not even true anymore in a practical sense.
- tptacek 3y ago
  That's the opposite of the conventional wisdom on the strengths of the judiciary, so much so that there's a norm that courts defer to legislatures on findings of fact, since they're so comparatively hamstrung at generating facts.

hennell 3y ago

I would have assumed that medical devices would have strict guidelines and heavy testing. A bit worrying if it's just suppliers discretion.