6 ms·

Actually, your pacemaker should be running code certified by unbiased, competent third party experts to be free of safety defects with the liability to match. T

by Veserv 3y ago

Actually, your pacemaker should be running code certified by unbiased, competent third party experts to be free of safety defects with the liability to match. That is what we demand of every other physical, electrical, chemical, and biological component of medical devices; seems only fair we hold the software to the same safety standard. Open source is orthogonal to quality at that level. It may be a desirable, additional quality to ensure openness and prevent obsolescence, but is no substitute for actual verification efforts by trained professionals specialized in the field actively tasked with analysis. All we have to do is look to critical software like OpenSSL and the Heartbleed vulnerability to see that “many eyes” is insufficient to achieve the necessary quality in even widely used, critical software. There is no guarantee of “any eyes”, and no guarantee they are competent or have sufficient domain knowledge (think of how long it takes to get up to speed on a new software project) to meaningfully assess the systems. We must have known, active, unbiased, competent third party verification for critical software just like we demand of all other components and fields. And yes, I am aware of the certification process for Class 3 medical device software, and no it is demonstrably insufficient for our current use cases as seen time and time again.

transpute 3y ago
> running code certified by unbiased, competent third party Running reproducible builds of open-source code certified by unbiased, competent third party experts, so that the semantics and runtime behavior of the code can be continuously evaluated by multiple unbiased, competent third party experts, since the business definition of "third party" can change over time. Where binary blobs are needed, the blobs can use reproducible builds and the certified source code can be escrowed so that the binary blobs can be reproducibly created by a third party.
- Veserv 3y ago
  Are you worried that a medical device manufacturer will intentionally replace certified software with known uncertified software? That would be shockingly stupid. There is exactly zero plausible deniability if they did that. That is “slam-dunk go directly to jail and get fined billions” stupid. There is exactly zero reason to worry about that at all at the device manufacturer level (component suppliers might lie which transitively affects the final product, but that should be handled when identifying the target of evaluation). If they want to be evil, there are plenty of ways that do not require them to put their evil in legally binding writing. If you are concerned about a hack or accidental deployment, that is a real concern, but image integrity, secure boot, and secure update are really part of the certified process/product.
  - transpute 3y ago
    > Are you worried that a medical device manufacturer will intentionally replace certified software with known uncertified software? Only that certified source code and software binaries can be continuously evaluated after production shipment, as original developers and certifiers move onto other projects. See the "Right to Repair" movement. Over timeframes of multiple years, we learn more about stacks of hardware, software and threats, as can be observed in the constant stream of CVEs for CPU microcode, firmware, operating systems and application code.
    - Veserv 3y ago
      That is just standard compulsory defect remediation. Like the hardware components they should be required to monitor and remediate any and all newly discovered defects in any components and the product. Again, this is something they already have to do for the physical components and which is done in literally every other industry to great effect. It is just not done for the software for some bone-headed reason. Note that failure to remediate known defects in components or covering it up is a very serious offense in basically every regulatory regime. That is basically “the thing” that gets you in trouble. Those standards should probably be made stricter, but that is for another time. Just getting software to “industry standard” is a major win.
      - transpute 3y ago
        > standards should probably be made stricter What do you think of the EU CRA requirements for critical software?
        Veserv 3y ago
        I only know about it broadly, but in terms of what I have read about it, it is almost reasonable except that it overly encumbers FOSS and non-commercial software. Liability should lay with the provider of the final product to use components fit for purpose, not with the component providers to make their component fit for every possible purpose. This is how it works on every other industry. Airplane manufacturers must use airplane-grade screws; they do not get to demand that every screw in Home Depot be fit for airplanes. If a company wants to use a FOSS library then they must make sure it fits their standards, or they must fund development and verification to make it meet their required standards. They do not get to unilaterally demand the library authors work harder to line their own pockets.
jevoten 3y ago
> Open source is orthogonal to quality at that level. "Quality" has many meanings. The article gives the example of Sandler, who was unable to read data from her pacemaker, because the correctly functioning, quality, high-assurance software was kept secret from her, and only the manufacturer could decipher it. Regardless, quality is a red herring - the real issue is sovereignty over our own bodies. Which is impossible if we have devices with secret functionality embedded in us. We have a right to know how they work, the same as we have a right to know what medicine doctors administer to us.
- Veserv 3y ago
  You do not receive a full chemical makeup, structure, and formulation of the medicine you are ingesting. You do not receive the entire production process of the medicine you are ingesting. You do not receive the full quality control process needed to verify the structure of the medicine you are ingesting. For that matter, even if you received those only a vanishingly small number of individuals would even be qualified to evaluate that information. No, what we as a society do is rely on a certification, verification, and validation process done by third party experts that guarantees the medicine fits a specification and is fit for purpose. This does not always get it right, but it is pretty good and works in literally every other industry. Competent unbiased third party verification is sufficient for safety-critical deployments. Openness is neither sufficient, nor necessary. A closed device with no safety defects is usable, a open device with safety defects is unacceptable. Openness may constitute other goods, such as freedom, agency, and such, but it does not provide safety, efficacy, and fitness for purpose which are the most important elements of medical devices (i.e it is orthogonal to effective operation).
  - transpute 3y ago
    > You do not receive a full chemical makeup, structure, and formulation of the medicine you are ingesting. Individual humans and their medical advisors choose what to ingest, from a marketplace of substances on a spectrum of transparency in content and consequences.
    - 3y ago
deleted 3y ago
[deleted]

by Veserv 3y ago

transpute 3y ago

> running code certified by unbiased, competent third party Running reproducible builds of open-source code certified by unbiased, competent third party experts, so that the semantics and runtime behavior of the code can be continuously evaluated by multiple unbiased, competent third party experts, since the business definition of "third party" can change over time. Where binary blobs are needed, the blobs can use reproducible builds and the certified source code can be escrowed so that the binary blobs can be reproducibly created by a third party.

Veserv 3y ago
Are you worried that a medical device manufacturer will intentionally replace certified software with known uncertified software? That would be shockingly stupid. There is exactly zero plausible deniability if they did that. That is “slam-dunk go directly to jail and get fined billions” stupid. There is exactly zero reason to worry about that at all at the device manufacturer level (component suppliers might lie which transitively affects the final product, but that should be handled when identifying the target of evaluation). If they want to be evil, there are plenty of ways that do not require them to put their evil in legally binding writing. If you are concerned about a hack or accidental deployment, that is a real concern, but image integrity, secure boot, and secure update are really part of the certified process/product.
- transpute 3y ago
  > Are you worried that a medical device manufacturer will intentionally replace certified software with known uncertified software? Only that certified source code and software binaries can be continuously evaluated after production shipment, as original developers and certifiers move onto other projects. See the "Right to Repair" movement. Over timeframes of multiple years, we learn more about stacks of hardware, software and threats, as can be observed in the constant stream of CVEs for CPU microcode, firmware, operating systems and application code.
  - Veserv 3y ago
    That is just standard compulsory defect remediation. Like the hardware components they should be required to monitor and remediate any and all newly discovered defects in any components and the product. Again, this is something they already have to do for the physical components and which is done in literally every other industry to great effect. It is just not done for the software for some bone-headed reason. Note that failure to remediate known defects in components or covering it up is a very serious offense in basically every regulatory regime. That is basically “the thing” that gets you in trouble. Those standards should probably be made stricter, but that is for another time. Just getting software to “industry standard” is a major win.
    - transpute 3y ago
      > standards should probably be made stricter What do you think of the EU CRA requirements for critical software?
      - Veserv 3y ago
        I only know about it broadly, but in terms of what I have read about it, it is almost reasonable except that it overly encumbers FOSS and non-commercial software. Liability should lay with the provider of the final product to use components fit for purpose, not with the component providers to make their component fit for every possible purpose. This is how it works on every other industry. Airplane manufacturers must use airplane-grade screws; they do not get to demand that every screw in Home Depot be fit for airplanes. If a company wants to use a FOSS library then they must make sure it fits their standards, or they must fund development and verification to make it meet their required standards. They do not get to unilaterally demand the library authors work harder to line their own pockets.

jevoten 3y ago

> Open source is orthogonal to quality at that level. "Quality" has many meanings. The article gives the example of Sandler, who was unable to read data from her pacemaker, because the correctly functioning, quality, high-assurance software was kept secret from her, and only the manufacturer could decipher it. Regardless, quality is a red herring - the real issue is sovereignty over our own bodies. Which is impossible if we have devices with secret functionality embedded in us. We have a right to know how they work, the same as we have a right to know what medicine doctors administer to us.

Veserv 3y ago
You do not receive a full chemical makeup, structure, and formulation of the medicine you are ingesting. You do not receive the entire production process of the medicine you are ingesting. You do not receive the full quality control process needed to verify the structure of the medicine you are ingesting. For that matter, even if you received those only a vanishingly small number of individuals would even be qualified to evaluate that information. No, what we as a society do is rely on a certification, verification, and validation process done by third party experts that guarantees the medicine fits a specification and is fit for purpose. This does not always get it right, but it is pretty good and works in literally every other industry. Competent unbiased third party verification is sufficient for safety-critical deployments. Openness is neither sufficient, nor necessary. A closed device with no safety defects is usable, a open device with safety defects is unacceptable. Openness may constitute other goods, such as freedom, agency, and such, but it does not provide safety, efficacy, and fitness for purpose which are the most important elements of medical devices (i.e it is orthogonal to effective operation).
- transpute 3y ago
  > You do not receive a full chemical makeup, structure, and formulation of the medicine you are ingesting. Individual humans and their medical advisors choose what to ingest, from a marketplace of substances on a spectrum of transparency in content and consequences.
  - 3y ago

deleted 3y ago

[deleted]