4 ms·

You most certainly don't want people to be able to modify safety critical code within a pacemaker. What most developers don't realise is the level of engineeri

You most certainly don't want people to be able to modify safety critical code within a pacemaker. What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with. What is a problem here, is that the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence. If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors, or simple things like that. Its the design of the system as a whole at fault, and that is already open. Medical devices such as these are not black boxes to the people that certify them, everything is open to them, source included. Having worked in that sort of area, I trust the systems that are in place.

adrianN 10y ago
Industrial control systems are also safety critical systems that have to adhere to very similar regulations as medical systems. Yet, when you ask hackers in that field you'll quickly learn that they have terrible code and abysmal security. Rules and regulations do very little to improve code quality and security, imho. Most things in these regulations are either best practices that any software engineer does (write tests, for example, or do code reviews), or just checkboxes for the QA to check. I'd argue that over time safety critical systems have worse code than normal software because refactoring is almost impossibly expensive due to all the paperwork that each software change ensues.
- flukus 10y ago
  >Rules and regulations do very little to improve code quality and security, imho. In fact, I'd say the restrict them. Adding a tonne of paperwork is a great way to prevent people from "noticing" problems.
  - TickleSteve 10y ago
    When the paperwork is describing the system design from high level down to low-level to sufficient detail that an external auditor can use it to examine the source then no, that is not true. The security/safety processes (as applicable) are examined at every level from system-design all the way down to implementation. Its not perfect, but its best practice we have for secure/safe product development.
    - flukus 10y ago
      If the paperwork was that sufficient then you could compile they paperwork.
      - TickleSteve 10y ago
        that was known as model-driven-development.... been thru that....
        flukus 10y ago
        Starts with a U and ends with an L? I'm so, so sorry.
- TickleSteve 10y ago
  industrial control != safety critical. "industrial control" covers both certified and non-certified code, you will have to be more specific than that for the purposes of this conversation. Quite often, cases such as the pacemaker are flaws in system design, not code. (e.g. no secure messaging, probably due to the lack of awareness when it was designed). That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code. True, the expense and overhead does indeed affect the level of change that is acceptable to a business, but given that change is allowed, the quality of that change is what we're discussing here and all I'm saying is that there is a lot more rigour involved than most developers here are aware of.
  - munificent 10y ago
    > That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code. I had assumed that as well until all of the horror stories around Toyota's firmware came to light. https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle_recalls https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...
    - TickleSteve 10y ago
      Unfortunately, we are human... Yes, the toyota case is a well publicised case. Consider though, the number of safety critical systems that are out there performing perfectly everyday. Of course, that is not proof of much, but the fact that you can name the Toyota case (and probably the Therac 25 case) means that the process generally works.
j-pb 10y ago
Software certification is a joke, and mostly consists of checking off a few boxes on workflow and development methods. Smart electricity meters are safety critical and I know students writing on it in their summer jobs. Airplanes are safety critical and they accidentally bridged the on plane wifi into the fly by wire system. Unless this stuff is written in COQ and formally verified, I won't trust it a bit.
- TickleSteve 10y ago
  Certification most certainly isn't a joke. It involves a hell of a lot of work, enough to double or triple the software portion of a project. Whatever certification you have been involved with may have been a joke. Safety-critical certification most certainly is not. BTW: I'm s/w architect on a set of smart meters. And they most certainly are not safety critical. (IMO They should be, but thats a separate issue). ...and no Wifi was "accidently" bridged. evidence?
  - throwanem 10y ago
    Not Wifi, but IIRC a guy called Chris Roberts was able to leverage a wireline connection to an underseat IFE controller into FADEC control on multiple aircraft. Last I heard of the case, he might be going to prison for it, so apparently at least some people find him credible. Similarly, witness recent revelations about car hackability - I forget if it was Blackhat or DEFCON where a couple of guys demoed a fully remote attack through the entertainment system's cell network access that culminated in the ability to override the steering, throttle, and brake controls. Maybe these are not what is meant by "safety critical" in your use of the phrase. If that's so, I think it would clarify matters greatly for you to define what you mean by it that's more specific than "can kill people if it goes wrong".
    - TickleSteve 10y ago
      A few points here: - Given physical access, all bets are off. - "safety critical" refers to software that has a SIL level(https://en.wikipedia.org/wiki/Safety_integrity_level https://en.wikipedia.org/wiki/Safety_integrity_level). Note the list of IEC/ANSI/EN standards. - Yes, the hack of the braking and steering system via the entertainment system is a breakdown of the certification system. That should have been found earlier. This is why critical systems are air-gapped. I personally have never worked on automotive systems so am not familiar with the regulations there, but I was/am extremely surprised this was possible.
mchaver 10y ago
Software Freedom Law Centers claims that the FDA requirements for medical devices put most of the responsibility of quality assurance on the manufacturer. It does not seem like the FDA provides much of an assurance of quality. Note: this is not my field of expertise, there is good chance I am very wrong, just following a short paper trail of laws, there maybe more laws (or practices) that I am missing that may indicate different FDA regulation controls for medical devices. https://www.softwarefreedom.org/resources/2010/transparent-medical-devices.pdf https://www.softwarefreedom.org/resources/2010/transparent-m... https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=820&showFR=1&subpartNode=21:8.0.1.1.12.2 https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFR...
michaelt 10y ago
rules and regulations [...] in a different league than what most developers are familiar with. [...] the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence. Imagine you were building a suspension bridge to the highest safety standards, and you had people with microscopes manually inspect every grain of sand and cement that went into the foundations. But the suspension cables were made of old washing lines. You would be achieving a very high and a very low standard of inspection at the same time. Some people would say standards of inspection are only as strong as their weakest link; that the whole claim of such inspection is to prove the absence of such weak links; and that a standard that has approved such a bridge has, by so doing, shown the claimed proof of safety is a useless joke. Other people would say the inspection is a checklist of common errors, rather than a proof of the safety of every aspect of the system; that proof of perfect safety was never its claim or goal; and that this merely shows there should be some extra boxes on the checklist, which was in any case a constantly evolving document.
- TickleSteve 10y ago
  Inspection in this case is a proof that the design you're certifying is fit-for-purpose. So in your example, it would show that the loading and stresses on your washing lines were sufficiently low to meet the safety margin of the bridge. Certification of safety-related and medical devices is not a check-box exercise, it looks at the dynamic and static behaviour of your system as a whole (not just the software).
  - michaelt 10y ago
    Perhaps I wasn't clear in my analogy. I mean the washing lines weren't fit for purpose (just like the insecure comms wasn't fit for purpose) but the inspection didn't identify them as unfit for purpose (just like the inspection didn't identify the insecure comms as unfit for purpose) Given that something unfit for purpose was certified, certification doesn't prove something is fit for purpose. Of course, certification may indicate something is probably more fit for purpose than something that failed the same certification or didn't attempt it.
ef4 10y ago
> everything is open to them, source included Certifiers typically don't review source code. They are operating at least two levels of abstraction away: they review documents that describe procedures that are supposed to ensure quality. > If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors That is a big fat citation needed. Researchers working without access to source code have demonstrated multiple vulnerabilities in safety-critical medical devices. We have no way of knowing how many more they would find if they also had access to the source.
CaptSpify 10y ago
I used to Reverse-Engineer medical equipment (MRIs, CTs, etc). The lack of engineering strictness in theses systems was amazing to me. They are horribly insecure, BY DESIGN My experience is that, no, this is not an acceptable solution. I have no reason to believe that implanted medical devices will be any better.

adrianN 10y ago

Industrial control systems are also safety critical systems that have to adhere to very similar regulations as medical systems. Yet, when you ask hackers in that field you'll quickly learn that they have terrible code and abysmal security. Rules and regulations do very little to improve code quality and security, imho. Most things in these regulations are either best practices that any software engineer does (write tests, for example, or do code reviews), or just checkboxes for the QA to check. I'd argue that over time safety critical systems have worse code than normal software because refactoring is almost impossibly expensive due to all the paperwork that each software change ensues.

flukus 10y ago
>Rules and regulations do very little to improve code quality and security, imho. In fact, I'd say the restrict them. Adding a tonne of paperwork is a great way to prevent people from "noticing" problems.
- TickleSteve 10y ago
  When the paperwork is describing the system design from high level down to low-level to sufficient detail that an external auditor can use it to examine the source then no, that is not true. The security/safety processes (as applicable) are examined at every level from system-design all the way down to implementation. Its not perfect, but its best practice we have for secure/safe product development.
  - flukus 10y ago
    If the paperwork was that sufficient then you could compile they paperwork.
    - TickleSteve 10y ago
      that was known as model-driven-development.... been thru that....
      - flukus 10y ago
        Starts with a U and ends with an L? I'm so, so sorry.
TickleSteve 10y ago
industrial control != safety critical. "industrial control" covers both certified and non-certified code, you will have to be more specific than that for the purposes of this conversation. Quite often, cases such as the pacemaker are flaws in system design, not code. (e.g. no secure messaging, probably due to the lack of awareness when it was designed). That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code. True, the expense and overhead does indeed affect the level of change that is acceptable to a business, but given that change is allowed, the quality of that change is what we're discussing here and all I'm saying is that there is a lot more rigour involved than most developers here are aware of.
- munificent 10y ago
  > That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code. I had assumed that as well until all of the horror stories around Toyota's firmware came to light. https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle_recalls https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...
  - TickleSteve 10y ago
    Unfortunately, we are human... Yes, the toyota case is a well publicised case. Consider though, the number of safety critical systems that are out there performing perfectly everyday. Of course, that is not proof of much, but the fact that you can name the Toyota case (and probably the Therac 25 case) means that the process generally works.

j-pb 10y ago

Software certification is a joke, and mostly consists of checking off a few boxes on workflow and development methods. Smart electricity meters are safety critical and I know students writing on it in their summer jobs. Airplanes are safety critical and they accidentally bridged the on plane wifi into the fly by wire system. Unless this stuff is written in COQ and formally verified, I won't trust it a bit.

TickleSteve 10y ago
Certification most certainly isn't a joke. It involves a hell of a lot of work, enough to double or triple the software portion of a project. Whatever certification you have been involved with may have been a joke. Safety-critical certification most certainly is not. BTW: I'm s/w architect on a set of smart meters. And they most certainly are not safety critical. (IMO They should be, but thats a separate issue). ...and no Wifi was "accidently" bridged. evidence?
- throwanem 10y ago
  Not Wifi, but IIRC a guy called Chris Roberts was able to leverage a wireline connection to an underseat IFE controller into FADEC control on multiple aircraft. Last I heard of the case, he might be going to prison for it, so apparently at least some people find him credible. Similarly, witness recent revelations about car hackability - I forget if it was Blackhat or DEFCON where a couple of guys demoed a fully remote attack through the entertainment system's cell network access that culminated in the ability to override the steering, throttle, and brake controls. Maybe these are not what is meant by "safety critical" in your use of the phrase. If that's so, I think it would clarify matters greatly for you to define what you mean by it that's more specific than "can kill people if it goes wrong".
  - TickleSteve 10y ago
    A few points here: - Given physical access, all bets are off. - "safety critical" refers to software that has a SIL level(https://en.wikipedia.org/wiki/Safety_integrity_level https://en.wikipedia.org/wiki/Safety_integrity_level). Note the list of IEC/ANSI/EN standards. - Yes, the hack of the braking and steering system via the entertainment system is a breakdown of the certification system. That should have been found earlier. This is why critical systems are air-gapped. I personally have never worked on automotive systems so am not familiar with the regulations there, but I was/am extremely surprised this was possible.

mchaver 10y ago

Software Freedom Law Centers claims that the FDA requirements for medical devices put most of the responsibility of quality assurance on the manufacturer. It does not seem like the FDA provides much of an assurance of quality. Note: this is not my field of expertise, there is good chance I am very wrong, just following a short paper trail of laws, there maybe more laws (or practices) that I am missing that may indicate different FDA regulation controls for medical devices. https://www.softwarefreedom.org/resources/2010/transparent-medical-devices.pdf https://www.softwarefreedom.org/resources/2010/transparent-m... https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=820&showFR=1&subpartNode=21:8.0.1.1.12.2 https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFR...

michaelt 10y ago

rules and regulations [...] in a different league than what most developers are familiar with. [...] the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence. Imagine you were building a suspension bridge to the highest safety standards, and you had people with microscopes manually inspect every grain of sand and cement that went into the foundations. But the suspension cables were made of old washing lines. You would be achieving a very high and a very low standard of inspection at the same time. Some people would say standards of inspection are only as strong as their weakest link; that the whole claim of such inspection is to prove the absence of such weak links; and that a standard that has approved such a bridge has, by so doing, shown the claimed proof of safety is a useless joke. Other people would say the inspection is a checklist of common errors, rather than a proof of the safety of every aspect of the system; that proof of perfect safety was never its claim or goal; and that this merely shows there should be some extra boxes on the checklist, which was in any case a constantly evolving document.

TickleSteve 10y ago
Inspection in this case is a proof that the design you're certifying is fit-for-purpose. So in your example, it would show that the loading and stresses on your washing lines were sufficiently low to meet the safety margin of the bridge. Certification of safety-related and medical devices is not a check-box exercise, it looks at the dynamic and static behaviour of your system as a whole (not just the software).
- michaelt 10y ago
  Perhaps I wasn't clear in my analogy. I mean the washing lines weren't fit for purpose (just like the insecure comms wasn't fit for purpose) but the inspection didn't identify them as unfit for purpose (just like the inspection didn't identify the insecure comms as unfit for purpose) Given that something unfit for purpose was certified, certification doesn't prove something is fit for purpose. Of course, certification may indicate something is probably more fit for purpose than something that failed the same certification or didn't attempt it.

ef4 10y ago

> everything is open to them, source included Certifiers typically don't review source code. They are operating at least two levels of abstraction away: they review documents that describe procedures that are supposed to ensure quality. > If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors That is a big fat citation needed. Researchers working without access to source code have demonstrated multiple vulnerabilities in safety-critical medical devices. We have no way of knowing how many more they would find if they also had access to the source.

CaptSpify 10y ago

I used to Reverse-Engineer medical equipment (MRIs, CTs, etc). The lack of engineering strictness in theses systems was amazing to me. They are horribly insecure, BY DESIGN My experience is that, no, this is not an acceptable solution. I have no reason to believe that implanted medical devices will be any better.