9 ms·

What justification do you have for asserting that the safety critical systems are actually separated/isolated? The people at Jeep probably thought their systems

by Veserv 6y ago

What justification do you have for asserting that the safety critical systems are actually separated/isolated? The people at Jeep probably thought their systems were secure, yet the researchers on the Jeep Cherokee attack demonstrated they were not and claim that they could have simultaneously affected 471,000(!) vehicles [1]. Do you think anybody at a car company would dare to claim that their systems are safe enough to bet the lives of 471,000 people on them and take responsibility if they were wrong? Do you think any engineer would, in good conscience, support such a statement and share that responsibility if asked directly? I doubt you could find a single engineer that would feel even remotely comfortable that their procedures are good enough for 1/100th that number and you would never find an executive who would dare claim such a thing in a legally binding way. Absolutely zero benefit of the doubt should be given to systems where a single error could cause grievous harm to tens of thousands of lives. Such systems should require an absolutely ironclad public legally-binding positive assertion of security with strict criminal liability for failure (blame can not be shifted) that is independently validated before we should even think of accepting their use. We already accept nothing less for systems that are less dangerous such as nuclear reactor meltdowns and systems that are equally dangerous such as nuclear weapons systems. If systems with failure modes many times worse than the atomic bombings of Japan can not be made safe, then they must not be made. Anything less would be so criminally irresponsible that we lack a word for the magnitude of irresponsibility. [1] https://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/ https://www.wired.com/2015/07/hackers-remotely-kill-jeep-hig...

AlotOfReading 6y ago
I'd guess the fact that GM hired those guys for Cruise [1] probably suggests they took it seriously and made sure their own cars were isolated. [1] https://www.detroitnews.com/story/business/autos/mobility/2017/07/31/gm-hires-jeep-hackers-cruise-automation/104158212/ https://www.detroitnews.com/story/business/autos/mobility/20...
- Veserv 6y ago
  This is exactly the sort of justification that I am cautioning against. Would it be great if they did take the problem seriously and solve it? Of course. But it is not our duty to justify their decisions when the consequences of their decisions may cause grievous harm to thousands. It is their job, the company's, to justify to the satisfaction of engineers and the people that we should trust the lives of thousands to millions on their systems. If anything, it is our job to be skeptical of their claims and grill them ruthlessly before they should even be allowed to consider deploying such systems. I mean, think of any other software system, would you trust anybody's lives to any of them? Would the engineers on those projects agree? Most software engineers I know would be horrified if their systems were used in something responsible for even a single life let alone thousands. The claim that we should trust these companies with hundreds of thousands of lives is an extraordinary claim that they should be required to provide extraordinary evidence for. And, even if we are convinced, they should still be ultimately liable for their choice of decision to make sure they are actually putting their money where their mouth is. It is what we demand from bridge builders, it is what we demand from nuclear reactor designs, it should be what we demand from mass-produced internet-connected safety-critical devices.
  - Smoosh 6y ago
    Oh boy, are you gonna be mad when you find out that Tesla are allowing the general public to beta test their self driving software on public roads!
DoingIsLearning 6y ago
> What justification do you have for asserting that the safety critical systems are actually separated/isolated? I don't need to justify anything, I am stating they are isolated in all the vehicles I am familiar with. [0] The example from the Jeep Cherokee is a pretty crude architecture defect, the only way that could happen is in vehicles with a single CAN bus, which really is something out of the 1990's. As constructive criticism you really ought to read the "In Comments" section of the guidelines at the bottom of the frontpage. [0] https://www.st.com/en/applications/body-and-convenience/automotive-gateway.html#overview https://www.st.com/en/applications/body-and-convenience/auto...
- Veserv 6y ago
  You state in a different comment chain that bus segregation does not mean no communication, so I do not understand how that is consistent with your claim that they are isolated. The Jeep Cherokee also did not have only a single CAN bus as evidenced by the diagram on page 8 [1]. The problem was that the radio unit was on both CAN buses. I ask for justification of your statement because you counter-argued the previous comment that stated: “It really should be illegal to have network connected cars. Any software security engineer knows that.” by stating that the safety critical systems are segregated and thus “the network bus in which your 'compromised' infotainment system is able to operate is completely separate from Engine, ABS, AEB, ESP, Airbags etc.” which implies that solution achieves the desired security properties. I claim the desired security properties are: “safe enough to bet the lives of 471,000 people on them and take responsibility if they were wrong”. I think, and I think most people would agree, that is accurate assessment of the potential consequences of catastrophic failure. That is a consequence comparable to the detonation of a nuclear warhead in a city, so we should apply similar standards to the required level of security. Hardly any software engineer would make a claim that their software can be trusted with an amount of lives within 3 orders of magnitude of that number (note 3 orders of magnitude is 471 which is comparable to an airplane crash, so 3 orders of magnitude less is critical avionics level). Therefore, making a claim that implies that level of security has already been achieved is an extraordinary claim and should thus require equally extraordinary evidence to be believed. Even with such justification I would still caution anybody else reading it since it would not be a legally binding claim by a liable entity, so even assuming you are commenting honestly and to the best of your knowledge (which I assume you are), and even correctly describing the truth as it is today, there are no consequences for any liable entity if they betray your trust. It should always be up to the liable entity to justify themselves to the people to our satisfaction before deploying systems with such societal-level consequences. [1] http://illmatics.com/Remote%20Car%20Hacking.pdf http://illmatics.com/Remote%20Car%20Hacking.pdf

by Veserv 6y ago

AlotOfReading 6y ago

I'd guess the fact that GM hired those guys for Cruise [1] probably suggests they took it seriously and made sure their own cars were isolated. [1] https://www.detroitnews.com/story/business/autos/mobility/2017/07/31/gm-hires-jeep-hackers-cruise-automation/104158212/ https://www.detroitnews.com/story/business/autos/mobility/20...

Veserv 6y ago
This is exactly the sort of justification that I am cautioning against. Would it be great if they did take the problem seriously and solve it? Of course. But it is not our duty to justify their decisions when the consequences of their decisions may cause grievous harm to thousands. It is their job, the company's, to justify to the satisfaction of engineers and the people that we should trust the lives of thousands to millions on their systems. If anything, it is our job to be skeptical of their claims and grill them ruthlessly before they should even be allowed to consider deploying such systems. I mean, think of any other software system, would you trust anybody's lives to any of them? Would the engineers on those projects agree? Most software engineers I know would be horrified if their systems were used in something responsible for even a single life let alone thousands. The claim that we should trust these companies with hundreds of thousands of lives is an extraordinary claim that they should be required to provide extraordinary evidence for. And, even if we are convinced, they should still be ultimately liable for their choice of decision to make sure they are actually putting their money where their mouth is. It is what we demand from bridge builders, it is what we demand from nuclear reactor designs, it should be what we demand from mass-produced internet-connected safety-critical devices.
- Smoosh 6y ago
  Oh boy, are you gonna be mad when you find out that Tesla are allowing the general public to beta test their self driving software on public roads!

DoingIsLearning 6y ago

> What justification do you have for asserting that the safety critical systems are actually separated/isolated? I don't need to justify anything, I am stating they are isolated in all the vehicles I am familiar with. [0] The example from the Jeep Cherokee is a pretty crude architecture defect, the only way that could happen is in vehicles with a single CAN bus, which really is something out of the 1990's. As constructive criticism you really ought to read the "In Comments" section of the guidelines at the bottom of the frontpage. [0] https://www.st.com/en/applications/body-and-convenience/automotive-gateway.html#overview https://www.st.com/en/applications/body-and-convenience/auto...

Veserv 6y ago
You state in a different comment chain that bus segregation does not mean no communication, so I do not understand how that is consistent with your claim that they are isolated. The Jeep Cherokee also did not have only a single CAN bus as evidenced by the diagram on page 8 [1]. The problem was that the radio unit was on both CAN buses. I ask for justification of your statement because you counter-argued the previous comment that stated: “It really should be illegal to have network connected cars. Any software security engineer knows that.” by stating that the safety critical systems are segregated and thus “the network bus in which your 'compromised' infotainment system is able to operate is completely separate from Engine, ABS, AEB, ESP, Airbags etc.” which implies that solution achieves the desired security properties. I claim the desired security properties are: “safe enough to bet the lives of 471,000 people on them and take responsibility if they were wrong”. I think, and I think most people would agree, that is accurate assessment of the potential consequences of catastrophic failure. That is a consequence comparable to the detonation of a nuclear warhead in a city, so we should apply similar standards to the required level of security. Hardly any software engineer would make a claim that their software can be trusted with an amount of lives within 3 orders of magnitude of that number (note 3 orders of magnitude is 471 which is comparable to an airplane crash, so 3 orders of magnitude less is critical avionics level). Therefore, making a claim that implies that level of security has already been achieved is an extraordinary claim and should thus require equally extraordinary evidence to be believed. Even with such justification I would still caution anybody else reading it since it would not be a legally binding claim by a liable entity, so even assuming you are commenting honestly and to the best of your knowledge (which I assume you are), and even correctly describing the truth as it is today, there are no consequences for any liable entity if they betray your trust. It should always be up to the liable entity to justify themselves to the people to our satisfaction before deploying systems with such societal-level consequences. [1] http://illmatics.com/Remote%20Car%20Hacking.pdf http://illmatics.com/Remote%20Car%20Hacking.pdf