3 ms·

So I'd be really interested in knowing how a failure like this happens in a human-safety-critical application. I assume every software engineer under the age o

So I'd be really interested in knowing how a failure like this happens in a human-safety-critical application. I assume every software engineer under the age of ~50 would have learned about events like the Therac-25 incidents during their education. Is there genuinely no robust software quality regime in place for critical systems like this in vehicles? There are so many worrying things here: such a bad failure occurring in the first place; the lack of any physical fallback or failsafe; the ability of the dealer to remotely control the car's operation (!!) – any of these by themselves fills me with dread.

hinkley 9y ago
There have been a lot of revolutions in the practice of software development but by and large, self reflection has evaded us. One of the leads and myself are the only people who call out the fact that we are humans and we will make mistakes. And on a 24/7 operation that means production issues will happen. The other lead insists we all act like robots and he gets frustrated when people don’t like his proposals.
oconnore 9y ago
Reliability is expensive and slow. Companies hate expensive and slow, especially when you have a demo that basically does what you want sans edge cases.
quadcore 9y ago
Oh actually, let say you compute a+b with a computer. Believe it or not, you aren't 100% sure the computer is gonna gives you the right answer. There is 3 types of failures that could happen, 1) a hardware failure, 2) a software failure, 3) a bit could be flipped by a cosmic particle.
- matthewmacleod 9y ago
  Right, but surely every engineer working on safety-critical systems would be aware of that, right? There are mitigations available, that I guess I naïvely assumed would be in place for a system that could end up killing dozens of people if it went wrong.
  - quadcore 9y ago
    Agreed.
  - Piskvorrr 9y ago
    Oh? https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle_recalls#Anti-lock_brake_software_recall https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...
jandrese 9y ago
Or the owner attempted to flash his own ECU and messed it up. Granted, this is more common in the rice-burning crowd than the Benz driving crowd, but there is still some overlap. Edit: Just saw that the driver was an "amateur car racer", which exponentially increases the chances that he re-flashed his ECU.
- matthewmacleod 9y ago
  That's interesting, in the sense that it might explain a software failure, but it doesn't really explain why this wasn't failsafe anyway (given the possibility of a non-flashed ECU failing) nor why remote access was possible. I guess something doesn't add up here.
  - jandrese 9y ago
    It's possibly Benz went to the trouble of doing a rigorous analysis of their ECU to verify that there were no faulting conditions that would impact safety and thus decided to use it as the backstop. The safety guarantee from the software could be compromised by an unauthorized modification. Personally I don't like trusting safety systems entirely to software, but I could see it being a decision a big corporation makes to simplify manufacturing (cut costs).
taneq 9y ago
You say that and yet people are letting self-driving software on Android phones control their vehicles. I don't get it either.
Humdeee 9y ago
> the ability of the dealer to remotely control the car's operation (!!) This could be misleading. From the small article at least, the only action I assume that was done was the cruise control being disengaged from it's faulty state. There is no evidence that a rogue service employee could play real life mario kart from afar (although it wouldn't surprise me). I also somewhat expected this from a high end car manufacturer in this day and age. Why is this inherently a bad thing? Surely it would do more good than bad, would it not? What bothers me is the poor response time. inb4 privacy concerns, big brother, etc. I'm thinking engine shutdowns during car chases (expensive vehicles are a target), etc.
- matthewmacleod 9y ago
  It's quite possible that there's no ability to actually drive the vehicle as such, but remote access is inherently a massive security hole regardless. Remember the remote Jeep hack from a couple of years ago, where the vehicle could be remotely controlled over DAB? (https://www.wired.com/2015/07/jeep-hack-chrysler-recalls-1-4m-vehicles-bug-fix/ https://www.wired.com/2015/07/jeep-hack-chrysler-recalls-1-4...)

hinkley 9y ago

There have been a lot of revolutions in the practice of software development but by and large, self reflection has evaded us. One of the leads and myself are the only people who call out the fact that we are humans and we will make mistakes. And on a 24/7 operation that means production issues will happen. The other lead insists we all act like robots and he gets frustrated when people don’t like his proposals.

oconnore 9y ago

Reliability is expensive and slow. Companies hate expensive and slow, especially when you have a demo that basically does what you want sans edge cases.

quadcore 9y ago

Oh actually, let say you compute a+b with a computer. Believe it or not, you aren't 100% sure the computer is gonna gives you the right answer. There is 3 types of failures that could happen, 1) a hardware failure, 2) a software failure, 3) a bit could be flipped by a cosmic particle.

matthewmacleod 9y ago
Right, but surely every engineer working on safety-critical systems would be aware of that, right? There are mitigations available, that I guess I naïvely assumed would be in place for a system that could end up killing dozens of people if it went wrong.
- quadcore 9y ago
  Agreed.
- Piskvorrr 9y ago
  Oh? https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle_recalls#Anti-lock_brake_software_recall https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...

jandrese 9y ago

Or the owner attempted to flash his own ECU and messed it up. Granted, this is more common in the rice-burning crowd than the Benz driving crowd, but there is still some overlap. Edit: Just saw that the driver was an "amateur car racer", which exponentially increases the chances that he re-flashed his ECU.

matthewmacleod 9y ago
That's interesting, in the sense that it might explain a software failure, but it doesn't really explain why this wasn't failsafe anyway (given the possibility of a non-flashed ECU failing) nor why remote access was possible. I guess something doesn't add up here.
- jandrese 9y ago
  It's possibly Benz went to the trouble of doing a rigorous analysis of their ECU to verify that there were no faulting conditions that would impact safety and thus decided to use it as the backstop. The safety guarantee from the software could be compromised by an unauthorized modification. Personally I don't like trusting safety systems entirely to software, but I could see it being a decision a big corporation makes to simplify manufacturing (cut costs).

taneq 9y ago

You say that and yet people are letting self-driving software on Android phones control their vehicles. I don't get it either.

Humdeee 9y ago

> the ability of the dealer to remotely control the car's operation (!!) This could be misleading. From the small article at least, the only action I assume that was done was the cruise control being disengaged from it's faulty state. There is no evidence that a rogue service employee could play real life mario kart from afar (although it wouldn't surprise me). I also somewhat expected this from a high end car manufacturer in this day and age. Why is this inherently a bad thing? Surely it would do more good than bad, would it not? What bothers me is the poor response time. inb4 privacy concerns, big brother, etc. I'm thinking engine shutdowns during car chases (expensive vehicles are a target), etc.

matthewmacleod 9y ago
It's quite possible that there's no ability to actually drive the vehicle as such, but remote access is inherently a massive security hole regardless. Remember the remote Jeep hack from a couple of years ago, where the vehicle could be remotely controlled over DAB? (https://www.wired.com/2015/07/jeep-hack-chrysler-recalls-1-4m-vehicles-bug-fix/ https://www.wired.com/2015/07/jeep-hack-chrysler-recalls-1-4...)