3 ms·

> How else would they do it than incremental? How about... not? I'm not being disingenious here. If there's autonomus tech, then I find it scary that it migh

> How else would they do it than incremental? How about... not? I'm not being disingenious here. If there's autonomus tech, then I find it scary that it might change its behaviour. You see, I've learned how (say) Autopilot responds in certain situations. I know when to trust it, and when not to. If this behaviour changes (perhaps even if it changes for the better!), that's a very dangerous thing. Some of my experience is now invalidated, but I don't know which part. But it mostly works as before, which gives me a false sense of security. Changing the driving behaviour of a vehicle, especially in totally un-obvious ways, is super dangerous, and such changes must have very strong justifications. Just think of that poor guy whose Tesla drove into the divider. Oh look, it behaves just like before -- except for its lane following behaviour.

Gustomaximus 8y ago
I dont understand how you would expect not to do incremental improvement. Build it perfectly the first time? That's not realistic. Pretty much everything humans do is incremental improvement. I understand your point that we dont want to introduce weaknesses in 'upgrades' but this should be a discussion about how to create robust QA rather that stopping incremental improvement.
- InternetOfStuff 8y ago
  > I dont understand how you would expect not to do incremental improvement I wonder if we're talking past each other? Sure, incrementally improve during engineering. But don't incrementally change the safety-critical behaviour of a car once it's in the customer's hands. > Build it perfectly the first time? That's not realistic. Absolutely agreed. But for subjects such as autopilots, predictability may very well trump improvement. I'm assuming that the initial iteration is already reasonably safe and useful (if it isn't it shouldn't have been shipped, right?). If the behaviour changes unpredictably (how will you predict autopilot changes caused by an update?), this may well be less safe than keeping the existing system, whoose quirks the dirver has noww learned. > Pretty much everything humans do is incremental improvement. > I understand your point that we dont want to introduce weaknesses in 'upgrades' The nasty thing is that even a change to objectively superior behaviour may be problematic, because the driver will expect the car to behave one way, when it now behaves a different way (regardless of merit). Not even a changelog would help in such cases. I feel like the autopilot ought to inform the driver in each situation where it used to decide one way, and now decides another way. That might be a reasonably safe way of allowing incremental change, but is likely too annoying for most people. > but this should be a discussion about how to create robust QA rather that stopping incremental improvement. Robust QA is certainly a must. (side note: I used to work in safety-critical automotive projects. I may have a more intimate understanding of the issues)

Gustomaximus 8y ago

I dont understand how you would expect not to do incremental improvement. Build it perfectly the first time? That's not realistic. Pretty much everything humans do is incremental improvement. I understand your point that we dont want to introduce weaknesses in 'upgrades' but this should be a discussion about how to create robust QA rather that stopping incremental improvement.

InternetOfStuff 8y ago
> I dont understand how you would expect not to do incremental improvement I wonder if we're talking past each other? Sure, incrementally improve during engineering. But don't incrementally change the safety-critical behaviour of a car once it's in the customer's hands. > Build it perfectly the first time? That's not realistic. Absolutely agreed. But for subjects such as autopilots, predictability may very well trump improvement. I'm assuming that the initial iteration is already reasonably safe and useful (if it isn't it shouldn't have been shipped, right?). If the behaviour changes unpredictably (how will you predict autopilot changes caused by an update?), this may well be less safe than keeping the existing system, whoose quirks the dirver has noww learned. > Pretty much everything humans do is incremental improvement. > I understand your point that we dont want to introduce weaknesses in 'upgrades' The nasty thing is that even a change to objectively superior behaviour may be problematic, because the driver will expect the car to behave one way, when it now behaves a different way (regardless of merit). Not even a changelog would help in such cases. I feel like the autopilot ought to inform the driver in each situation where it used to decide one way, and now decides another way. That might be a reasonably safe way of allowing incremental change, but is likely too annoying for most people. > but this should be a discussion about how to create robust QA rather that stopping incremental improvement. Robust QA is certainly a must. (side note: I used to work in safety-critical automotive projects. I may have a more intimate understanding of the issues)