11 ms·

A common phrase in aircraft cockpits nowadays is "What the heck is it doing now?" as pilots have migrated from actually flying the plane to simply being glorifi

A common phrase in aircraft cockpits nowadays is "What the heck is it doing now?" as pilots have migrated from actually flying the plane to simply being glorified systems managers. While planes have become so, so, so much safer because of all this automation, pilots uncertainty regarding autopilot functioning is a major concern nowadays, and the reason for several accidents. There are very interesting HCI challenges around properly communicating to the pilot/driver "what the heck it is doing" and clearly communicating just how much control the human has or doesn't have at any given point. This "announcement" certainly doesn't inspire any confidence that they have really thought this through deeply enough (I think they probably have, but it should be communicated like it). As a huge Tesla fan, I can't help but feel like I need to hold my breath now and make sure something terrible doesn't happen because of this, and it ends up leading to even more regs setting us back on the path to full car automation.

S_A_P 11y ago
Queue the James Bond movie sub plot where an evil villain takes control of his tesla/whatever autonomous car and he has to use a combination of hacker shell commands and sheer brute strength to save both him self and bond girl he was seducing while the car chauffeured him back to his luxury hotel suite...
- mschuster91 11y ago
  I recommend the Scorpions episode S01E20 "Crossroads", where a similar situation unfolds at the end.
- gregpilling 11y ago
  https://www.youtube.com/watch?v=qKAME9fAA-4 https://www.youtube.com/watch?v=qKAME9fAA-4 If the bad guys had only hacked Bond in this scene, it could have been much different!
- tacojuan 11y ago
  In the newer Ghost in the Shell series, an antagonist writes a virus that takes control of all the cars in the city, causing a massive physical/real-world denial of service.
Nadya 11y ago
>While planes have become so, so, so much safer because of all this automation, pilots uncertainty regarding autopilot functioning is a major concern nowadays, and the reason for several accidents. If % of accidents caused by pilot interference on a working system > % of accidents caused by system malfunctioning and pilot ignoring it: people will still be against not allowing pilots to interfere. Even when it causes more accidents... There's something about humans trusting humans more than machines that I don't fully understand. Systems can make mistakes but the amount of human mistakes is often exponentially greater to a degree of absurdity that humans are even trusted at all and yet people will side with the human over the machine. Humans will always want human oversight - even when that oversight does more harm than good once automation reaches a certain threshold... Special note: I'm not aware of avionics and the data on pilot interference w/ the system vs failure of pilot to intervene. So maybe this example doesn't hold very well for avionics...
- tracker1 11y ago
  A human can make judgement calls in unexpected situations. Objectively, a human is more likely to see falling debris and react to it than a self-driving car (of course depending on the programming and sensors). Or when the road is slick and you are doing 5 under, and you see someone pass you doing 5 over, with a car following too closely to it... you have a better understanding that there's likely to be an accident ahead... (I've seen this happen several times). A computer wouldn't predict that most likely. Unless ALL cars are automated, it is a must that a driver be able to take over quickly.
  - baudehlo 11y ago
    I still am yet to understand how a self driving car deals with a cop directing traffic. Or even a construction worker holding traffic back from a backing up backhoe. But maybe I am not aware of the genius of the technology yet.
    - TeMPOraL 11y ago
      It's called image processing.
JulianMorrison 11y ago
There's an old joke about how the new planes have a pilot, for the sake of safety, and a dog, to bite the pilot if he tries to touch the controls. It sounds like it's time to introduce the dog?
- sandworm101 11y ago
  There is more to being a pilot than the stick-and-rudder of flying the plane. Judgment calls such as whether to attempt a landing under less-than-ideal conditions are far more important.
- detaro 11y ago
  More like a dog to gently nudge the pilot if he is relying on a system that isn't working. Most issues I've read about happened when the systems either weren't engaged properly or automatically disengaged because their operating parameters weren't met anymore, suddenly forcing the pilots to take control. Example Air France 447, where (presumably) the airspeed sensors where blocked by ice, which lead to the autopilot to disengage. There is also the theory that the pilots then made some of their mistakes based on the belief that the avionics would stop them from bringing the plane in an unsafe state and didn't realize that the system wasn't able to do so with missing information. (I hope I'm presenting this correctly, but that's what I remember reading afterwards) There is also the training aspect: if a system takes care of something 99,9% of the time, the pilot is less experienced in the 0,1% where it doesn't. There is a reason the safest airline pilots often fly way smaller aircraft as a hobby and get some instinct for manual flying there.
  - ekianjo 11y ago
    No need for theories we know very well what happened there in that flight and its basically the copilots who crashed a perfectly fly worthy planed.
    - TuxerLulz 11y ago
      There is no basically in an aircraft accident. If you wish to make such a statement, please explain what actually happened in this flight (icing of the pitot tubes, deactivation of normal flight control law for the fly-by-wire system, flight at night over the ocean and over a storm, stall under the specifications for the stall warning system). It's respectful for the crew.
morb 11y ago
99% invisible did a pretty good episode on that: http://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ http://99percentinvisible.org/episode/children-of-the-magent...
- ChicagoBoy11 11y ago
  Yes! - <3 Roman Mars' voice.
  - prawn 11y ago
    I like 99%I's content but Mars' voice is my least favourite part of it. Sounds a bit smug and is not always clearly audible in the car. Much prefer Freakonomics.
Sven7 11y ago
The "what the heck is it doing" problem may not be a problem with inexperienced pilots.
- bdamm 11y ago
  As an inexperienced pilot who has quite recently had the "what the heck is it doing" experience with my (40 yr old, analog computer) autopilot system, I can assure you that it is.
  - Sven7 11y ago
    Maybe it should be no-experience pilots then :) But joking aside thats where these systems will eventually take us.
rzzzt 11y ago
There are videos on YouTube of Bosch fitting their automated driving technology into a Model S [1] and performing demonstration runs on a private road. The software shown in the video displays quite a lot of information about what the sensors consider relevant from the car's surroundings. [1] https://youtu.be/KwD1hjlbhwU https://youtu.be/KwD1hjlbhwU
- agumonkey 11y ago
  She's barely 49% confident. I understand she doesn't want to enjoy a demo effect in a moving vehicle with 2 guests but it's a bit disturbing to see her hands constantly willing to grab the wheel.
- fdank 11y ago
  This is so clearly a fake demonstration. Watch the way the "automation" follows the probe car. It's so clearly leading it on and the woman is just going through an act.
  - rzzzt 11y ago
    In another video a different pair of accompanying passengers record a similar sequence of events, so the demonstration itself is probably scripted and rehearsed. But I wouldn't rule out that there is an actual piece of software working under these (controlled) conditions.
sandworm101 11y ago
As a Tesla enthusiast, would you buy a Tesla that wasn't capable of breaking the speed limit? Perhaps one that couldn't ever under any circumstances move faster than 70mph? That doesn't require fancy tech. It could be done today in an instant. That's where I see the real sticking point for automation. Driving isn't about getting from A to B, nor is safety the top priority. If it were, there wouldn't be such a thing as a V-8 (or whatever the electric equivalent is). I find it very ironic that a performance car like the Tesla might promote "the path to full car automation". You think the gun debate is tricky? Try telling people they cannot do 51 in a 50 anymore.
- alexland 11y ago
  It's a temporary problem though. Speed limits are set to try and keep people safe, by enforcing a maximum speed. Once computers drive the cars, the speed limit can be cranked up because reaction time goes up drastically and distracted drivers are nonexistent. I expect to see lots of changes in driving laws once automation takes over for exactly this reason.
  - Leszek 11y ago
    It's more likely to be cranked down though, because that's more efficient and people might be less frustrated with slow driving if they can kill time on their phones during the drive.
    - sandworm101 11y ago
      Kill time? Regardless of who is driving, time spend sitting the car is time spend away from family/pets/work/food/bathrooms and everything else important.
      - iopq 11y ago
        Don't worry, Tesla model S comes with a bathroom, a car office, a mini kitchen and a petting zoo
tsunamifury 11y ago
Direct control vs abstract control vs intelligence interaction has been a long running topic in HCI. For the most part it has been evolving in funny ways (i.e. skumorphism used metaphors from direct control to execute abstract control). I think the design language will evolve slowly with the users at a speed roughly related to the adoption curve. It may tell users everything that the car is deciding to do now, and as confidence in the system increases with adoption, it will do less and less. That is why more advance self-driving car researchers are working on the harder problems of getting abstract and intelligent interaction with a car working. The real market problem is just telling you car "take me to the store" rather than just getting it to drive down a straight lane.
wereHamster 11y ago
> pilots uncertainty regarding autopilot functioning is [...] the reason for several accidents. And compared to that, how many accidents were prevented by the use of autopilot? How many times did the autopilot make a better decision than a human would have? You can't compare to an absolutely perfect state (zero accidents). You have to compare to the current situation.
- ChicagoBoy11 11y ago
  I think in my comment i mentioned the fact that, taken as a whole, autopilot systems have made planes "so, so, so much safer." I was only remarking on the possibility that some of the issues that this automation have now created in aircrafts might also be seen in cars, especially when there are things like still keeping your hands on the steering wheel despite the fact that the car is in control. And, unlike your rational analysis, I don't trust politicians or the media to think about it like that. Instead, I'm sure the first accident that could even be remotely attributable to an automated system would immediately ignite a firestorm of bull-shit articles and possibly damaging regulation.
brlewis 11y ago
Sounds analagous to the transparency/traceability Sussman wants from AI: https://news.ycombinator.com/item?id=10388795 https://news.ycombinator.com/item?id=10388795

S_A_P 11y ago

Queue the James Bond movie sub plot where an evil villain takes control of his tesla/whatever autonomous car and he has to use a combination of hacker shell commands and sheer brute strength to save both him self and bond girl he was seducing while the car chauffeured him back to his luxury hotel suite...

mschuster91 11y ago
I recommend the Scorpions episode S01E20 "Crossroads", where a similar situation unfolds at the end.
gregpilling 11y ago
https://www.youtube.com/watch?v=qKAME9fAA-4 https://www.youtube.com/watch?v=qKAME9fAA-4 If the bad guys had only hacked Bond in this scene, it could have been much different!
tacojuan 11y ago
In the newer Ghost in the Shell series, an antagonist writes a virus that takes control of all the cars in the city, causing a massive physical/real-world denial of service.

Nadya 11y ago

>While planes have become so, so, so much safer because of all this automation, pilots uncertainty regarding autopilot functioning is a major concern nowadays, and the reason for several accidents. If % of accidents caused by pilot interference on a working system > % of accidents caused by system malfunctioning and pilot ignoring it: people will still be against not allowing pilots to interfere. Even when it causes more accidents... There's something about humans trusting humans more than machines that I don't fully understand. Systems can make mistakes but the amount of human mistakes is often exponentially greater to a degree of absurdity that humans are even trusted at all and yet people will side with the human over the machine. Humans will always want human oversight - even when that oversight does more harm than good once automation reaches a certain threshold... Special note: I'm not aware of avionics and the data on pilot interference w/ the system vs failure of pilot to intervene. So maybe this example doesn't hold very well for avionics...

tracker1 11y ago
A human can make judgement calls in unexpected situations. Objectively, a human is more likely to see falling debris and react to it than a self-driving car (of course depending on the programming and sensors). Or when the road is slick and you are doing 5 under, and you see someone pass you doing 5 over, with a car following too closely to it... you have a better understanding that there's likely to be an accident ahead... (I've seen this happen several times). A computer wouldn't predict that most likely. Unless ALL cars are automated, it is a must that a driver be able to take over quickly.
- baudehlo 11y ago
  I still am yet to understand how a self driving car deals with a cop directing traffic. Or even a construction worker holding traffic back from a backing up backhoe. But maybe I am not aware of the genius of the technology yet.
  - TeMPOraL 11y ago
    It's called image processing.

JulianMorrison 11y ago

There's an old joke about how the new planes have a pilot, for the sake of safety, and a dog, to bite the pilot if he tries to touch the controls. It sounds like it's time to introduce the dog?

sandworm101 11y ago
There is more to being a pilot than the stick-and-rudder of flying the plane. Judgment calls such as whether to attempt a landing under less-than-ideal conditions are far more important.
detaro 11y ago
More like a dog to gently nudge the pilot if he is relying on a system that isn't working. Most issues I've read about happened when the systems either weren't engaged properly or automatically disengaged because their operating parameters weren't met anymore, suddenly forcing the pilots to take control. Example Air France 447, where (presumably) the airspeed sensors where blocked by ice, which lead to the autopilot to disengage. There is also the theory that the pilots then made some of their mistakes based on the belief that the avionics would stop them from bringing the plane in an unsafe state and didn't realize that the system wasn't able to do so with missing information. (I hope I'm presenting this correctly, but that's what I remember reading afterwards) There is also the training aspect: if a system takes care of something 99,9% of the time, the pilot is less experienced in the 0,1% where it doesn't. There is a reason the safest airline pilots often fly way smaller aircraft as a hobby and get some instinct for manual flying there.
- ekianjo 11y ago
  No need for theories we know very well what happened there in that flight and its basically the copilots who crashed a perfectly fly worthy planed.
  - TuxerLulz 11y ago
    There is no basically in an aircraft accident. If you wish to make such a statement, please explain what actually happened in this flight (icing of the pitot tubes, deactivation of normal flight control law for the fly-by-wire system, flight at night over the ocean and over a storm, stall under the specifications for the stall warning system). It's respectful for the crew.

morb 11y ago

99% invisible did a pretty good episode on that: http://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ http://99percentinvisible.org/episode/children-of-the-magent...

ChicagoBoy11 11y ago
Yes! - <3 Roman Mars' voice.
- prawn 11y ago
  I like 99%I's content but Mars' voice is my least favourite part of it. Sounds a bit smug and is not always clearly audible in the car. Much prefer Freakonomics.

Sven7 11y ago

The "what the heck is it doing" problem may not be a problem with inexperienced pilots.

bdamm 11y ago
As an inexperienced pilot who has quite recently had the "what the heck is it doing" experience with my (40 yr old, analog computer) autopilot system, I can assure you that it is.
- Sven7 11y ago
  Maybe it should be no-experience pilots then :) But joking aside thats where these systems will eventually take us.

rzzzt 11y ago

There are videos on YouTube of Bosch fitting their automated driving technology into a Model S [1] and performing demonstration runs on a private road. The software shown in the video displays quite a lot of information about what the sensors consider relevant from the car's surroundings. [1] https://youtu.be/KwD1hjlbhwU https://youtu.be/KwD1hjlbhwU

agumonkey 11y ago
She's barely 49% confident. I understand she doesn't want to enjoy a demo effect in a moving vehicle with 2 guests but it's a bit disturbing to see her hands constantly willing to grab the wheel.
fdank 11y ago
This is so clearly a fake demonstration. Watch the way the "automation" follows the probe car. It's so clearly leading it on and the woman is just going through an act.
- rzzzt 11y ago
  In another video a different pair of accompanying passengers record a similar sequence of events, so the demonstration itself is probably scripted and rehearsed. But I wouldn't rule out that there is an actual piece of software working under these (controlled) conditions.

sandworm101 11y ago

As a Tesla enthusiast, would you buy a Tesla that wasn't capable of breaking the speed limit? Perhaps one that couldn't ever under any circumstances move faster than 70mph? That doesn't require fancy tech. It could be done today in an instant. That's where I see the real sticking point for automation. Driving isn't about getting from A to B, nor is safety the top priority. If it were, there wouldn't be such a thing as a V-8 (or whatever the electric equivalent is). I find it very ironic that a performance car like the Tesla might promote "the path to full car automation". You think the gun debate is tricky? Try telling people they cannot do 51 in a 50 anymore.

alexland 11y ago
It's a temporary problem though. Speed limits are set to try and keep people safe, by enforcing a maximum speed. Once computers drive the cars, the speed limit can be cranked up because reaction time goes up drastically and distracted drivers are nonexistent. I expect to see lots of changes in driving laws once automation takes over for exactly this reason.
- Leszek 11y ago
  It's more likely to be cranked down though, because that's more efficient and people might be less frustrated with slow driving if they can kill time on their phones during the drive.
  - sandworm101 11y ago
    Kill time? Regardless of who is driving, time spend sitting the car is time spend away from family/pets/work/food/bathrooms and everything else important.
    - iopq 11y ago
      Don't worry, Tesla model S comes with a bathroom, a car office, a mini kitchen and a petting zoo

tsunamifury 11y ago

Direct control vs abstract control vs intelligence interaction has been a long running topic in HCI. For the most part it has been evolving in funny ways (i.e. skumorphism used metaphors from direct control to execute abstract control). I think the design language will evolve slowly with the users at a speed roughly related to the adoption curve. It may tell users everything that the car is deciding to do now, and as confidence in the system increases with adoption, it will do less and less. That is why more advance self-driving car researchers are working on the harder problems of getting abstract and intelligent interaction with a car working. The real market problem is just telling you car "take me to the store" rather than just getting it to drive down a straight lane.

wereHamster 11y ago

> pilots uncertainty regarding autopilot functioning is [...] the reason for several accidents. And compared to that, how many accidents were prevented by the use of autopilot? How many times did the autopilot make a better decision than a human would have? You can't compare to an absolutely perfect state (zero accidents). You have to compare to the current situation.

ChicagoBoy11 11y ago
I think in my comment i mentioned the fact that, taken as a whole, autopilot systems have made planes "so, so, so much safer." I was only remarking on the possibility that some of the issues that this automation have now created in aircrafts might also be seen in cars, especially when there are things like still keeping your hands on the steering wheel despite the fact that the car is in control. And, unlike your rational analysis, I don't trust politicians or the media to think about it like that. Instead, I'm sure the first accident that could even be remotely attributable to an automated system would immediately ignite a firestorm of bull-shit articles and possibly damaging regulation.

brlewis 11y ago

Sounds analagous to the transparency/traceability Sussman wants from AI: https://news.ycombinator.com/item?id=10388795 https://news.ycombinator.com/item?id=10388795