5 ms·

The history of human machine interactions makes it clear that if you have the human as the primary control (in the main decision loop, operating things) and the

by Syonyk 5y ago

The history of human machine interactions makes it clear that if you have the human as the primary control (in the main decision loop, operating things) and the automation watching over their shoulders, it works pretty well. The opposite - where the automation handles the common case, with the human watching, simply does not work. It's not worked in aviation, it's not worked in industrial systems, it's not worked in self driving cars. This isn't a novel bit of research, this is well understood. "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work.

antisthenes 5y ago
> "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work. Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety. I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone. At least with an unreliable system that Tesla demonstrated here.
ethbr0 5y ago
That's a great way to put it! It should be indicative of the kind and volume of licensing and training we require of commercial pilots, in order to use a more limited and situationally simple autopilot. And we still get Boeing-style @&$#-ups at the UX interface.
- Syonyk 5y ago
  The use of the term "autopilot" is particularly cute here, because any pilot who's ever used one knows that autopilots are dumb as rocks, can't really be trusted, and don't do nearly as much as non-pilots tend to assume they do. "You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important. "In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.
nkingsy 5y ago
The watching is the sticking point. It seems this could be ameliorated if the automation could be 100% trusted to know when it needs an intervention. Solving for that, even a system that was extremely conservative would be useful, as you are completely freed up when it is operating, even if that is only 20% of the time.
Theodores 5y ago
A century ago horses had Full Self Driving with the human having to be vigilant and in charge. The horse could go crazy in scary situations endangering others but you could train that out and make a good war horse. Basic stuff like following the road around a bend was something the horse could do but you could not tell it to autonomously go across town to pick someone up. People were okay with horses and knew what to expect. Tesla need to rebrand autopilot to 'horse mode' and reset people's expectations of FSD.
- jazzyjackson 5y ago
  Horses could at least make their own way back home, and they were self-fueling to boot!
  - QuibbleQuota 5y ago
    Definitely not zero emissions though…
  - Nohortax 5y ago
    The problem is that they are self-fueling all the time where grass is if you are not aware or if you didn't teach well your horse to avoid eating each blade of grass on the way. The journey might be very long haha
- foobiekr 5y ago
  Horses do not run into concrete pillars.
  - ahartmetz 5y ago
    Yeah. For the purpose of behavior prediction while driving, horses can be approximated as very dull, very easily panicked humans. They have all the regular instincts (collision avoidance, fear of heights etc) that keep animals alive. FSD is a complete alien that might try to kill itself for the most trivial and inscrutable reason like the sensor noise excited the neural network in a certain way.
bumby 5y ago
>"I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" Do you have thoughts on why this is? Is it because, almost by definition, the remaining 0.1% are the toughest problems so the hardest to catch/solve? Or is it a human-factors issue where people get complacent when they know they can count on the machine 99.9% of the time and they lose the discipline to be vigilant enough to catch the remaining problems?
- jazzyjackson 5y ago
  You might be interested in a couple of episodes of 99% invisible, called “Children of the Magenta” about pilots becoming complacent because the machine usually takes care of problems for them. Basically if you grow up with a self driving car and never learn how to handle small fuck ups, how can you ever be expected to recover from a big fuckup? (Bad sensor reading, bad weather, shredded tire, etc)

by Syonyk 5y ago

antisthenes 5y ago

> "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work. Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety. I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone. At least with an unreliable system that Tesla demonstrated here.

ethbr0 5y ago

That's a great way to put it! It should be indicative of the kind and volume of licensing and training we require of commercial pilots, in order to use a more limited and situationally simple autopilot. And we still get Boeing-style @&$#-ups at the UX interface.

Syonyk 5y ago
The use of the term "autopilot" is particularly cute here, because any pilot who's ever used one knows that autopilots are dumb as rocks, can't really be trusted, and don't do nearly as much as non-pilots tend to assume they do. "You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important. "In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.

nkingsy 5y ago

The watching is the sticking point. It seems this could be ameliorated if the automation could be 100% trusted to know when it needs an intervention. Solving for that, even a system that was extremely conservative would be useful, as you are completely freed up when it is operating, even if that is only 20% of the time.

Theodores 5y ago

A century ago horses had Full Self Driving with the human having to be vigilant and in charge. The horse could go crazy in scary situations endangering others but you could train that out and make a good war horse. Basic stuff like following the road around a bend was something the horse could do but you could not tell it to autonomously go across town to pick someone up. People were okay with horses and knew what to expect. Tesla need to rebrand autopilot to 'horse mode' and reset people's expectations of FSD.

jazzyjackson 5y ago
Horses could at least make their own way back home, and they were self-fueling to boot!
- QuibbleQuota 5y ago
  Definitely not zero emissions though…
- Nohortax 5y ago
  The problem is that they are self-fueling all the time where grass is if you are not aware or if you didn't teach well your horse to avoid eating each blade of grass on the way. The journey might be very long haha
foobiekr 5y ago
Horses do not run into concrete pillars.
- ahartmetz 5y ago
  Yeah. For the purpose of behavior prediction while driving, horses can be approximated as very dull, very easily panicked humans. They have all the regular instincts (collision avoidance, fear of heights etc) that keep animals alive. FSD is a complete alien that might try to kill itself for the most trivial and inscrutable reason like the sensor noise excited the neural network in a certain way.

bumby 5y ago

>"I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" Do you have thoughts on why this is? Is it because, almost by definition, the remaining 0.1% are the toughest problems so the hardest to catch/solve? Or is it a human-factors issue where people get complacent when they know they can count on the machine 99.9% of the time and they lose the discipline to be vigilant enough to catch the remaining problems?

jazzyjackson 5y ago
You might be interested in a couple of episodes of 99% invisible, called “Children of the Magenta” about pilots becoming complacent because the machine usually takes care of problems for them. Basically if you grow up with a self driving car and never learn how to handle small fuck ups, how can you ever be expected to recover from a big fuckup? (Bad sensor reading, bad weather, shredded tire, etc)