5 ms·

> drivers who are not trained to specifically respond to it's mistakes What would this entail? Perhaps some sort of "license" which allows the user to operator

> drivers who are not trained to specifically respond to it's mistakes What would this entail? Perhaps some sort of "license" which allows the user to operator a motor vehicle?

ethbr0 5y ago
Safely operating a motor vehicle via a steering wheel, accelerator, and brake (and sometimes clutch and gearshift) is a completely different skillset than monitoring an automated system in realtime. Novel skill requirements include: interpreting FSD UI, anticipating common errors, recognizing intent behind FSD actions, & remaining vigilant even during periods of autonomous success.
- Syonyk 5y ago
  The history of human machine interactions makes it clear that if you have the human as the primary control (in the main decision loop, operating things) and the automation watching over their shoulders, it works pretty well. The opposite - where the automation handles the common case, with the human watching, simply does not work. It's not worked in aviation, it's not worked in industrial systems, it's not worked in self driving cars. This isn't a novel bit of research, this is well understood. "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work.
  - antisthenes 5y ago
    > "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work. Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety. I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone. At least with an unreliable system that Tesla demonstrated here.
  - ethbr0 5y ago
    That's a great way to put it! It should be indicative of the kind and volume of licensing and training we require of commercial pilots, in order to use a more limited and situationally simple autopilot. And we still get Boeing-style @&$#-ups at the UX interface.
    - Syonyk 5y ago
      The use of the term "autopilot" is particularly cute here, because any pilot who's ever used one knows that autopilots are dumb as rocks, can't really be trusted, and don't do nearly as much as non-pilots tend to assume they do. "You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important. "In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.
  - nkingsy 5y ago
    The watching is the sticking point. It seems this could be ameliorated if the automation could be 100% trusted to know when it needs an intervention. Solving for that, even a system that was extremely conservative would be useful, as you are completely freed up when it is operating, even if that is only 20% of the time.
  - Theodores 5y ago
    A century ago horses had Full Self Driving with the human having to be vigilant and in charge. The horse could go crazy in scary situations endangering others but you could train that out and make a good war horse. Basic stuff like following the road around a bend was something the horse could do but you could not tell it to autonomously go across town to pick someone up. People were okay with horses and knew what to expect. Tesla need to rebrand autopilot to 'horse mode' and reset people's expectations of FSD.
    - jazzyjackson 5y ago
      Horses could at least make their own way back home, and they were self-fueling to boot!
      - QuibbleQuota 5y ago
        Definitely not zero emissions though…
      - Nohortax 5y ago
        The problem is that they are self-fueling all the time where grass is if you are not aware or if you didn't teach well your horse to avoid eating each blade of grass on the way. The journey might be very long haha
    - foobiekr 5y ago
      Horses do not run into concrete pillars.
      - ahartmetz 5y ago
        Yeah. For the purpose of behavior prediction while driving, horses can be approximated as very dull, very easily panicked humans. They have all the regular instincts (collision avoidance, fear of heights etc) that keep animals alive. FSD is a complete alien that might try to kill itself for the most trivial and inscrutable reason like the sensor noise excited the neural network in a certain way.
  - bumby 5y ago
    >"I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" Do you have thoughts on why this is? Is it because, almost by definition, the remaining 0.1% are the toughest problems so the hardest to catch/solve? Or is it a human-factors issue where people get complacent when they know they can count on the machine 99.9% of the time and they lose the discipline to be vigilant enough to catch the remaining problems?
    - jazzyjackson 5y ago
      You might be interested in a couple of episodes of 99% invisible, called “Children of the Magenta” about pilots becoming complacent because the machine usually takes care of problems for them. Basically if you grow up with a self driving car and never learn how to handle small fuck ups, how can you ever be expected to recover from a big fuckup? (Bad sensor reading, bad weather, shredded tire, etc)
- mbreese 5y ago
  I'm going to posit that none of those novel skill requirements are actually requirements. You don't need to interpret a UI, or recognize intent to know that the car shouldn't turn the wrong way on a one-way street or hit a planter. Anticipating common errors and remaining vigilant while driving is a requirement for all driving, not when dealing with FSD. If you are capable of driving a car safely to begin with, then you're capable of recognizing then the car is going to do something wrong and turning the wheel and/or hitting the brakes. I wouldn't personally trust a FSD beta, but only if it is properly supervised, I think it could be tested safely. But, there's the problem -- how can you really properly supervise this? How many drivers are going to try this and just let the car drive w/o paying attention? Or, using it under controlled conditions, but then giving it too much trust in more adverse conditions?
  - ethbr0 5y ago
    What makes them required is the speed that a reaction is expected. In both the quick and velocity senses of the word. Unless the user is able to understand and anticipate common reactions, the entire system is unsafe at >25 mph or so. Or, to put it another way, I'd love to see the statistics on how many drivers can react quickly enough to avert an unexpected turn into an exit divider at 60 mph, after a number of days of interruption free FSD.
    - mbreese 5y ago
      We’re largely talking about an update where the primary benefit is low-speed automation — driving in areas where the speed limits are low because the obstacles are many.
- tshaddox 5y ago
  No, it’s just unsafe (and presumably illegal in most places) to operate a motor vehicle without maintaining constant awareness of the road and control of the vehicle. There’s no additional training or license that permits you to stop maintaining that awareness and control, just like there’s no additional training that permits you to drive under the influence of alcohol.
  - throwaway0a5e 5y ago
    >There’s no additional training or license that permits you to stop maintaining that awareness and control Being a cop on duty often exempts you from distracted driving laws. I'm not sure how they can do stuff on a laptop while driving safely but I assume the state knows best. /s
  - kemitche 5y ago
    The theoretical "additional license" would be in addition to maintaining awareness, not a substitution. So all the normal road awareness, plus being informed of likely FSD failure conditions, and anticipating them / being ready and capable to intercede.
    - tshaddox 5y ago
      If you maintain control at all times, you shouldn’t need additional training for any of the car’s features. We don’t require additional training for cruise control, lane-keep assistance, anti-lock brakes, etc.
  - ectopod 5y ago
    I agree, but if you have to maintain constant control of the vehicle then it isn't self-driving.
simion314 5y ago
Maybe people that are trained with such failure videos that show what can go wrong and NOT with propaganda videos that only show the good parts, causing the driver not to pay attention.
vkou 5y ago
A license that allows you to operate a motor vehicle with a beta self-driving feature. It's very similar to a regular motor vehicle, but has different failure modes. A motorcycle is similar to an automobile, but has different failure modes, and needs a special license. A >5 tonne truck is very similar to an automobile, but has different failure modes, and needs a special license. An automobile that usually drives itself, but sometimes tries to crash itself has different failure modes from an automobile that does not try to drive itself.
AlotOfReading 5y ago
California at least has a specific license for this called the Autonomous Vehicle Operator license. It enforces some minimal amount of training beyond simply having a regular driver's license.
bestouff 5y ago
I guess any driving instructor worth its salt would have the required skills to correct the vehicle if it attempts to do something weird. After all FSD is a still-in-training (maybe for an unusual long time) driver.
- Syonyk 5y ago
  Probably not. It fails in ways that are totally different from how humans typically fail at driving. Student drivers have the ability to comprehend and understand the 3D world around them - it's just a matter of learning the proper control manipulations to make the car do what you want, and then how to interact with other traffic in safe and expected ways. Tesla's FSD system seems to fail at basic 3D world comprehension and reading, often enough.
visarga 5y ago
Perhaps operating a motor vehicle is different from supervising an AI doing that same task.
bigtex 5y ago
Giant obxonious flashing lights and blinking signs stating "Student Self Driving tech on board"
- kyleee 5y ago
  "Student Full Self Driver"
bobsomers 5y ago
Safety drivers at proper AV companies usually go through several weeks of rigorous classroom instruction and test track driving with a professional driving instructor, including lots of practice with the real AVs on closed courses with simulated system faults, takeover situations, etc. Anecdotally I've seen trained safety drivers take over from unexpected system failures at speeds near the floor of human reaction time. They are some of the best and most attentive drivers on the road. Average folks beta testing Tesla's software for them are woefully under-prepared to safely mitigate the risks these vehicles pose. In several cases they've meaninglessly died so that Tesla could learn a lesson everyone else saw coming months in advance.
paxys 5y ago
What part of the driving test covers taking over control from an automated vehicle with a split second notice?

> drivers who are not trained to specifically respond to it's mistakes What would this entail? Perhaps some sort of "license" which allows the user to operator a motor vehicle?

ethbr0 5y ago

Safely operating a motor vehicle via a steering wheel, accelerator, and brake (and sometimes clutch and gearshift) is a completely different skillset than monitoring an automated system in realtime. Novel skill requirements include: interpreting FSD UI, anticipating common errors, recognizing intent behind FSD actions, & remaining vigilant even during periods of autonomous success.

Syonyk 5y ago
The history of human machine interactions makes it clear that if you have the human as the primary control (in the main decision loop, operating things) and the automation watching over their shoulders, it works pretty well. The opposite - where the automation handles the common case, with the human watching, simply does not work. It's not worked in aviation, it's not worked in industrial systems, it's not worked in self driving cars. This isn't a novel bit of research, this is well understood. "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work.
- antisthenes 5y ago
  > "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work. Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety. I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone. At least with an unreliable system that Tesla demonstrated here.
- ethbr0 5y ago
  That's a great way to put it! It should be indicative of the kind and volume of licensing and training we require of commercial pilots, in order to use a more limited and situationally simple autopilot. And we still get Boeing-style @&$#-ups at the UX interface.
  - Syonyk 5y ago
    The use of the term "autopilot" is particularly cute here, because any pilot who's ever used one knows that autopilots are dumb as rocks, can't really be trusted, and don't do nearly as much as non-pilots tend to assume they do. "You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important. "In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.
- nkingsy 5y ago
  The watching is the sticking point. It seems this could be ameliorated if the automation could be 100% trusted to know when it needs an intervention. Solving for that, even a system that was extremely conservative would be useful, as you are completely freed up when it is operating, even if that is only 20% of the time.
- Theodores 5y ago
  A century ago horses had Full Self Driving with the human having to be vigilant and in charge. The horse could go crazy in scary situations endangering others but you could train that out and make a good war horse. Basic stuff like following the road around a bend was something the horse could do but you could not tell it to autonomously go across town to pick someone up. People were okay with horses and knew what to expect. Tesla need to rebrand autopilot to 'horse mode' and reset people's expectations of FSD.
  - jazzyjackson 5y ago
    Horses could at least make their own way back home, and they were self-fueling to boot!
    - QuibbleQuota 5y ago
      Definitely not zero emissions though…
    - Nohortax 5y ago
      The problem is that they are self-fueling all the time where grass is if you are not aware or if you didn't teach well your horse to avoid eating each blade of grass on the way. The journey might be very long haha
  - foobiekr 5y ago
    Horses do not run into concrete pillars.
    - ahartmetz 5y ago
      Yeah. For the purpose of behavior prediction while driving, horses can be approximated as very dull, very easily panicked humans. They have all the regular instincts (collision avoidance, fear of heights etc) that keep animals alive. FSD is a complete alien that might try to kill itself for the most trivial and inscrutable reason like the sensor noise excited the neural network in a certain way.
- bumby 5y ago
  >"I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" Do you have thoughts on why this is? Is it because, almost by definition, the remaining 0.1% are the toughest problems so the hardest to catch/solve? Or is it a human-factors issue where people get complacent when they know they can count on the machine 99.9% of the time and they lose the discipline to be vigilant enough to catch the remaining problems?
  - jazzyjackson 5y ago
    You might be interested in a couple of episodes of 99% invisible, called “Children of the Magenta” about pilots becoming complacent because the machine usually takes care of problems for them. Basically if you grow up with a self driving car and never learn how to handle small fuck ups, how can you ever be expected to recover from a big fuckup? (Bad sensor reading, bad weather, shredded tire, etc)
mbreese 5y ago
I'm going to posit that none of those novel skill requirements are actually requirements. You don't need to interpret a UI, or recognize intent to know that the car shouldn't turn the wrong way on a one-way street or hit a planter. Anticipating common errors and remaining vigilant while driving is a requirement for all driving, not when dealing with FSD. If you are capable of driving a car safely to begin with, then you're capable of recognizing then the car is going to do something wrong and turning the wheel and/or hitting the brakes. I wouldn't personally trust a FSD beta, but only if it is properly supervised, I think it could be tested safely. But, there's the problem -- how can you really properly supervise this? How many drivers are going to try this and just let the car drive w/o paying attention? Or, using it under controlled conditions, but then giving it too much trust in more adverse conditions?
- ethbr0 5y ago
  What makes them required is the speed that a reaction is expected. In both the quick and velocity senses of the word. Unless the user is able to understand and anticipate common reactions, the entire system is unsafe at >25 mph or so. Or, to put it another way, I'd love to see the statistics on how many drivers can react quickly enough to avert an unexpected turn into an exit divider at 60 mph, after a number of days of interruption free FSD.
  - mbreese 5y ago
    We’re largely talking about an update where the primary benefit is low-speed automation — driving in areas where the speed limits are low because the obstacles are many.
tshaddox 5y ago
No, it’s just unsafe (and presumably illegal in most places) to operate a motor vehicle without maintaining constant awareness of the road and control of the vehicle. There’s no additional training or license that permits you to stop maintaining that awareness and control, just like there’s no additional training that permits you to drive under the influence of alcohol.
- throwaway0a5e 5y ago
  >There’s no additional training or license that permits you to stop maintaining that awareness and control Being a cop on duty often exempts you from distracted driving laws. I'm not sure how they can do stuff on a laptop while driving safely but I assume the state knows best. /s
- kemitche 5y ago
  The theoretical "additional license" would be in addition to maintaining awareness, not a substitution. So all the normal road awareness, plus being informed of likely FSD failure conditions, and anticipating them / being ready and capable to intercede.
  - tshaddox 5y ago
    If you maintain control at all times, you shouldn’t need additional training for any of the car’s features. We don’t require additional training for cruise control, lane-keep assistance, anti-lock brakes, etc.
- ectopod 5y ago
  I agree, but if you have to maintain constant control of the vehicle then it isn't self-driving.

simion314 5y ago

Maybe people that are trained with such failure videos that show what can go wrong and NOT with propaganda videos that only show the good parts, causing the driver not to pay attention.

vkou 5y ago

A license that allows you to operate a motor vehicle with a beta self-driving feature. It's very similar to a regular motor vehicle, but has different failure modes. A motorcycle is similar to an automobile, but has different failure modes, and needs a special license. A >5 tonne truck is very similar to an automobile, but has different failure modes, and needs a special license. An automobile that usually drives itself, but sometimes tries to crash itself has different failure modes from an automobile that does not try to drive itself.

AlotOfReading 5y ago

California at least has a specific license for this called the Autonomous Vehicle Operator license. It enforces some minimal amount of training beyond simply having a regular driver's license.

bestouff 5y ago

I guess any driving instructor worth its salt would have the required skills to correct the vehicle if it attempts to do something weird. After all FSD is a still-in-training (maybe for an unusual long time) driver.

Syonyk 5y ago
Probably not. It fails in ways that are totally different from how humans typically fail at driving. Student drivers have the ability to comprehend and understand the 3D world around them - it's just a matter of learning the proper control manipulations to make the car do what you want, and then how to interact with other traffic in safe and expected ways. Tesla's FSD system seems to fail at basic 3D world comprehension and reading, often enough.

visarga 5y ago

Perhaps operating a motor vehicle is different from supervising an AI doing that same task.

bigtex 5y ago

Giant obxonious flashing lights and blinking signs stating "Student Self Driving tech on board"

kyleee 5y ago
"Student Full Self Driver"

bobsomers 5y ago

Safety drivers at proper AV companies usually go through several weeks of rigorous classroom instruction and test track driving with a professional driving instructor, including lots of practice with the real AVs on closed courses with simulated system faults, takeover situations, etc. Anecdotally I've seen trained safety drivers take over from unexpected system failures at speeds near the floor of human reaction time. They are some of the best and most attentive drivers on the road. Average folks beta testing Tesla's software for them are woefully under-prepared to safely mitigate the risks these vehicles pose. In several cases they've meaninglessly died so that Tesla could learn a lesson everyone else saw coming months in advance.

paxys 5y ago

What part of the driving test covers taking over control from an automated vehicle with a split second notice?