8 ms·

Summoned Tesla Collides with Jet

67 points by mzs 5y ago

sinuhe69 5y ago
I think that’s a prime example for the shortcomings of machine learning. It can only “learn” from what is known, anything unknown can be potentially a catastrophe. I believe with the current AI in its infancy, a combination of rule-based and machine learning would be a more dependable approach.
- joshribakoff 5y ago
  The problem with your statement is that AI drivers do rely on traditional logic already, but that cannot be applied to the perception part of the stack. It doesn’t matter if your rule is “if perceives truck, stop” if perception is telling you there is no truck stopped there. The rule also can’t be “if white pixel, stop” or “if Lidar bounces off steam and returns, stop”
- andbberger 5y ago
  if you're doing vision based obstacle avoidance and you're training your model on specific classes of objects and not just object you're doing it wrong
tailspin2019 5y ago
I was expecting a catch. Perhaps a play on words or something. Nope. Just an actual Tesla colliding with an actual jet. What’s surprising to me is that it keeps moving post collision… seems odd that it can meet with unexpected resistance and not trigger some sort of emergency measure. (eg. stopping) I used to be bullish about Tesla’s “only cameras” approach, and I still think they’ll crack FSD in some form, but I am increasingly wondering if there needs to be additional sensors for an abundance of extra safety around collision avoidance.
- throwaway0a5e 5y ago
  >seems odd that it can meet with unexpected resistance and not trigger some sort of emergency measure. (eg. stopping) This is a requirement unless you want it getting stuck on minor bumps and whatnot.
  - tailspin2019 5y ago
    But… minor bumps != airplane…
    - throwaway0a5e 5y ago
      Does an airplane provide substantially more resistance than trying to roll through a decent sized pothole without the benefit of much momentum? That airplane is small enough it probably gets dragged around by hand when it's not chocked. The chocks small airports tend to use are pretty small to fit under the wheel fairings many small planes have. I can see it pushing the aircraft around by the tail.
      - tailspin2019 5y ago
        Fair enough. So judging by resistance alone may not be enough…. Needs additional forward facing sensors then? (For both long and short distance object detection?) Earlier on in Autopilot/FSD beta testing it was easier to overlook edge cases like this but I think we’re several years into this now and I’m starting to feel that at this stage it should be almost impossible for these cars to be driving into stationary objects. I’m a proponent of what Tesla are doing but they need to solve this ASAP now, whether by hardware or software, before it erodes too much more confidence (especially as it’s all too easy for people to extrapolate from these documented edge cases and believe these vehicles to be less safe than they actually are).
      - FireBeyond 5y ago
        > That airplane is small enough it probably gets dragged around by hand when it's not chocked. Vision Jet weighs 6,000 lb at MTOW. So, unlikely.
- dreamcompiler 5y ago
  This seems exactly the same kind of error as the Florida Tesla that underran a crossing semi trailer and killed the driver. Teslas seem bad at detecting obstacles that are at about windshield height but don't touch the ground. Possibly because they don't train the system with such obstacles. Or not enough, anyway.
IronWolve 5y ago
So the real question, does your car insurance cover your car in autopilot mode doing damage to an airplane?
- kylehotchkiss 5y ago
  Seems like the airplane owner would sue the owner of the car who should be covered by the insurance for some property damages and the car insurance might go back and sue tesla to recover funds given how unusual the situation is
deleted 5y ago
[deleted]
rvz 5y ago
It doesn't even stop after hitting the jet. It just keeps going right through it. I don't think you can take this autopilot, and FSD (Fools Self Driving) contraption seriously or remotely ready for customers to use; even for a 'beta'. Since it's functionality doesn't even work as advertised, puts the lives of other drivers on the road at risk and it is completely useless at the most dangerous time to drive, which is at night. For example, as a simple test against another autonomous vehicle with LIDAR at night. [0] The Tesla with FSD failed to stop and just ran over the object. The best part is 100,000 crash dummies have this software installed and seem to be taking the risk in trying it out, even when it malfunctions on the public roads and what we can see here. This contraption needs to be thoroughly investigated. [0] https://twitter.com/TaylorOgan/status/1478802681141645322 https://twitter.com/TaylorOgan/status/1478802681141645322
- booboofixer 5y ago
  > It doesn't even stop after hitting the jet. It just keeps going right through it. I thought that was the worst part of it too, assuming theres nobody inside the car at the wheel. It keeps going, stops, then moves again and looks to go around whatever obstacle it thinks exists. What if this situation can be replicated on an empty road where the tesla runs over a human being? In the worst case, does the tesla keep trying to drive over the human being and turns a minor injury into a fatality?
- misiti3780 5y ago
  how do you propose we get to level 4 without a large number of humans collecting the data and providing real-time feedback?
  - roughly 5y ago
    Are you talking about the humans inside or outside the car?
    - misiti3780 5y ago
      inside driving the cars
netsharc 5y ago
This car reviewer tried it (with his mom holding the button) and it would've pancaked him too (they tried it twice, once with his mom's motherly instinct letting go of the button): https://youtu.be/kW2W2l2kXFM?t=410 https://youtu.be/kW2W2l2kXFM?t=410
robga 5y ago
Autopilot envy?
roughly 5y ago
I remember part of the report on the Uber that ran the person over was that the Uber’s classifier couldn’t figure out what a person with a bike in the road was, so it didn’t make it into the decision tree. I’m curious if that’s what happened here - Tesla never trained their cars to recognize airplanes, so it just didn’t “see” it.
- julianlam 5y ago
  Suppose we should just get ready for ReCAPTCHA challenges involving various parts of airplanes?
  - roughly 5y ago
    The ever relevant XKCD: https://xkcd.com/1897/ https://xkcd.com/1897/
- badwolf 5y ago
  The part that baffles me is that there isn't a "There's an object, not sure what it is, but there's something, I should stop/avoid it" instead it seems to be "I don't know what this object is. Oh well, it's probably fine to plow into it"
- giarc 5y ago
  I wonder if it couldn't "see" the jet since it has an all white underbelly and perhaps it blended into the sky or some hanger behind it.
- anikan_vader 5y ago
  The Uber self-driving car was actually a Volvo -- its logs show that the Volvo emergency braking system had detected a pedestrian crossing almost 2 seconds before the collision. It would've initiated emergency braking, but Uber had disabled this feature, since, of course, they were testing their own shitty self-driving software. The Uber software chose to take no action, the Uber employee at the wheel was apparently paying no attention, and Uber had disabled the Volvo software, the only agent involved that actually thought it would be a good idea to pay attention and try not to kill a woman with a bike on a dark night. Uber shouldn't be allowed to test their software on public roads in my opinion.
BXLE_1-1-BitIs1 5y ago
Insurance policies can exclude coverage off public roads. If the Tesla and aircraft owner are one and same, the owner may be holding the bag for the damage. The aircraft owner and insurer could sue Tesla, but likely Tesla's terms of use require the car only be on public roads and private driveways.
metamuas 5y ago
Can someone explain to me why Tesla didn't bootstrap itself with cameras/LIDAR, then remove the LIDAR? I mean teaching an AI depth perception and the ability to model objects by vision, without including training data from LIDAR just seems kind of silly.
- yunyu 5y ago
  How many people would buy a Tesla that costs 3x as much?
- mariebks 5y ago
  That’s exactly what they did. They used LiDAR to train their vision neural nets to accurately compute the distance of every pixel. https://cleantechnica.com/2020/08/03/tesla-achieved-the-accuracy-of-lidar-with-its-advanced-computer-vision-tech/amp/ https://cleantechnica.com/2020/08/03/tesla-achieved-the-accu...
  - mariebks 5y ago
    https://www.teslarati.com/tesla-model-x-lidar-equipment-photos/amp/ https://www.teslarati.com/tesla-model-x-lidar-equipment-phot...
58028641 5y ago
Tesla very clearly states: "Smart Summon is a BETA feature. You must continually monitor the vehicle and its surroundings and stay prepared to take immediate action at any time. It is the driver's responsibility to use Smart Summon safely, responsibly, and as intended." "When using Smart Summon, you must maintain a clear line of sight between you and Model Y and stay prepared to stop the vehicle at any time by releasing the button on the mobile app." https://www.tesla.com/ownersmanual/modely/is_is/GUID-6B9A1AEA-579C-400E-A7A6-E4916BCD5DED.html https://www.tesla.com/ownersmanual/modely/is_is/GUID-6B9A1AE...
- twayt 5y ago
  If these are the only kind of edge cases where smart summon fails, I'd say it is actually doing pretty good.
  - codezero 5y ago
    I saw someone using it on the top of the parking garage at Disney land. It just circled around but I found it fascinating to watch and am looking forward to when it can go down the ramp :)
- sillyinseattle 5y ago
  How thoughtful of Tesla ;-) One nearly backed into me a few months back. I banged on the car to alert the driver. It stopped. There was no one in it. Owner was summoning her car in a grocery store parking lot. People.
  - rich_sasha 5y ago
    Rest assured, if you were maimed, it would be the owner’s responsibility, not Tesla’s, so all is good. Presumably the driver is insured, right? And insurance covers the his nonsense...
    - sorokod 5y ago
      Isn't Tesla in car insurance itself? Interesting how that recursive setup would work out.
mongol 5y ago
Imagine flying with an airline that had a beta version of the autopilot installed in its airplane. The plane crashes and you get maimed. The airline is responsible but the airplane manufacturer is in the clear. It was a beta version, after all.
- deleted 5y ago
  [deleted]
- sorokod 5y ago
  Would such an airplane be legally allowed to fly passengers? If so, I'd be going after the authority that made this legal.
kappuchino 5y ago
Question to the experts: Is this kind of a classification or a detection problem ... or both? I remember that visual algorithms for classificaion can get confused by objects or situations they were not trained on, so in the result there were unpredictable outcomes in situations that seemed safe before. Or is it more like a "shape and size of a plane" issue?
- salawat 5y ago
  It's likely because it's an overhang. System has the same tordid reputation with trailers.
anikan_vader 5y ago
Not colliding with immobile objects at low speed is a solved problem. Tesla's just choosing to go without radar/lidar. This inability to model the 3d world makes me very skeptical that Tesla's self-driving cars are competitive with Google's. Honestly it's embarrassing; if a human behaved this way behind the wheel of a car, I would assume they were high on something stronger than weed.
- ragingrobot 5y ago
  Why high? Someone else in the thread made a remark about the insinuation of a "phone in front of my face" but in fact this could very easily happen by a sober person just simply not paying attention, and that is fairly common.

67 points by mzs 5y ago

I think that’s a prime example for the shortcomings of machine learning. It can only “learn” from what is known, anything unknown can be potentially a catastrophe. I believe with the current AI in its infancy, a combination of rule-based and machine learning would be a more dependable approach.

joshribakoff 5y ago
The problem with your statement is that AI drivers do rely on traditional logic already, but that cannot be applied to the perception part of the stack. It doesn’t matter if your rule is “if perceives truck, stop” if perception is telling you there is no truck stopped there. The rule also can’t be “if white pixel, stop” or “if Lidar bounces off steam and returns, stop”
andbberger 5y ago
if you're doing vision based obstacle avoidance and you're training your model on specific classes of objects and not just object you're doing it wrong

tailspin2019 5y ago

I was expecting a catch. Perhaps a play on words or something. Nope. Just an actual Tesla colliding with an actual jet. What’s surprising to me is that it keeps moving post collision… seems odd that it can meet with unexpected resistance and not trigger some sort of emergency measure. (eg. stopping) I used to be bullish about Tesla’s “only cameras” approach, and I still think they’ll crack FSD in some form, but I am increasingly wondering if there needs to be additional sensors for an abundance of extra safety around collision avoidance.

throwaway0a5e 5y ago
>seems odd that it can meet with unexpected resistance and not trigger some sort of emergency measure. (eg. stopping) This is a requirement unless you want it getting stuck on minor bumps and whatnot.
- tailspin2019 5y ago
  But… minor bumps != airplane…
  - throwaway0a5e 5y ago
    Does an airplane provide substantially more resistance than trying to roll through a decent sized pothole without the benefit of much momentum? That airplane is small enough it probably gets dragged around by hand when it's not chocked. The chocks small airports tend to use are pretty small to fit under the wheel fairings many small planes have. I can see it pushing the aircraft around by the tail.
    - tailspin2019 5y ago
      Fair enough. So judging by resistance alone may not be enough…. Needs additional forward facing sensors then? (For both long and short distance object detection?) Earlier on in Autopilot/FSD beta testing it was easier to overlook edge cases like this but I think we’re several years into this now and I’m starting to feel that at this stage it should be almost impossible for these cars to be driving into stationary objects. I’m a proponent of what Tesla are doing but they need to solve this ASAP now, whether by hardware or software, before it erodes too much more confidence (especially as it’s all too easy for people to extrapolate from these documented edge cases and believe these vehicles to be less safe than they actually are).
    - FireBeyond 5y ago
      > That airplane is small enough it probably gets dragged around by hand when it's not chocked. Vision Jet weighs 6,000 lb at MTOW. So, unlikely.
dreamcompiler 5y ago
This seems exactly the same kind of error as the Florida Tesla that underran a crossing semi trailer and killed the driver. Teslas seem bad at detecting obstacles that are at about windshield height but don't touch the ground. Possibly because they don't train the system with such obstacles. Or not enough, anyway.

IronWolve 5y ago

So the real question, does your car insurance cover your car in autopilot mode doing damage to an airplane?

kylehotchkiss 5y ago
Seems like the airplane owner would sue the owner of the car who should be covered by the insurance for some property damages and the car insurance might go back and sue tesla to recover funds given how unusual the situation is

deleted 5y ago

[deleted]

rvz 5y ago

It doesn't even stop after hitting the jet. It just keeps going right through it. I don't think you can take this autopilot, and FSD (Fools Self Driving) contraption seriously or remotely ready for customers to use; even for a 'beta'. Since it's functionality doesn't even work as advertised, puts the lives of other drivers on the road at risk and it is completely useless at the most dangerous time to drive, which is at night. For example, as a simple test against another autonomous vehicle with LIDAR at night. [0] The Tesla with FSD failed to stop and just ran over the object. The best part is 100,000 crash dummies have this software installed and seem to be taking the risk in trying it out, even when it malfunctions on the public roads and what we can see here. This contraption needs to be thoroughly investigated. [0] https://twitter.com/TaylorOgan/status/1478802681141645322 https://twitter.com/TaylorOgan/status/1478802681141645322

booboofixer 5y ago
> It doesn't even stop after hitting the jet. It just keeps going right through it. I thought that was the worst part of it too, assuming theres nobody inside the car at the wheel. It keeps going, stops, then moves again and looks to go around whatever obstacle it thinks exists. What if this situation can be replicated on an empty road where the tesla runs over a human being? In the worst case, does the tesla keep trying to drive over the human being and turns a minor injury into a fatality?
misiti3780 5y ago
how do you propose we get to level 4 without a large number of humans collecting the data and providing real-time feedback?
- roughly 5y ago
  Are you talking about the humans inside or outside the car?
  - misiti3780 5y ago
    inside driving the cars

netsharc 5y ago

This car reviewer tried it (with his mom holding the button) and it would've pancaked him too (they tried it twice, once with his mom's motherly instinct letting go of the button): https://youtu.be/kW2W2l2kXFM?t=410 https://youtu.be/kW2W2l2kXFM?t=410

robga 5y ago

Autopilot envy?

roughly 5y ago

I remember part of the report on the Uber that ran the person over was that the Uber’s classifier couldn’t figure out what a person with a bike in the road was, so it didn’t make it into the decision tree. I’m curious if that’s what happened here - Tesla never trained their cars to recognize airplanes, so it just didn’t “see” it.

julianlam 5y ago
Suppose we should just get ready for ReCAPTCHA challenges involving various parts of airplanes?
- roughly 5y ago
  The ever relevant XKCD: https://xkcd.com/1897/ https://xkcd.com/1897/
badwolf 5y ago
The part that baffles me is that there isn't a "There's an object, not sure what it is, but there's something, I should stop/avoid it" instead it seems to be "I don't know what this object is. Oh well, it's probably fine to plow into it"
giarc 5y ago
I wonder if it couldn't "see" the jet since it has an all white underbelly and perhaps it blended into the sky or some hanger behind it.
anikan_vader 5y ago
The Uber self-driving car was actually a Volvo -- its logs show that the Volvo emergency braking system had detected a pedestrian crossing almost 2 seconds before the collision. It would've initiated emergency braking, but Uber had disabled this feature, since, of course, they were testing their own shitty self-driving software. The Uber software chose to take no action, the Uber employee at the wheel was apparently paying no attention, and Uber had disabled the Volvo software, the only agent involved that actually thought it would be a good idea to pay attention and try not to kill a woman with a bike on a dark night. Uber shouldn't be allowed to test their software on public roads in my opinion.

BXLE_1-1-BitIs1 5y ago

Insurance policies can exclude coverage off public roads. If the Tesla and aircraft owner are one and same, the owner may be holding the bag for the damage. The aircraft owner and insurer could sue Tesla, but likely Tesla's terms of use require the car only be on public roads and private driveways.

metamuas 5y ago

Can someone explain to me why Tesla didn't bootstrap itself with cameras/LIDAR, then remove the LIDAR? I mean teaching an AI depth perception and the ability to model objects by vision, without including training data from LIDAR just seems kind of silly.

yunyu 5y ago
How many people would buy a Tesla that costs 3x as much?
mariebks 5y ago
That’s exactly what they did. They used LiDAR to train their vision neural nets to accurately compute the distance of every pixel. https://cleantechnica.com/2020/08/03/tesla-achieved-the-accuracy-of-lidar-with-its-advanced-computer-vision-tech/amp/ https://cleantechnica.com/2020/08/03/tesla-achieved-the-accu...
- mariebks 5y ago
  https://www.teslarati.com/tesla-model-x-lidar-equipment-photos/amp/ https://www.teslarati.com/tesla-model-x-lidar-equipment-phot...

58028641 5y ago

Tesla very clearly states: "Smart Summon is a BETA feature. You must continually monitor the vehicle and its surroundings and stay prepared to take immediate action at any time. It is the driver's responsibility to use Smart Summon safely, responsibly, and as intended." "When using Smart Summon, you must maintain a clear line of sight between you and Model Y and stay prepared to stop the vehicle at any time by releasing the button on the mobile app." https://www.tesla.com/ownersmanual/modely/is_is/GUID-6B9A1AEA-579C-400E-A7A6-E4916BCD5DED.html https://www.tesla.com/ownersmanual/modely/is_is/GUID-6B9A1AE...

twayt 5y ago
If these are the only kind of edge cases where smart summon fails, I'd say it is actually doing pretty good.
- codezero 5y ago
  I saw someone using it on the top of the parking garage at Disney land. It just circled around but I found it fascinating to watch and am looking forward to when it can go down the ramp :)
sillyinseattle 5y ago
How thoughtful of Tesla ;-) One nearly backed into me a few months back. I banged on the car to alert the driver. It stopped. There was no one in it. Owner was summoning her car in a grocery store parking lot. People.
- rich_sasha 5y ago
  Rest assured, if you were maimed, it would be the owner’s responsibility, not Tesla’s, so all is good. Presumably the driver is insured, right? And insurance covers the his nonsense...
  - sorokod 5y ago
    Isn't Tesla in car insurance itself? Interesting how that recursive setup would work out.

mongol 5y ago

Imagine flying with an airline that had a beta version of the autopilot installed in its airplane. The plane crashes and you get maimed. The airline is responsible but the airplane manufacturer is in the clear. It was a beta version, after all.

deleted 5y ago
[deleted]
sorokod 5y ago
Would such an airplane be legally allowed to fly passengers? If so, I'd be going after the authority that made this legal.

kappuchino 5y ago

Question to the experts: Is this kind of a classification or a detection problem ... or both? I remember that visual algorithms for classificaion can get confused by objects or situations they were not trained on, so in the result there were unpredictable outcomes in situations that seemed safe before. Or is it more like a "shape and size of a plane" issue?

salawat 5y ago
It's likely because it's an overhang. System has the same tordid reputation with trailers.

anikan_vader 5y ago

Not colliding with immobile objects at low speed is a solved problem. Tesla's just choosing to go without radar/lidar. This inability to model the 3d world makes me very skeptical that Tesla's self-driving cars are competitive with Google's. Honestly it's embarrassing; if a human behaved this way behind the wheel of a car, I would assume they were high on something stronger than weed.

ragingrobot 5y ago
Why high? Someone else in the thread made a remark about the insinuation of a "phone in front of my face" but in fact this could very easily happen by a sober person just simply not paying attention, and that is fairly common.