5 ms·

Tesla Full Self Driving Beta Causes a Crash

121 points by camjohnson26 5y ago

SQueeeeeL 5y ago
My amateur interpretation of this is that the vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line in the road, which caused it to believe it was seriously out of lane. After verring, the algorithm reoriented, and realized an impact was imminent and diverted off the road. I think what's interesting about this example is that in this case, an alert human driver would not be confused in this way. It shows issues with the particular machine learning approach to automated driving.
- jarolinvargas 5y ago
  The driver intervened and lost control. It looks like Autopilot was going all in on that incoming vehicle.
- rightiousrob 5y ago
  I don't think that a root cause analysis like this is good for anything else than improving the algorithm. It's the same as with face detection. The algorithms are better far superior to humans, but they fail at occasions where a human would not even think twice
- rasz 5y ago
  >vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line not the lights, but the vertical lens flares captured by single front facing center mounted camera cluster - afaik Tesla still has no stereo vision. >the algorithm reoriented more like driver jerked to the right while having a heart attack and seeing itself going into a full frontal collision thanks to move fast and break things mentality
  - catlikesshrimp 5y ago
    Wow, you just slapped me with this idea: - What should be the core tenet of the self driving system for a car? - Why, of course it should be "Move fast and break things"!
- deleted 5y ago
  [deleted]
rvz 5y ago
Oh dear. FSD (Fools Self Driving) Beta once again causing havoc and putting the life of this driver at risk and deciding to hit another car after the driver regaining control of the vehicle very quickly and going off road, especially in the most dangerous time to drive, which is at night. Had the driver not kept their eyes on the road or was sleeping in the car, this system would have crashed into that incoming vehicle. This video just proves that night-vision is non-existent in this 'safety-critical' system. Outrageous.
cblconfederate 5y ago
Thankfully we have no teslas here yet, but i gotta watch out when they arrive. Is it possible to tell if a tesla has self-diving?
- devortel 5y ago
  Not from the exterior. When full self driving beta mode is active, there is a circular, blue steering wheel icon visible at the top of the car's control screen. So unless you're inside the Tesla, you won't be able to tell.
  - bdcravens 5y ago
    I don't think we are far from legislation being required to mandate some sort of indicator. (if not outlawing these features)
    - catlikesshrimp 5y ago
      Yes, please. Just like "Driving instructor, keep your distance", a Tesla should say "Tesla FSD, slow down"
      - rad_gruchalski 5y ago
        Why should I slow down? Get away from it asap!
  - rsynnott 5y ago
    I would not bet on this continuing to be the case, mind you. If regulators allow this sort of thing to continue at all, I'd expect warning lights visible to pedestrians and other drivers will be required.
zardo 5y ago
If you have to be at the ready to take over from the car trying to kill you at all times, wouldn't you rather just drive?
- tshaddox 5y ago
  That depends on the probability (say, per mile) that you’ll need to take over.
  - faeyanpiraat 5y ago
    A lower probability is even worse psychologically imho.
    - tshaddox 5y ago
      That also depends on how low.
      - zardo 5y ago
        It's high enough that the people building it say it's not safe to take your hands off the wheel or your eyes off the road.
    - whateveracct 5y ago
      I guarantee reaction time to self driving is way worse than an active driver in the same situation.
- zaphirplane 5y ago
  Objectively only If you perform statistically better than the automatic system
  - zardo 5y ago
    So far so good. I suppose there are a few acute medical situations that might cause me to fuck up driving on par with what FSD can manage to do.
kelnos 5y ago
Can we change this to link to the original video, at https://www.youtube.com/watch?v=7VhrG-7SBZg https://www.youtube.com/watch?v=7VhrG-7SBZg ? Video description contains details about what happened, which might not be clear from the video snippet. FSD swerved into oncoming traffic, and the driver manually pulled back to the correct lane, but lost control and went off the road.
- pja 5y ago
  That video has been removed by the uploader according to YouTube.
  - deleted 5y ago
    [deleted]
AtlasBarfed 5y ago
Is there corroboration that this actually is self-driving? I suppose we'd need Tesla's cooperation. Is it possible to replay the "AI view" of the same thing? Can any FSD beta users chime in? I will say that two lane highways (aka offset impact at 110mph+ IF/BIG IF they aren't speeding) are pretty damn dangerous and I'd only do self-driving on them with a LOT of repeats. I think I'm getting close to the point of trusting self-driving on divided highways under good conditions, the big caveat being stopped cars/obstacles. It sure doesn't help that every Tesla post on HN has comments that seem like obvious astroturfs. It's like jalopnik in the comments. Transparency is a must. The new NHTSA head doesn't help, we do need someone to hold Tesla's claim to the light, but appointing someone with publicly posted anti-Tesla screeds? That doesn't help anything. Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year and be a legitimate boon to the economy. NHTSA needs a dedicated group of government officials with legit industry and software experience to test, regulate, and analyze incidents. Plus it would be apocalypse to one of my less favorite sectors in the world: the auto insurance industry. NHTSA should have immediate delivery of Self-Driving accidents with all the footage and any recorder information immediately uploaded to government servers with an active software harness to replay the logs. And it's not like this would be occaisionally tested, even with high-quality theoretical self driving, there will be daily accidents to replay from every significant vendor.
- avereveard 5y ago
  > impact at 110mph can we get the "frontal impact == impact at sum of speed" myth die already? it's been thoroughly debunked over and over. the only way it's relevant is if there's a large weight difference between vehicles.
- catlikesshrimp 5y ago
  "Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year..." Launching a self driving car which crashes head on into traffic is not a good start. If they used unmaned Teslas for the live traffic tests they would be risking less lives. /s It is interesting that consumers kind of volunteer to test a potentially lethal product. I say "kind of volunteer" because they are actually paying for the car.
siddarthd2919 5y ago
It looks like auto highbeam assist turned of and it might have caused a weird vision bug
- bdcravens 5y ago
  There's just something I don't like about the word "bug". It suggests a condition of the software, not a condition of the developer. Maybe it's the full phrase "caused a ... bug". As if the bug wasn't present until the condition occurred. Unless that condition is actually altering the commands in the software, the bug was already there. I could even talk about what "weird" means in this context. It's not a poke at you - I heard phrases like "caused a weird bug" all the time in our industry. In generally, and especially in situations involving safety, it's important we not engage in minimization, but call it was it is: a flaw, where conditions in the operating environment weren't properly considered.
  - mirker 5y ago
    Well, given the “flaw” is a machine learning model, who was the developer and how do they relate to the training data?
josephcsible 5y ago
I'm suspicious that this may not have occurred as described for 2 reasons: the recording doesn't show the part of the screen where the autosteer indicator is, and the original uploader took down the video as soon as it got significant attention.
- camjohnson26 5y ago
  The original poster gave more details here: https://teslamotorsclub.com/tmc/threads/fsd-beta-attempts-to-kill-me-causes-accident.249490/ https://teslamotorsclub.com/tmc/threads/fsd-beta-attempts-to...
a-dub 5y ago
that wasn't a crash! the car simply decided that the turn was dangerous and took it upon itself to disrupt the road construction industrial complex by forging a new and safer path.
aaron695 5y ago
This is $TSLAQ (you can see from the higher tweets) It's disappointing HN is part of something so lame or to stupid to know they are part of it. If you have money invested and are part of the hack, perhaps not lame. But most dumb lamers are doing it just to follow. You can see anti-TSLAQ will get flagged/downvoted by the "bots" on HN (This account is shadowbanned, it's hard to use search for examples since they get flagged)
sidibe 5y ago
It doesn't surprise me that it's this particular scenario, I've seen a lot of Youtube videos of them going way over the center line on turns in this kind of road (2 lane rural with 55 mph) even in broad daylight
chapchaps 5y ago
Don’t forget with this sensational “headline” we don’t actually know the truth. We can’t see the driver in the video and we have only his word to go by. Statistically it’s much more likely he’s lying versus FSD causing the accident in this situation. FSD has a long way to go but it’s already demonstrably safer than humans. Regardless, Hope this guy is okay and wasn’t injured in the accident.
- ivraatiems 5y ago
  Sorry, but that isn't true of Tesla's FSD. It's notoriously unsafe. Here's some evidence: https://www.motorbiscuit.com/tesla-beta-testers-say-full-self-driving-mode-is-dangerous/ https://www.motorbiscuit.com/tesla-beta-testers-say-full-sel... https://www.inputmag.com/tech/tesla-recalls-12000-cars-over-dangerous-full-self-driving-beta-error https://www.inputmag.com/tech/tesla-recalls-12000-cars-over-... https://www.roadandtrack.com/news/a35878363/teslas-full-self-driving-beta-is-just-laughably-bad-and-potentially-dangerous/ https://www.roadandtrack.com/news/a35878363/teslas-full-self... https://electrek.co/2021/11/15/tesla-serious-phantom-braking-problem-autopilot/ https://electrek.co/2021/11/15/tesla-serious-phantom-braking... https://www.businessinsider.com/tesla-full-self-driving-beta-videos-show-flaws-wide-release-2021-3 https://www.businessinsider.com/tesla-full-self-driving-beta... (with video!) Or just look on Youtube for videos of people using the FSD, and see how frequently they have to stop it from doing something dangerous.
jondwillis 5y ago
For what it is worth, I want to try FSD beta, and have a safety score of 99, and can’t. You have to jump through so many hoops to even have FSD beta right now, that this is the devil’s deal (i.e. potential manslaughter) you accept by even trying it. Tesla should investigate, and continue to keep FSD in beta with these strong obstacles while it improves. And eventually (arguably), it needs to put up or pay up for level five FSD which it has been selling as vaporware for something like 5 years now. Maybe this is where public and private law needs to step in. But in my opinion, casual outrage over FSD is unwarranted until Tesla rolls out software that is less safe than humans to a lot of its users that will operate it as if it is level 5 regardless of actual performance.
dzhiurgis 5y ago
Spend 10k on FSD tip jar Fail to buy real tyres Or is Tesla’s stability control so bad that it underperforms 2 decade old economy cars?
jaimex2 5y ago
It was fake. Delete the post.
- Borgz 5y ago
  Evidence?
irthomasthomas 5y ago
There are loads of videos of Tesla self driving fails, like this one posted yesterday, where the Tesla tries to hit a pedestrian and prevents the driver from taking control, tries to drive on the train track, and stops for the Wells Fargo sign. https://mobile.twitter.com/TaylorOgan/status/1469404579439824899 https://mobile.twitter.com/TaylorOgan/status/146940457943982...
- rsynnott 5y ago
  Er... Yes, this definitely seems like it'll be ready to use any day now. Bloody hell, people _pay_ for this?
oxplot 5y ago
Flagged this. Some random video with no official investigation and countless speculations. Keep in mind that Tesla will sue your ass for defamation if you post BS video and lie about it. Just ask the Chinese dude who posted about Tesla accelerating by itself into a crash. Turns out he stepped on the pedal himself but of course when someone on twitter posts about a Tesla fuck-up, it’s taken as gospel.
- AbrahamParangi 5y ago
  The degree to which people are willing to jump to the defense of Tesla self-driving is truly incredible. The experience of “your Tesla tries to do something INSANE” is shared by literally every Tesla driver I know.
  - oxplot 5y ago
    Yes, I’ll happily jump to Tesla’s defense (not that they need it) when baseless crap is thrown around, because for every 1 of me, there are 100 who can’t wait to see Tesla crash and burn.

121 points by camjohnson26 5y ago

My amateur interpretation of this is that the vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line in the road, which caused it to believe it was seriously out of lane. After verring, the algorithm reoriented, and realized an impact was imminent and diverted off the road. I think what's interesting about this example is that in this case, an alert human driver would not be confused in this way. It shows issues with the particular machine learning approach to automated driving.

jarolinvargas 5y ago
The driver intervened and lost control. It looks like Autopilot was going all in on that incoming vehicle.
rightiousrob 5y ago
I don't think that a root cause analysis like this is good for anything else than improving the algorithm. It's the same as with face detection. The algorithms are better far superior to humans, but they fail at occasions where a human would not even think twice
rasz 5y ago
>vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line not the lights, but the vertical lens flares captured by single front facing center mounted camera cluster - afaik Tesla still has no stereo vision. >the algorithm reoriented more like driver jerked to the right while having a heart attack and seeing itself going into a full frontal collision thanks to move fast and break things mentality
- catlikesshrimp 5y ago
  Wow, you just slapped me with this idea: - What should be the core tenet of the self driving system for a car? - Why, of course it should be "Move fast and break things"!
deleted 5y ago
[deleted]

rvz 5y ago

Oh dear. FSD (Fools Self Driving) Beta once again causing havoc and putting the life of this driver at risk and deciding to hit another car after the driver regaining control of the vehicle very quickly and going off road, especially in the most dangerous time to drive, which is at night. Had the driver not kept their eyes on the road or was sleeping in the car, this system would have crashed into that incoming vehicle. This video just proves that night-vision is non-existent in this 'safety-critical' system. Outrageous.

cblconfederate 5y ago

Thankfully we have no teslas here yet, but i gotta watch out when they arrive. Is it possible to tell if a tesla has self-diving?

devortel 5y ago
Not from the exterior. When full self driving beta mode is active, there is a circular, blue steering wheel icon visible at the top of the car's control screen. So unless you're inside the Tesla, you won't be able to tell.
- bdcravens 5y ago
  I don't think we are far from legislation being required to mandate some sort of indicator. (if not outlawing these features)
  - catlikesshrimp 5y ago
    Yes, please. Just like "Driving instructor, keep your distance", a Tesla should say "Tesla FSD, slow down"
    - rad_gruchalski 5y ago
      Why should I slow down? Get away from it asap!
- rsynnott 5y ago
  I would not bet on this continuing to be the case, mind you. If regulators allow this sort of thing to continue at all, I'd expect warning lights visible to pedestrians and other drivers will be required.

zardo 5y ago

If you have to be at the ready to take over from the car trying to kill you at all times, wouldn't you rather just drive?

tshaddox 5y ago
That depends on the probability (say, per mile) that you’ll need to take over.
- faeyanpiraat 5y ago
  A lower probability is even worse psychologically imho.
  - tshaddox 5y ago
    That also depends on how low.
    - zardo 5y ago
      It's high enough that the people building it say it's not safe to take your hands off the wheel or your eyes off the road.
  - whateveracct 5y ago
    I guarantee reaction time to self driving is way worse than an active driver in the same situation.
zaphirplane 5y ago
Objectively only If you perform statistically better than the automatic system
- zardo 5y ago
  So far so good. I suppose there are a few acute medical situations that might cause me to fuck up driving on par with what FSD can manage to do.

kelnos 5y ago

Can we change this to link to the original video, at https://www.youtube.com/watch?v=7VhrG-7SBZg https://www.youtube.com/watch?v=7VhrG-7SBZg ? Video description contains details about what happened, which might not be clear from the video snippet. FSD swerved into oncoming traffic, and the driver manually pulled back to the correct lane, but lost control and went off the road.

pja 5y ago
That video has been removed by the uploader according to YouTube.
- deleted 5y ago
  [deleted]

AtlasBarfed 5y ago

Is there corroboration that this actually is self-driving? I suppose we'd need Tesla's cooperation. Is it possible to replay the "AI view" of the same thing? Can any FSD beta users chime in? I will say that two lane highways (aka offset impact at 110mph+ IF/BIG IF they aren't speeding) are pretty damn dangerous and I'd only do self-driving on them with a LOT of repeats. I think I'm getting close to the point of trusting self-driving on divided highways under good conditions, the big caveat being stopped cars/obstacles. It sure doesn't help that every Tesla post on HN has comments that seem like obvious astroturfs. It's like jalopnik in the comments. Transparency is a must. The new NHTSA head doesn't help, we do need someone to hold Tesla's claim to the light, but appointing someone with publicly posted anti-Tesla screeds? That doesn't help anything. Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year and be a legitimate boon to the economy. NHTSA needs a dedicated group of government officials with legit industry and software experience to test, regulate, and analyze incidents. Plus it would be apocalypse to one of my less favorite sectors in the world: the auto insurance industry. NHTSA should have immediate delivery of Self-Driving accidents with all the footage and any recorder information immediately uploaded to government servers with an active software harness to replay the logs. And it's not like this would be occaisionally tested, even with high-quality theoretical self driving, there will be daily accidents to replay from every significant vendor.

avereveard 5y ago
> impact at 110mph can we get the "frontal impact == impact at sum of speed" myth die already? it's been thoroughly debunked over and over. the only way it's relevant is if there's a large weight difference between vehicles.
catlikesshrimp 5y ago
"Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year..." Launching a self driving car which crashes head on into traffic is not a good start. If they used unmaned Teslas for the live traffic tests they would be risking less lives. /s It is interesting that consumers kind of volunteer to test a potentially lethal product. I say "kind of volunteer" because they are actually paying for the car.

siddarthd2919 5y ago

It looks like auto highbeam assist turned of and it might have caused a weird vision bug

bdcravens 5y ago
There's just something I don't like about the word "bug". It suggests a condition of the software, not a condition of the developer. Maybe it's the full phrase "caused a ... bug". As if the bug wasn't present until the condition occurred. Unless that condition is actually altering the commands in the software, the bug was already there. I could even talk about what "weird" means in this context. It's not a poke at you - I heard phrases like "caused a weird bug" all the time in our industry. In generally, and especially in situations involving safety, it's important we not engage in minimization, but call it was it is: a flaw, where conditions in the operating environment weren't properly considered.
- mirker 5y ago
  Well, given the “flaw” is a machine learning model, who was the developer and how do they relate to the training data?

josephcsible 5y ago

I'm suspicious that this may not have occurred as described for 2 reasons: the recording doesn't show the part of the screen where the autosteer indicator is, and the original uploader took down the video as soon as it got significant attention.

camjohnson26 5y ago
The original poster gave more details here: https://teslamotorsclub.com/tmc/threads/fsd-beta-attempts-to-kill-me-causes-accident.249490/ https://teslamotorsclub.com/tmc/threads/fsd-beta-attempts-to...

a-dub 5y ago

that wasn't a crash! the car simply decided that the turn was dangerous and took it upon itself to disrupt the road construction industrial complex by forging a new and safer path.

aaron695 5y ago

This is $TSLAQ (you can see from the higher tweets) It's disappointing HN is part of something so lame or to stupid to know they are part of it. If you have money invested and are part of the hack, perhaps not lame. But most dumb lamers are doing it just to follow. You can see anti-TSLAQ will get flagged/downvoted by the "bots" on HN (This account is shadowbanned, it's hard to use search for examples since they get flagged)

sidibe 5y ago

It doesn't surprise me that it's this particular scenario, I've seen a lot of Youtube videos of them going way over the center line on turns in this kind of road (2 lane rural with 55 mph) even in broad daylight

chapchaps 5y ago

Don’t forget with this sensational “headline” we don’t actually know the truth. We can’t see the driver in the video and we have only his word to go by. Statistically it’s much more likely he’s lying versus FSD causing the accident in this situation. FSD has a long way to go but it’s already demonstrably safer than humans. Regardless, Hope this guy is okay and wasn’t injured in the accident.

ivraatiems 5y ago
Sorry, but that isn't true of Tesla's FSD. It's notoriously unsafe. Here's some evidence: https://www.motorbiscuit.com/tesla-beta-testers-say-full-self-driving-mode-is-dangerous/ https://www.motorbiscuit.com/tesla-beta-testers-say-full-sel... https://www.inputmag.com/tech/tesla-recalls-12000-cars-over-dangerous-full-self-driving-beta-error https://www.inputmag.com/tech/tesla-recalls-12000-cars-over-... https://www.roadandtrack.com/news/a35878363/teslas-full-self-driving-beta-is-just-laughably-bad-and-potentially-dangerous/ https://www.roadandtrack.com/news/a35878363/teslas-full-self... https://electrek.co/2021/11/15/tesla-serious-phantom-braking-problem-autopilot/ https://electrek.co/2021/11/15/tesla-serious-phantom-braking... https://www.businessinsider.com/tesla-full-self-driving-beta-videos-show-flaws-wide-release-2021-3 https://www.businessinsider.com/tesla-full-self-driving-beta... (with video!) Or just look on Youtube for videos of people using the FSD, and see how frequently they have to stop it from doing something dangerous.

jondwillis 5y ago

For what it is worth, I want to try FSD beta, and have a safety score of 99, and can’t. You have to jump through so many hoops to even have FSD beta right now, that this is the devil’s deal (i.e. potential manslaughter) you accept by even trying it. Tesla should investigate, and continue to keep FSD in beta with these strong obstacles while it improves. And eventually (arguably), it needs to put up or pay up for level five FSD which it has been selling as vaporware for something like 5 years now. Maybe this is where public and private law needs to step in. But in my opinion, casual outrage over FSD is unwarranted until Tesla rolls out software that is less safe than humans to a lot of its users that will operate it as if it is level 5 regardless of actual performance.

dzhiurgis 5y ago

Spend 10k on FSD tip jar Fail to buy real tyres Or is Tesla’s stability control so bad that it underperforms 2 decade old economy cars?

jaimex2 5y ago

It was fake. Delete the post.

Borgz 5y ago
Evidence?

irthomasthomas 5y ago

There are loads of videos of Tesla self driving fails, like this one posted yesterday, where the Tesla tries to hit a pedestrian and prevents the driver from taking control, tries to drive on the train track, and stops for the Wells Fargo sign. https://mobile.twitter.com/TaylorOgan/status/1469404579439824899 https://mobile.twitter.com/TaylorOgan/status/146940457943982...

rsynnott 5y ago
Er... Yes, this definitely seems like it'll be ready to use any day now. Bloody hell, people _pay_ for this?

oxplot 5y ago

Flagged this. Some random video with no official investigation and countless speculations. Keep in mind that Tesla will sue your ass for defamation if you post BS video and lie about it. Just ask the Chinese dude who posted about Tesla accelerating by itself into a crash. Turns out he stepped on the pedal himself but of course when someone on twitter posts about a Tesla fuck-up, it’s taken as gospel.

AbrahamParangi 5y ago
The degree to which people are willing to jump to the defense of Tesla self-driving is truly incredible. The experience of “your Tesla tries to do something INSANE” is shared by literally every Tesla driver I know.
- oxplot 5y ago
  Yes, I’ll happily jump to Tesla’s defense (not that they need it) when baseless crap is thrown around, because for every 1 of me, there are 100 who can’t wait to see Tesla crash and burn.