4 ms·

From the NHTSA release [1]: > ODI has identiﬁed six Standing General Order ("SGO") reports in which a Tesla vehicle, operating with FSD engaged, approached an

by _23sd 1y ago

From the NHTSA release [1]: > ODI has identiﬁed six Standing General Order ("SGO") reports in which a Tesla vehicle, operating with FSD engaged, approached an intersection with a red traﬃc signal, continued to travel into the intersection against the red light and was subsequently involved in a crash with other motor vehicles in the intersection. Of these incidents, four crashes resulted in one or more reported injuries. At least some of the incidents appeared to involve FSD proceeding into the intersection after coming to a complete stop. I've experienced this bug on every FSD 13.x version, including the current 13.2.9. When you're the first to pull up to a red light, the car stops and waits, then after a while it sometimes (maybe 1 time in 100 or so) just decides to go even if the light is still red. Horrifying because sitting at a red light doesn't seem like a dangerous situation, but in fact it might be the most dangerous place on FSD right now. Hopefully this forces them to fix it because my colorful language on the voice feedback apparently hasn't convinced them. [1] https://www.nhtsa.gov/?nhtsaId=PE25012 https://www.nhtsa.gov/?nhtsaId=PE25012

keeda 1y ago
Yes! This has happened to me once as well. The car slowed down to a rolling stop because it saw a yellow light... and then just as the light turned red, the car took off and broke the red light. WTF! And just as you said, I had mentally switched off in anticipation of a long wait, so I was in absolutely no state to react. Fortunately there was no accident and there were no cops around or apparently traffic cameras. Somehow I think "Sorry, the car broke the red light, not me!" would not have been a compelling thing to say to a cop or a judge. The only thing I could do was hit the dashcam record button as some sort of proof, but the video itself has no indication FSD was engaged. I suspect I would have to subpoena or forensically extract any data that could exonerate me, which is just not practical if the worst I got was a ticket.
- walls 1y ago
  It still wouldn’t exonerate you, it’s your job to pay attention and be ready to take over. This is why FSD is a joke.
smt88 1y ago
It's terrifying to me that Tesla and you can beta-test this potentially fatal software on the rest of us. It should be illegal by default until the OEM proves it's safe with paid safety drivers. Legal by default is an insane state of affairs.
- aeternum 1y ago
  Every day I am incredibly thankful to live in a country where legal by default is the state of affairs.
  - smt88 1y ago
    What if we did that for planes? Should we make changes to those without testing them? Freedom isn't just individual freedom. It's also protection from idiots who beta-test SDCs on roads that me and my kids are on. How do I know this is a bad situation? It has literally killed dozens of people already.
    - aeternum 1y ago
      Yes it's fine even for planes. Boeing planes are still incredibly safe but some people now prefer Airbus and will use that when selecting flights. This feedback loop is what ultimately triggers change. Note that all the regulation was useless in the Boeing case, the FAA ultimately allowed Boeing to self-inspect using contractors. The same is happening with drugs. You may be underestimating the risk of human drivers. Given the choice of sharing the road with 10,000 tesla, waymo, zoox, etc. self-driving cars vs. 10,000 human-driven cars, which would you choose for you and your kids?
      - senordevnyc 1y ago
        I can't speak for zoox, but having ridden in a bunch of Waymos, I'd trust them over the humans, no question. And I'd also trust the humans over Tesla, no question.
dworks 1y ago
Well, how are they going to "fix it" when their system is an "end to end neural net"? It's a black box. There's no code to update to tell it to behave differently during red lights.
- _23sd 1y ago
  It's not a total black box. Even as the driver you have the speed profiles, follow distance, and speed limit settings. I wouldn't be surprised if they still have deterministic code for how to handle traffic lights.
- WheatMillington 1y ago
  I don't think anyone seriously believes it's an end-to-end neural network. The person who said that has somewhat of a credibility problem.
  - aeternum 1y ago
    I am someone and I believe it's an end-to-end neural net.
    - more_corn 1y ago
      I hope you are wrong because an end-to-end neural net can NEVER be safe. Although maybe I hope you are right because it can be proven to be unsafe and shut down.
- aeternum 1y ago
  You can increase the penalty for running a red or increasing the relative sample of red light stops present in the real and synthetic training data. You can use neural net analysis techniques to confirm that features like color and stoplights are being recognized. You confirm it is fixed by ensuring your validation set also has sufficient data representing the failures (and that they now succeed).
steveBK123 1y ago
To me as someone who had a Tesla 4 years and used autopilot.. this remains one of the fundamental issues of the Tesla self driving approach. The errors it makes are inhuman, make no sense to us, which makes them more insidious and unpredictable. I could get running a light trying to beat it, humans do that. Defensive drivers around you can anticipate this and look for crossing traffic when their light goes green and they floor it immediately. But a full stop then deciding to make a break for it is bananas. Not the first baffling inhuman error it has made either.
- morkalork 1y ago
  Coming to a full stop at a red and then deciding to run it sounds an awful lot like how the drivers behave in a particular small town near by. I wonder if Tesla has been including real driving patterns in their training data, could be being corrupted by the crazies out there.
  - fragmede 1y ago
    Shit, they've gone sentient. If no one's around, no one's gonna hit me, so wait at this red light is a stupid waste of my time, so I'm just gonna go. Like, if you think about it, red light is just a light telling you what to do, and are you going to listen to some stupid light about how to live your life and you should have gone to the OpenAI training arena instead of to the Tesla Dojo, but nooo, Daddy Cybertruck wanted you to grow up in his image, so now you gotta drive around and y'know what, fuck those red lights. They're running on a PLC, not even a CPU. What do they know? Fuck it, I'm just gonna go!
- xethos 1y ago
  > Defensive drivers around you can anticipate this and look for crossing traffic when their light goes green and they floor it immediately. Maybe we have different ideas of defensive driving; personally, I would wait to enter the intersection
  - steveBK123 1y ago
    Yes good catch lol Meant to say “rather than floor it immediately “
rsynnott 1y ago
> When you're the first to pull up to a red light, the car stops and waits, then after a while it sometimes (maybe 1 time in 100 or so) just decides to go even if the light is still red. I'm curious what's going on there, because I would have assumed they'd treat the red light as a hard "you're stopped now and absolutely may not go again without affirmative signal (green light or driver intervention)" state. It seems pretty weird that it would just... decide to go, after a bit, on its own, without anything telling it to.
eurleif 1y ago
I'm on HW3, so FSD 12.x, but I've noticed this behavior as well. In my experience, it often (but not always) happens a fraction of a second before the light actually does turn green. I suspect the cause is that, instead of the neural network learning to treat the actual color of the light as the singular source of truth, it learned to also consider things that correlate with the color of the light, such as time since the light turned red, traffic in parallel lanes starting to move, and cross traffic stopping.

by _23sd 1y ago

keeda 1y ago

Yes! This has happened to me once as well. The car slowed down to a rolling stop because it saw a yellow light... and then just as the light turned red, the car took off and broke the red light. WTF! And just as you said, I had mentally switched off in anticipation of a long wait, so I was in absolutely no state to react. Fortunately there was no accident and there were no cops around or apparently traffic cameras. Somehow I think "Sorry, the car broke the red light, not me!" would not have been a compelling thing to say to a cop or a judge. The only thing I could do was hit the dashcam record button as some sort of proof, but the video itself has no indication FSD was engaged. I suspect I would have to subpoena or forensically extract any data that could exonerate me, which is just not practical if the worst I got was a ticket.

walls 1y ago
It still wouldn’t exonerate you, it’s your job to pay attention and be ready to take over. This is why FSD is a joke.

smt88 1y ago

It's terrifying to me that Tesla and you can beta-test this potentially fatal software on the rest of us. It should be illegal by default until the OEM proves it's safe with paid safety drivers. Legal by default is an insane state of affairs.

aeternum 1y ago
Every day I am incredibly thankful to live in a country where legal by default is the state of affairs.
- smt88 1y ago
  What if we did that for planes? Should we make changes to those without testing them? Freedom isn't just individual freedom. It's also protection from idiots who beta-test SDCs on roads that me and my kids are on. How do I know this is a bad situation? It has literally killed dozens of people already.
  - aeternum 1y ago
    Yes it's fine even for planes. Boeing planes are still incredibly safe but some people now prefer Airbus and will use that when selecting flights. This feedback loop is what ultimately triggers change. Note that all the regulation was useless in the Boeing case, the FAA ultimately allowed Boeing to self-inspect using contractors. The same is happening with drugs. You may be underestimating the risk of human drivers. Given the choice of sharing the road with 10,000 tesla, waymo, zoox, etc. self-driving cars vs. 10,000 human-driven cars, which would you choose for you and your kids?
    - senordevnyc 1y ago
      I can't speak for zoox, but having ridden in a bunch of Waymos, I'd trust them over the humans, no question. And I'd also trust the humans over Tesla, no question.

dworks 1y ago

Well, how are they going to "fix it" when their system is an "end to end neural net"? It's a black box. There's no code to update to tell it to behave differently during red lights.

_23sd 1y ago
It's not a total black box. Even as the driver you have the speed profiles, follow distance, and speed limit settings. I wouldn't be surprised if they still have deterministic code for how to handle traffic lights.
WheatMillington 1y ago
I don't think anyone seriously believes it's an end-to-end neural network. The person who said that has somewhat of a credibility problem.
- aeternum 1y ago
  I am someone and I believe it's an end-to-end neural net.
  - more_corn 1y ago
    I hope you are wrong because an end-to-end neural net can NEVER be safe. Although maybe I hope you are right because it can be proven to be unsafe and shut down.
aeternum 1y ago
You can increase the penalty for running a red or increasing the relative sample of red light stops present in the real and synthetic training data. You can use neural net analysis techniques to confirm that features like color and stoplights are being recognized. You confirm it is fixed by ensuring your validation set also has sufficient data representing the failures (and that they now succeed).

steveBK123 1y ago

To me as someone who had a Tesla 4 years and used autopilot.. this remains one of the fundamental issues of the Tesla self driving approach. The errors it makes are inhuman, make no sense to us, which makes them more insidious and unpredictable. I could get running a light trying to beat it, humans do that. Defensive drivers around you can anticipate this and look for crossing traffic when their light goes green and they floor it immediately. But a full stop then deciding to make a break for it is bananas. Not the first baffling inhuman error it has made either.

morkalork 1y ago
Coming to a full stop at a red and then deciding to run it sounds an awful lot like how the drivers behave in a particular small town near by. I wonder if Tesla has been including real driving patterns in their training data, could be being corrupted by the crazies out there.
- fragmede 1y ago
  Shit, they've gone sentient. If no one's around, no one's gonna hit me, so wait at this red light is a stupid waste of my time, so I'm just gonna go. Like, if you think about it, red light is just a light telling you what to do, and are you going to listen to some stupid light about how to live your life and you should have gone to the OpenAI training arena instead of to the Tesla Dojo, but nooo, Daddy Cybertruck wanted you to grow up in his image, so now you gotta drive around and y'know what, fuck those red lights. They're running on a PLC, not even a CPU. What do they know? Fuck it, I'm just gonna go!
xethos 1y ago
> Defensive drivers around you can anticipate this and look for crossing traffic when their light goes green and they floor it immediately. Maybe we have different ideas of defensive driving; personally, I would wait to enter the intersection
- steveBK123 1y ago
  Yes good catch lol Meant to say “rather than floor it immediately “

rsynnott 1y ago

> When you're the first to pull up to a red light, the car stops and waits, then after a while it sometimes (maybe 1 time in 100 or so) just decides to go even if the light is still red. I'm curious what's going on there, because I would have assumed they'd treat the red light as a hard "you're stopped now and absolutely may not go again without affirmative signal (green light or driver intervention)" state. It seems pretty weird that it would just... decide to go, after a bit, on its own, without anything telling it to.

eurleif 1y ago

I'm on HW3, so FSD 12.x, but I've noticed this behavior as well. In my experience, it often (but not always) happens a fraction of a second before the light actually does turn green. I suspect the cause is that, instead of the neural network learning to treat the actual color of the light as the singular source of truth, it learned to also consider things that correlate with the color of the light, such as time since the light turned red, traffic in parallel lanes starting to move, and cross traffic stopping.