5 ms·

I don’t understand why self-driving systems don’t have any kind of fallback protections. Like however hard full self driving is to get right, surely a hard rule

by thorum 2y ago

I don’t understand why self-driving systems don’t have any kind of fallback protections. Like however hard full self driving is to get right, surely a hard rule for “don’t drive at full speed into solid objects, no matter how good an idea it seems at the time to the AI” is not that hard to implement?

dheera 2y ago
That video also seems like they were driving insanely fast on a foggy day. Quite possibly setting the speed offset way higher than the speed limit. A human could have easily not seen the train as well at that speed. During fog your speed limit should be determined by your braking distance and visibility. Period. I wonder whether the autopilot would have performed better if travelling at a more reasonable speed.
- robryan 2y ago
  You can see the flashing light from way back. You would also probably hear the train.
- GuB-42 2y ago
  He doesn't seem to drive that fast, he drove at a constant speed only to stop at the very last second. Had he been driving too fast, he would have been right into the train and probably not here to post on Twitter. The car didn't slow down even as the train was clearly visible. An attentive human would have not only seen the train but also the very obvious flashing lights and have absolutely not problem stopping in time. EDIT: From the link, it seems FSD was set by the car to about 61~63 MPH on a 55 MPH road, which is the speed most people drive on ( https://teslamotorsclub.com/tmc/threads/ap-fsd-related-crashes.264798/page-9#post-8264597 https://teslamotorsclub.com/tmc/threads/ap-fsd-related-crash... ). Considering the conditions, it is a little fast, but not unreasonably so. It took about 5 seconds from seeing the train to the crash, most of it at constant speed, so, a distance of about 140m. At that speed, stopping distance is about 100m, enough to stop in time. And that's for the unlit train, lights can be seen from much further as they are all visible at the start of the video.
jcranmer 2y ago
My understanding is that it's actually quite difficult to implement a rule for "don't drive at full speed into solid objects" because solid objects are somewhat difficult to reliably identify.
- TaylorAlexander 2y ago
  Actually Tesla has solved the identification problem. “If we collide with it, it was solid” Has been working flawlessly for them.
kajecounterhack 2y ago
Camera only perception systems don't have any indication of what is solid vs not. This is why Cruise and Waymo use LiDAR. They have exactly what you're talking about. Even then it's hard -- consider soft vegetation or trash. Furthermore: tailgater collisions are a thing. You can't just stop on a dime, especially on fast roads (even just 45mph roads).
- mcintyre1994 2y ago
  I’m always amazed when people talk about using Tesla self driving features. Every time I get in a Tesla the driver has the screen showing this little block version of how it perceives what’s around it. It always misses stuff, cars jump in and out, things randomly vanish. Maybe that UI is buggy or doesn’t really show what the self driving would use, but I can’t imagine seeing that and then wanting to let the car drive itself.
  - jh00ker 2y ago
    There's a probabilistic nature to how the image sensor data is interpreted. The neural network processes data to classify objects and predict what each object is. It doesn't always have a perfect static representation of each object. It's constantly updating its understanding based on the image sensor data. That jiggly jitter you see is basically the system refining its prediction in real-time. Or at least that's how I understand it...
    - TacticalCoder 2y ago
      > That jiggly jitter you see is basically the system refining its prediction in real-time. GP is saying that a "car that jumps in and out" and "things that randomly vanish" do not look very refined. Just like missing a freaking moving train doesn't look very refined.
    - AlotOfReading 2y ago
      Other companies have put quite a lot of effort into perception stability because it has a large effect on the downstream quality. It's hard to estimate higher order derivatives like velocity and acceleration well if your position estimates are unstable.
FourHand451 2y ago
I think that's a good idea in theory, but it still relies on the car knowing when it's looking at a solid object with 100% accuracy.
resolutebat 2y ago
Tesla uses only cameras for FSD, not lidar, so it can't tell what's a solid object and what's not. Driving in foggy conditions where everything is fuzzy to the camera isn't helping. I'm surprised FSD stayed engaged, and I'm also surprised the driver thought it was a good idea to use FSD in this weather.
- groby_b 2y ago
  It turns out that if you loudly proclaim something is "Fully Self Driving", with a tiny asterisk, people assume it's fully self driving and engage it in all sorts of places where they shouldn't. And that fault lies squarely with marketing, and much less with individual drivers.
- samegene321 2y ago
  These types of comments are unhelpful. The expectations on the driver are getting vaguer and vaguer. Can't drive in damp weather? Yet it's Full Self Driving? Tesla wants to market it as powerful AI but leave all responsibility with the driver, don't encourage that narrative.
  - resolutebat 2y ago
    I'm not a fan of Elon's branding either, but if you've ever actually driven a Tesla, it's very obvious very quickly that "FSD" is not fully autonomous driving, and Tesla itself festoons "Beta" all over the place. So, yes, the driver deserves some responsibility for relying on it in manifestly unsuitable conditions.
    - FeloniousHam 2y ago
      100%. Whatever the actual limits, flaws, and failings of FSD, the outsize criticism of the branding is crazymaking. (as I've commented in many of these threads) no one who actually uses the product for even a short period of time is deluded into thinking it is fully autonomous by the labeling. My pet theory for some of these dramatic fails: the driver is not risk-averse, and knowingly uses FSD at some limit "to see what happens" (I know I do it from time to time, though not in anything truly risky).
binarymax 2y ago
Here’s the problem: Tesla only uses camera input - no lidar or other sensor tech. So it’s a visual model ensemble that needs to decide what a solid object is. Models don’t reason. They classify and predict. If the prediction is wrong, explaining why is pretty hard, or maybe impossible. So you just need to iterate over and over on the data to try and fix it. It might, and it might not.
deleted 2y ago
[deleted]
londons_explore 2y ago
I think the problem is those hard rules end up causing problems in more cases than they solve problems. Eg. Driving into an apparently 'solid' cloud of smoke from the truck in front might be a good idea.
pwagland 2y ago
The biggest challenge is identifying said solid object. It is only within the last five years that cars with radar have been able to detect stationary objects with some level of reliability, prior to that, if it wasn't moving, there is a good chance that your Adaptive Cruise Control wouldn't detect it, as otherwise they had a tendency to stop for the bin that is on the side of the road, or for parked cars. A train is a particularly challenging problem, since it is moving at high speed, even with two cameras it is card to really figure out how far away it is. Just because it is easy for humans, doesn't mean it is easy for computers, and vice-versa.
- jonathankoren 2y ago
  If only there was some sort of technology that could directly measure the distance to objects. Perhaps as some sort of point cloud. Guess we'll never know.
LokiSnake 2y ago
It's possible/likely that if such a safeguard is implemented, the self-driving systems would be unusable as it'd trip the safeguard either way too frequently or at unacceptable times.
bagels 2y ago
The fallback is the driver doesn't let it drive at 60 mph up to 50 feet from a train blocking the road.
kelnos 2y ago
Obviously they have that. The issue is that they don't recognize solid, stationary objects all the time. Which you'd think would be easy, but turns out it isn't.

by thorum 2y ago

dheera 2y ago

That video also seems like they were driving insanely fast on a foggy day. Quite possibly setting the speed offset way higher than the speed limit. A human could have easily not seen the train as well at that speed. During fog your speed limit should be determined by your braking distance and visibility. Period. I wonder whether the autopilot would have performed better if travelling at a more reasonable speed.

robryan 2y ago
You can see the flashing light from way back. You would also probably hear the train.
GuB-42 2y ago
He doesn't seem to drive that fast, he drove at a constant speed only to stop at the very last second. Had he been driving too fast, he would have been right into the train and probably not here to post on Twitter. The car didn't slow down even as the train was clearly visible. An attentive human would have not only seen the train but also the very obvious flashing lights and have absolutely not problem stopping in time. EDIT: From the link, it seems FSD was set by the car to about 61~63 MPH on a 55 MPH road, which is the speed most people drive on ( https://teslamotorsclub.com/tmc/threads/ap-fsd-related-crashes.264798/page-9#post-8264597 https://teslamotorsclub.com/tmc/threads/ap-fsd-related-crash... ). Considering the conditions, it is a little fast, but not unreasonably so. It took about 5 seconds from seeing the train to the crash, most of it at constant speed, so, a distance of about 140m. At that speed, stopping distance is about 100m, enough to stop in time. And that's for the unlit train, lights can be seen from much further as they are all visible at the start of the video.

jcranmer 2y ago

My understanding is that it's actually quite difficult to implement a rule for "don't drive at full speed into solid objects" because solid objects are somewhat difficult to reliably identify.

TaylorAlexander 2y ago
Actually Tesla has solved the identification problem. “If we collide with it, it was solid” Has been working flawlessly for them.

kajecounterhack 2y ago

Camera only perception systems don't have any indication of what is solid vs not. This is why Cruise and Waymo use LiDAR. They have exactly what you're talking about. Even then it's hard -- consider soft vegetation or trash. Furthermore: tailgater collisions are a thing. You can't just stop on a dime, especially on fast roads (even just 45mph roads).

mcintyre1994 2y ago
I’m always amazed when people talk about using Tesla self driving features. Every time I get in a Tesla the driver has the screen showing this little block version of how it perceives what’s around it. It always misses stuff, cars jump in and out, things randomly vanish. Maybe that UI is buggy or doesn’t really show what the self driving would use, but I can’t imagine seeing that and then wanting to let the car drive itself.
- jh00ker 2y ago
  There's a probabilistic nature to how the image sensor data is interpreted. The neural network processes data to classify objects and predict what each object is. It doesn't always have a perfect static representation of each object. It's constantly updating its understanding based on the image sensor data. That jiggly jitter you see is basically the system refining its prediction in real-time. Or at least that's how I understand it...
  - TacticalCoder 2y ago
    > That jiggly jitter you see is basically the system refining its prediction in real-time. GP is saying that a "car that jumps in and out" and "things that randomly vanish" do not look very refined. Just like missing a freaking moving train doesn't look very refined.
  - AlotOfReading 2y ago
    Other companies have put quite a lot of effort into perception stability because it has a large effect on the downstream quality. It's hard to estimate higher order derivatives like velocity and acceleration well if your position estimates are unstable.

FourHand451 2y ago

I think that's a good idea in theory, but it still relies on the car knowing when it's looking at a solid object with 100% accuracy.

resolutebat 2y ago

Tesla uses only cameras for FSD, not lidar, so it can't tell what's a solid object and what's not. Driving in foggy conditions where everything is fuzzy to the camera isn't helping. I'm surprised FSD stayed engaged, and I'm also surprised the driver thought it was a good idea to use FSD in this weather.

groby_b 2y ago
It turns out that if you loudly proclaim something is "Fully Self Driving", with a tiny asterisk, people assume it's fully self driving and engage it in all sorts of places where they shouldn't. And that fault lies squarely with marketing, and much less with individual drivers.
samegene321 2y ago
These types of comments are unhelpful. The expectations on the driver are getting vaguer and vaguer. Can't drive in damp weather? Yet it's Full Self Driving? Tesla wants to market it as powerful AI but leave all responsibility with the driver, don't encourage that narrative.
- resolutebat 2y ago
  I'm not a fan of Elon's branding either, but if you've ever actually driven a Tesla, it's very obvious very quickly that "FSD" is not fully autonomous driving, and Tesla itself festoons "Beta" all over the place. So, yes, the driver deserves some responsibility for relying on it in manifestly unsuitable conditions.
  - FeloniousHam 2y ago
    100%. Whatever the actual limits, flaws, and failings of FSD, the outsize criticism of the branding is crazymaking. (as I've commented in many of these threads) no one who actually uses the product for even a short period of time is deluded into thinking it is fully autonomous by the labeling. My pet theory for some of these dramatic fails: the driver is not risk-averse, and knowingly uses FSD at some limit "to see what happens" (I know I do it from time to time, though not in anything truly risky).

binarymax 2y ago

Here’s the problem: Tesla only uses camera input - no lidar or other sensor tech. So it’s a visual model ensemble that needs to decide what a solid object is. Models don’t reason. They classify and predict. If the prediction is wrong, explaining why is pretty hard, or maybe impossible. So you just need to iterate over and over on the data to try and fix it. It might, and it might not.

deleted 2y ago

[deleted]

londons_explore 2y ago

I think the problem is those hard rules end up causing problems in more cases than they solve problems. Eg. Driving into an apparently 'solid' cloud of smoke from the truck in front might be a good idea.

pwagland 2y ago

The biggest challenge is identifying said solid object. It is only within the last five years that cars with radar have been able to detect stationary objects with some level of reliability, prior to that, if it wasn't moving, there is a good chance that your Adaptive Cruise Control wouldn't detect it, as otherwise they had a tendency to stop for the bin that is on the side of the road, or for parked cars. A train is a particularly challenging problem, since it is moving at high speed, even with two cameras it is card to really figure out how far away it is. Just because it is easy for humans, doesn't mean it is easy for computers, and vice-versa.

jonathankoren 2y ago
If only there was some sort of technology that could directly measure the distance to objects. Perhaps as some sort of point cloud. Guess we'll never know.

LokiSnake 2y ago

It's possible/likely that if such a safeguard is implemented, the self-driving systems would be unusable as it'd trip the safeguard either way too frequently or at unacceptable times.

bagels 2y ago

The fallback is the driver doesn't let it drive at 60 mph up to 50 feet from a train blocking the road.

kelnos 2y ago

Obviously they have that. The issue is that they don't recognize solid, stationary objects all the time. Which you'd think would be easy, but turns out it isn't.