4 ms·

What baffles me is that the car doesn’t have any trouble plowing right into things which are visible in the GUI. It looks like it sees the cyclist and goes righ

What baffles me is that the car doesn’t have any trouble plowing right into things which are visible in the GUI. It looks like it sees the cyclist and goes right ahead into them. Anybody have an explanation for this behavior?

judge2020 5y ago
Tesla hired a bunch of vision people, and to their credit, it detects pretty much everything in its FOV all the time. But these results show that vision AI isn't driving AI and the hard part about driving isn't seeing things, it's predicting their future and handling the situation.
nexuist 5y ago
The pathfinding algorithm is separate from the object detection neural network. It's possible for it to incorrectly identify paths going through solid objects, however, I would hope they have dumb code to detect that the path is invalid and throw it out when that happens. Maybe they don't? Part of the difficulty of trying to have reasonable discussions about FSD is that so much of it is black-boxy nobody really knows what's going on. I don't care if Tesla wants to keep its ML models proprietary, but I feel like we deserve to know how it uses those models to make driving decisions on public roads. And not just Tesla either; every self driving company should be required to open source their "dumb" code so that we understand them better. The competitive advantage comes from the ML models anyways, so it wouldn't be anti competitive to regulate that disclosure.
- KennyBlanken 5y ago
  "Don't drive a path through an object" seems like such a core function that a video like this is proof FSD is intrinsically broken.
- g_p 5y ago
  The most recent proposed legal framework in the UK for self driving vehicles (summary at https://www.bbc.com/news/technology-60126014 https://www.bbc.com/news/technology-60126014) recommends that legislation be passed so that "data to understand fault and liability following a collision must be accessible; [There are] sanctions for carmakers who fail to reveal how their systems work" The proposals are very detailed but do seem to be addressing a number of areas around how autonomy is marketed, and how to ensure vehicles are safe and that vehicle makers are making available material information on the inner workings of their software.
- leobg 5y ago
  That’s the fundamental problem with machine learning. You get no control over what the machine has actually learned. It’s like GPT-3. In 9 cases out of ten, it responds in a way that lets you think the model has “grokked” facts of the world, and a human like sense of logic. In truth, it hasn’t. It’s just pretending. It’s taking the easy path by parroting back what it has seen before. They say that ML requires a re-thinking on the part of the developer. Whereas with traditional programming it would be his job to “structure the problem and the path to its solution”, in ML, he should hold back his human notions and let the network discover its own structure. To use an analogy: Whereas we teach our children concepts step by step - you and me, identifying objects, gasping objects, referring to objects, etc., all the way up in complexity to writing thoughtful comments on HN - the neural network is to be bombarded by unfiltered real world data. And the hope is that, if you just do this long enough, the AI will come up with the important underlying concepts by itself. That is to say, for GPT-3, the idea that words are really describing a real world out there, with objects that have certain properties, and that claims may be true or false. Or, for Tesla, that the car is a physical object in space, and that it can collide with other objects. This is very unintuitive. It might be the right approach. But if I was building a self driving car, I would want to build that system block by block. Learn the basics. And only proceed to the next level of complexity once I have verified that the previous step has been properly learned. 1. Let the car learn to predict it’s own acceleration, steering and braking. 2. Repeat on different road surfaces. 3. Add other objects to avoid. 4. Train ability to tell objects from non-obstacles (fog, raindrop on camera) 5. Train lane markings, traffic lights on empty roads. ... X. Throw real world data at it. Now you can hope (and validate) that the model will have the foundation to make sense of it. Like a kid after Highschool. It will still make dumb mistakes. But at least you can talk about them and understand why they happened and take corrective action. Because, while you do not know the other’s mind, you do speak the same language. With GPT-3 and with FSD, we just have no idea.
draw_down 5y ago
banana_giraffe 5y ago
Not just the GUI, the collision avoidance assist alarm was going off.
mustachionut 5y ago
This cyclist went on to murder three people. The Tesla was simply choosing the better of two bad options to minimize fatality.
- ape4 5y ago
  Actually they went on to sue Tesla, saving many lives
qbasic_forever 5y ago
It's telling if Tesla doesn't have an explanation either. Remember when they had early autopilot accidents/fatalities there was an almost instantaneous response from Tesla--even Elon himself tweeting out detailed root cause analysis of accidents that would show the safety driver being impaired, the system working as intended, the pedestrian being impaired, etc. So if an incident like this happens and we don't hear anything from Tesla it tells me there's a far, far bigger problem here with their failure to collect diagnostics, or failure to analyze these problems. Ideally we should see a hyper detailed root cause analysis even going into the vision model inputs in this exact case and exactly how they failed to classify and detect this cyclist, and most importantly how the beta software is being improved to fix issues like this. Silence is deafening here.
stefan_ 5y ago
The GUI is the biggest indictment of it all. Have you seen that line of the trajectory planner? It's spazzing out all the time, even if the car ends up going the right way. Their system literally has no concept of planning and is permanently reevaluating, hence why it goes pedal to the metal hard steering into the cyclist at a frames notice.
heavyset_go 5y ago
> It looks like it sees the cyclist and goes right ahead into them. Anybody have an explanation for this behavior? Its sensitivity was probably lowered because it would brake "too often" for a comfortable consumer product if it was sufficiently cautious enough to avoid hitting pedestrians. This is what happened with the Uber self-driving vehicle that struck and killed a pedestrian[1]. The system detected the pedestrian as an obstruction, but Uber had programmed the system to ignore it because of "phantom braking". Instead of braking, it proceeded to strike and kill the pedestrian. [1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg

judge2020 5y ago

Tesla hired a bunch of vision people, and to their credit, it detects pretty much everything in its FOV all the time. But these results show that vision AI isn't driving AI and the hard part about driving isn't seeing things, it's predicting their future and handling the situation.

nexuist 5y ago

The pathfinding algorithm is separate from the object detection neural network. It's possible for it to incorrectly identify paths going through solid objects, however, I would hope they have dumb code to detect that the path is invalid and throw it out when that happens. Maybe they don't? Part of the difficulty of trying to have reasonable discussions about FSD is that so much of it is black-boxy nobody really knows what's going on. I don't care if Tesla wants to keep its ML models proprietary, but I feel like we deserve to know how it uses those models to make driving decisions on public roads. And not just Tesla either; every self driving company should be required to open source their "dumb" code so that we understand them better. The competitive advantage comes from the ML models anyways, so it wouldn't be anti competitive to regulate that disclosure.

KennyBlanken 5y ago
"Don't drive a path through an object" seems like such a core function that a video like this is proof FSD is intrinsically broken.
g_p 5y ago
The most recent proposed legal framework in the UK for self driving vehicles (summary at https://www.bbc.com/news/technology-60126014 https://www.bbc.com/news/technology-60126014) recommends that legislation be passed so that "data to understand fault and liability following a collision must be accessible; [There are] sanctions for carmakers who fail to reveal how their systems work" The proposals are very detailed but do seem to be addressing a number of areas around how autonomy is marketed, and how to ensure vehicles are safe and that vehicle makers are making available material information on the inner workings of their software.
leobg 5y ago
That’s the fundamental problem with machine learning. You get no control over what the machine has actually learned. It’s like GPT-3. In 9 cases out of ten, it responds in a way that lets you think the model has “grokked” facts of the world, and a human like sense of logic. In truth, it hasn’t. It’s just pretending. It’s taking the easy path by parroting back what it has seen before. They say that ML requires a re-thinking on the part of the developer. Whereas with traditional programming it would be his job to “structure the problem and the path to its solution”, in ML, he should hold back his human notions and let the network discover its own structure. To use an analogy: Whereas we teach our children concepts step by step - you and me, identifying objects, gasping objects, referring to objects, etc., all the way up in complexity to writing thoughtful comments on HN - the neural network is to be bombarded by unfiltered real world data. And the hope is that, if you just do this long enough, the AI will come up with the important underlying concepts by itself. That is to say, for GPT-3, the idea that words are really describing a real world out there, with objects that have certain properties, and that claims may be true or false. Or, for Tesla, that the car is a physical object in space, and that it can collide with other objects. This is very unintuitive. It might be the right approach. But if I was building a self driving car, I would want to build that system block by block. Learn the basics. And only proceed to the next level of complexity once I have verified that the previous step has been properly learned. 1. Let the car learn to predict it’s own acceleration, steering and braking. 2. Repeat on different road surfaces. 3. Add other objects to avoid. 4. Train ability to tell objects from non-obstacles (fog, raindrop on camera) 5. Train lane markings, traffic lights on empty roads. ... X. Throw real world data at it. Now you can hope (and validate) that the model will have the foundation to make sense of it. Like a kid after Highschool. It will still make dumb mistakes. But at least you can talk about them and understand why they happened and take corrective action. Because, while you do not know the other’s mind, you do speak the same language. With GPT-3 and with FSD, we just have no idea.

draw_down 5y ago

banana_giraffe 5y ago

Not just the GUI, the collision avoidance assist alarm was going off.

mustachionut 5y ago

This cyclist went on to murder three people. The Tesla was simply choosing the better of two bad options to minimize fatality.

ape4 5y ago
Actually they went on to sue Tesla, saving many lives

qbasic_forever 5y ago

It's telling if Tesla doesn't have an explanation either. Remember when they had early autopilot accidents/fatalities there was an almost instantaneous response from Tesla--even Elon himself tweeting out detailed root cause analysis of accidents that would show the safety driver being impaired, the system working as intended, the pedestrian being impaired, etc. So if an incident like this happens and we don't hear anything from Tesla it tells me there's a far, far bigger problem here with their failure to collect diagnostics, or failure to analyze these problems. Ideally we should see a hyper detailed root cause analysis even going into the vision model inputs in this exact case and exactly how they failed to classify and detect this cyclist, and most importantly how the beta software is being improved to fix issues like this. Silence is deafening here.

stefan_ 5y ago

The GUI is the biggest indictment of it all. Have you seen that line of the trajectory planner? It's spazzing out all the time, even if the car ends up going the right way. Their system literally has no concept of planning and is permanently reevaluating, hence why it goes pedal to the metal hard steering into the cyclist at a frames notice.

heavyset_go 5y ago

> It looks like it sees the cyclist and goes right ahead into them. Anybody have an explanation for this behavior? Its sensitivity was probably lowered because it would brake "too often" for a comfortable consumer product if it was sufficiently cautious enough to avoid hitting pedestrians. This is what happened with the Uber self-driving vehicle that struck and killed a pedestrian[1]. The system detected the pedestrian as an obstruction, but Uber had programmed the system to ignore it because of "phantom braking". Instead of braking, it proceeded to strike and kill the pedestrian. [1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg