7 ms·

This is one reason I never understood Tesla's vision-based approach. In order to be accepted, self-driving cars don't need to be just somewhat better than huma

This is one reason I never understood Tesla's vision-based approach. In order to be accepted, self-driving cars don't need to be just somewhat better than humans most of the time. They need to vastly better in every situation, as you mention, to the point that they'll need every sensory advantage they can get. I got out of a ticket once because I didn't see a "no through traffic" sign against a bright sunset. No chance that same cop gives a self-driving car a pass, nor should he.

szundi 4y ago
Karpathy claimed they worked for years and could not reap the benefits from multiple sensors however hard they tried. He seemed really convinced and does not get to me as one who tells stuff just to justify cost reductions, like Elon sometimes is carried away.
- muglug 4y ago
  It’s also possible they weren’t able to see benefits given the processors that were available to them at the time. It’s also possible that they were so far behind Waymo in the journey to FSD that they weren’t yet at a point where multiple sensors would make a significant difference.
  - alexose 4y ago
    This is my read on it as well. I listened to the same interview (it was definitely on the front page, if not at #1 for a while). I stepped away feeling like Karpathy had described a lot of good business reasons for not using other sensors, but not a lot of good technical reasons. Sensor fusion is hard, yes, but maybe not harder than perfectly re-projecting 2D pixel images into 3D vector space. Just my interpretation, but it felt to me like a hail mary because they were fully committed to being the first mover. Waiting around for LiDAR prices to come down would have meant that Waymo would have beat them.
- impulser_ 4y ago
  I always thought it came down to two reason, costs and looks. Telsa has to sell a car people want to drive daily. They can't have a bunch of lidar sensors on their cars no one would buy them even if they could drive themselves. Also they would most likely cost a lot more due to lidar sensors not being cheap compared to normal cameras. Unlike Waymo who doesn't care about selling cars to people to drive daily. No one going to care that the taxi they are taking looks ugly as long as it get them to the place they are going for cheaper. The cost is also a less of a factor due to them being able to produce an income by charging people to ride in them.
  - rootusrootus 4y ago
    > They can't have a bunch of lidar sensors on their cars Why not? There are production cars with lidar now that isn't the big spinning thing on top of the car.
- righttoolforjob 4y ago
  Then something else was the bottleneck at the time. It is very easy to prove that some sensors in some situations will be able to perceive things that other sensors cannot. In those situations the additional sensors are crucial first steps. I would guess the bottleneck is shitty reliance on statistical machine learning with a long tail of unhandled edge cases. Each case very uncommon, but in aggregate a very important sum.
- typon 4y ago
  You are misunderstanding the situation. Karpathy claimed that sensor fusion of radar, sonar and vision isn’t working well. He made no such claim about Lidar. Lidar is the sensor that is the crucial difference between Waymo and Tesla's approach to self driving.
  - stergios 4y ago
    The reason he claimed sensor fusion was not working well was due to vendor versioning. He claimed the same sensor from different manufacturing batches performed differently and thus needed to be re-characterized, which then has follow on effects in various math models. Multiply this by many sensors, and the need for replacement parts inventory for a decade or two and the problem becomes intractable was his claim on his most recent appearance on the LF podcast.
    - cbsmith 4y ago
      Almost sounds like a supply chain/manufacturing problem than a software problem.
- ErikVandeWater 4y ago
  Wasn't the Summon feature downgraded when they moved to a vision-only approach? One YT video I saw compared a version 1 and a more recent version and the vision-only wasn't able to do as much. Contradicts the idea additional sensors do not add value.
  - fastball 4y ago
    I think the idea is less "additional sensors don't add value in various scenarios" and more "we are 100% certain a vision-only system can perform well on existing road infrastructure; we are not sure about sensor fusion systems".
- hackmiester 4y ago
  The problem is, in reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now very difficult to drive with any automation enabled, even simple cruise control, due to the limitations of vision-only driving. And this is leading to them slamming on brakes at inopportune times, blinding oncoming drivers with the bright headlights, etc. - issues that did not happen when other sensors were used.
  - brandonagr2 4y ago
    In reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now better to drive with automation enabled than with the extraneous sensors, even simple cruise control is better and phantom braking is actually decreased vs spurious radar returns that existed previously, due to the limitations of low resolution radar. Auto high beams is now better at not blinding oncoming drivers with bright headlights, etc. - issues that happened all the time when other sensors were used previously.
    - papertokyo 4y ago
      How is your experience the direct opposite of the parent post, given the same change?
      - kiratp 4y ago
        Believe it or not, if you spend enough time with a Tesla, you quickly realize that actually detecting things is a solved problem. The thing they need to improve and are doing so rapidly is actual trajectory policy calculations. And that’s not going to get better with more sensors seeing the same things it already sees.
  - kiratp 4y ago
    This claim is not true. As someone who drives a Tesla with the FSD beta, the vehicle has been getting progressively better since 2018. It’s drives smoother and brakes more predictably dive they stopped using the front radar.
- upbeat_general 4y ago
  If you're referring to the Lex Friedman interview, at the start of his answer, he mentions it was a cost-based decision. And that radar/ultrasonic wasn't worth it for them, due to the additional time it took. Not that it wasn't helpful, just more effort that could be better spent elsewhere.
  - 10x_contrarian 4y ago
    He clearly states that extra sensors "contribute noise and entropy into everything. And they bloat stuff." https://www.youtube.com/watch?v=_W1JBAfV4Io https://www.youtube.com/watch?v=_W1JBAfV4Io Essentially that trying to utilize multiple sensors cripples any progress (given that resources will never be infinite).
    - pclmulqdq 4y ago
      It's too bad that none of their competitors, who all have better results, disagree.
    - Ono-Sendai 4y ago
      While technically true that extra sensors contribute noise, surely with the proper programming they should be helpful. You just need to weight the information from you extra sensors appropriately based on your confidence the signal is accurate. For example, if your lidar sensor is 99.999% percent sure there is an obstacle in front of you, surely it's helpful to take that information into account, even if it is a tiny bit uncertain/noisy.
    - c4ptnjack 4y ago
      Wouldn't that be more related to parallel processing power and throughout capabilities than feasibility? More data in any situation where bandwidth is already maximized will lead to entropy and noise. However, if the capabilities were there to process all of that data in low latency scenarios with headroom to spare, surely adding additional sensors and data points would lead to a more complete model of spatial awareness for the car. That's all hypothetical and reliant on ignoring the realities of running a business and tech development lol
taeric 4y ago
I also never understood why we had to use "vision" approaches that have the same visual spectrum as what humans see. Any sort of sensor on a device is already synthetic, why limit the spectrum that you attach it to? Should use light sensors, sound sensors, gps, everything.
- wongarsu 4y ago
  So far nobody really cared how road markings, signs etc look outside of the human visual spectrum, so there's likely a lot more variation there. Both in terms of how things look like when new, and in terms of acceptable wear.
  - taeric 4y ago
    Right. But that is in designing the existing roads. I'm talking about the cars. And I'm specifically asking why not adding more options? I don't mind a camera being part of the solution at all. Gives an obvious path to human labeling of training data. But, why not have more?
  - pclmulqdq 4y ago
    We have a good idea how all this stuff looks in the infrared spectrum, and that could help a lot in the sensor fusion.
- rootusrootus 4y ago
  And even then, the best cameras today are still a long ways off from matching human eyesight. Tesla's cameras are not state of the art, either.
  - taeric 4y ago
    I wouldn't be shocked if the best cameras are better. :) Point stands that the ones being put in the cars aren't.
    - pclmulqdq 4y ago
rootusrootus 4y ago
One of my biggest concerns about Tesla's vision based approach is that it appears to be entirely about cutting costs. Nothing about it says they actually think it is superior; the cameras on a Model 3/Y are mediocre. They were mediocre the day the Model 3 was first released. If you were going to rely on a vision system in a serious way, you'd at least invest in better camera tech. Hell, Subaru EyeSight has a significantly better camera setup, last I checked, and who looks to Subaru as a technical leader? Someone else said it here on HN, and I think they're absolutely right -- Tesla is all about vertical integration, and this is preventing them from excelling at anything other than saving pennies. A good part of why the new EV competition is doing everything better is they didn't roll their own tech. They bought packaged solutions from companies that only do one thing, but do it well.
- cryptoegorophy 4y ago
  have you driven a tesla autopilot at night in heavy rain with no visibility of the lanes? I did, I was amazed at how good it was. I couldn't see lanes but it did and judging by the picture on the screen it saw them very well. More pixels doesn't mean better, same as more ram doesnt mean better (iOS example).
  - rootusrootus 4y ago
    Yes, I'm on my second Tesla in fact. I have even earned the dubious honor of having been pulled over for suspected drunk driving because I used AP on a trip home from my MIL's house after Thanksgiving dinner. I was amused by how much trouble AP was having with mild corners on I-5, the state trooper behind me was less amused. Well, at least until he established that I was sober, then we had a good laugh about autopilot and he gave my kids sticker badges. A happy ending for everyone but I didn't activate AP again that night. Maybe the problem was that it was clear and dry? Ya think it would have been better with some road spray on the cameras? Subsequent to that, I ended up abandoning AP altogether because it was so jarring when it would decide an overpass looked like something it should slow down for. My wife suggested that she'd be happier if I just used my foot for the accelerator. For some reason we still ended up buying another one, but after the phantom braking on day one (empty freeway, overpass, apparently this is very much now "a thing"), I guess I'll just skip AP before even starting. I wonder if I could ask Tesla to deactivate AP altogether so I could have traditional cruise control?
jkeddo 4y ago
They are standing firm on the vision-only approach because it is the correct approach. FSD cannot be perfected unless the Tesla team puts 100% exclusive focus on perfecting vision based models that don't have lidar as a fallback. Tesla can only use lidar again for redundancy only after vision is fully solved problem. https://tvtropes.org/pmwiki/pmwiki.php/Main/BurningTheShips https://tvtropes.org/pmwiki/pmwiki.php/Main/BurningTheShips
- seanmcdirmid 4y ago
  Combing multiple sensory inputs is a hard problem in itself. Say Tesla uses LIDAR with vision...they can have a great vision experience, a great LIDAR experience, but fuses those two experiences together can create lots of problems that wind up being worse than either by itself.
- r00fus 4y ago
  So the beatings will continue until morale improves, eh? This is like saying we need to get to the moon using steam power and ONLY THEN we can improve by using rockets. What if it's completely infeasible to solely use vision for FSD? Even humans use audio processing to augment the vision.
mhb 4y ago
Tesla May Be Adding A Radar https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-may-be-adding-a-radar-which-is-the-crutch-they-definitely-need/ https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-...
- jcrash 4y ago
  Archived: https://archive.vn/20221212143745/https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-may-be-adding-a-radar-which-is-the-crutch-they-definitely-need/ https://archive.vn/20221212143745/https://www.forbes.com/sit...

szundi 4y ago

Karpathy claimed they worked for years and could not reap the benefits from multiple sensors however hard they tried. He seemed really convinced and does not get to me as one who tells stuff just to justify cost reductions, like Elon sometimes is carried away.

muglug 4y ago
It’s also possible they weren’t able to see benefits given the processors that were available to them at the time. It’s also possible that they were so far behind Waymo in the journey to FSD that they weren’t yet at a point where multiple sensors would make a significant difference.
- alexose 4y ago
  This is my read on it as well. I listened to the same interview (it was definitely on the front page, if not at #1 for a while). I stepped away feeling like Karpathy had described a lot of good business reasons for not using other sensors, but not a lot of good technical reasons. Sensor fusion is hard, yes, but maybe not harder than perfectly re-projecting 2D pixel images into 3D vector space. Just my interpretation, but it felt to me like a hail mary because they were fully committed to being the first mover. Waiting around for LiDAR prices to come down would have meant that Waymo would have beat them.
impulser_ 4y ago
I always thought it came down to two reason, costs and looks. Telsa has to sell a car people want to drive daily. They can't have a bunch of lidar sensors on their cars no one would buy them even if they could drive themselves. Also they would most likely cost a lot more due to lidar sensors not being cheap compared to normal cameras. Unlike Waymo who doesn't care about selling cars to people to drive daily. No one going to care that the taxi they are taking looks ugly as long as it get them to the place they are going for cheaper. The cost is also a less of a factor due to them being able to produce an income by charging people to ride in them.
- rootusrootus 4y ago
  > They can't have a bunch of lidar sensors on their cars Why not? There are production cars with lidar now that isn't the big spinning thing on top of the car.
righttoolforjob 4y ago
Then something else was the bottleneck at the time. It is very easy to prove that some sensors in some situations will be able to perceive things that other sensors cannot. In those situations the additional sensors are crucial first steps. I would guess the bottleneck is shitty reliance on statistical machine learning with a long tail of unhandled edge cases. Each case very uncommon, but in aggregate a very important sum.
typon 4y ago
You are misunderstanding the situation. Karpathy claimed that sensor fusion of radar, sonar and vision isn’t working well. He made no such claim about Lidar. Lidar is the sensor that is the crucial difference between Waymo and Tesla's approach to self driving.
- stergios 4y ago
  The reason he claimed sensor fusion was not working well was due to vendor versioning. He claimed the same sensor from different manufacturing batches performed differently and thus needed to be re-characterized, which then has follow on effects in various math models. Multiply this by many sensors, and the need for replacement parts inventory for a decade or two and the problem becomes intractable was his claim on his most recent appearance on the LF podcast.
  - cbsmith 4y ago
    Almost sounds like a supply chain/manufacturing problem than a software problem.
ErikVandeWater 4y ago
Wasn't the Summon feature downgraded when they moved to a vision-only approach? One YT video I saw compared a version 1 and a more recent version and the vision-only wasn't able to do as much. Contradicts the idea additional sensors do not add value.
- fastball 4y ago
  I think the idea is less "additional sensors don't add value in various scenarios" and more "we are 100% certain a vision-only system can perform well on existing road infrastructure; we are not sure about sensor fusion systems".
hackmiester 4y ago
The problem is, in reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now very difficult to drive with any automation enabled, even simple cruise control, due to the limitations of vision-only driving. And this is leading to them slamming on brakes at inopportune times, blinding oncoming drivers with the bright headlights, etc. - issues that did not happen when other sensors were used.
- brandonagr2 4y ago
  In reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now better to drive with automation enabled than with the extraneous sensors, even simple cruise control is better and phantom braking is actually decreased vs spurious radar returns that existed previously, due to the limitations of low resolution radar. Auto high beams is now better at not blinding oncoming drivers with bright headlights, etc. - issues that happened all the time when other sensors were used previously.
  - papertokyo 4y ago
    How is your experience the direct opposite of the parent post, given the same change?
    - kiratp 4y ago
      Believe it or not, if you spend enough time with a Tesla, you quickly realize that actually detecting things is a solved problem. The thing they need to improve and are doing so rapidly is actual trajectory policy calculations. And that’s not going to get better with more sensors seeing the same things it already sees.
- kiratp 4y ago
  This claim is not true. As someone who drives a Tesla with the FSD beta, the vehicle has been getting progressively better since 2018. It’s drives smoother and brakes more predictably dive they stopped using the front radar.
upbeat_general 4y ago
If you're referring to the Lex Friedman interview, at the start of his answer, he mentions it was a cost-based decision. And that radar/ultrasonic wasn't worth it for them, due to the additional time it took. Not that it wasn't helpful, just more effort that could be better spent elsewhere.
- 10x_contrarian 4y ago
  He clearly states that extra sensors "contribute noise and entropy into everything. And they bloat stuff." https://www.youtube.com/watch?v=_W1JBAfV4Io https://www.youtube.com/watch?v=_W1JBAfV4Io Essentially that trying to utilize multiple sensors cripples any progress (given that resources will never be infinite).
  - pclmulqdq 4y ago
    It's too bad that none of their competitors, who all have better results, disagree.
  - Ono-Sendai 4y ago
    While technically true that extra sensors contribute noise, surely with the proper programming they should be helpful. You just need to weight the information from you extra sensors appropriately based on your confidence the signal is accurate. For example, if your lidar sensor is 99.999% percent sure there is an obstacle in front of you, surely it's helpful to take that information into account, even if it is a tiny bit uncertain/noisy.
  - c4ptnjack 4y ago
    Wouldn't that be more related to parallel processing power and throughout capabilities than feasibility? More data in any situation where bandwidth is already maximized will lead to entropy and noise. However, if the capabilities were there to process all of that data in low latency scenarios with headroom to spare, surely adding additional sensors and data points would lead to a more complete model of spatial awareness for the car. That's all hypothetical and reliant on ignoring the realities of running a business and tech development lol

taeric 4y ago

I also never understood why we had to use "vision" approaches that have the same visual spectrum as what humans see. Any sort of sensor on a device is already synthetic, why limit the spectrum that you attach it to? Should use light sensors, sound sensors, gps, everything.

wongarsu 4y ago
So far nobody really cared how road markings, signs etc look outside of the human visual spectrum, so there's likely a lot more variation there. Both in terms of how things look like when new, and in terms of acceptable wear.
- taeric 4y ago
  Right. But that is in designing the existing roads. I'm talking about the cars. And I'm specifically asking why not adding more options? I don't mind a camera being part of the solution at all. Gives an obvious path to human labeling of training data. But, why not have more?
- pclmulqdq 4y ago
  We have a good idea how all this stuff looks in the infrared spectrum, and that could help a lot in the sensor fusion.
rootusrootus 4y ago
And even then, the best cameras today are still a long ways off from matching human eyesight. Tesla's cameras are not state of the art, either.
- taeric 4y ago
  I wouldn't be shocked if the best cameras are better. :) Point stands that the ones being put in the cars aren't.
  - pclmulqdq 4y ago

rootusrootus 4y ago

One of my biggest concerns about Tesla's vision based approach is that it appears to be entirely about cutting costs. Nothing about it says they actually think it is superior; the cameras on a Model 3/Y are mediocre. They were mediocre the day the Model 3 was first released. If you were going to rely on a vision system in a serious way, you'd at least invest in better camera tech. Hell, Subaru EyeSight has a significantly better camera setup, last I checked, and who looks to Subaru as a technical leader? Someone else said it here on HN, and I think they're absolutely right -- Tesla is all about vertical integration, and this is preventing them from excelling at anything other than saving pennies. A good part of why the new EV competition is doing everything better is they didn't roll their own tech. They bought packaged solutions from companies that only do one thing, but do it well.

cryptoegorophy 4y ago
have you driven a tesla autopilot at night in heavy rain with no visibility of the lanes? I did, I was amazed at how good it was. I couldn't see lanes but it did and judging by the picture on the screen it saw them very well. More pixels doesn't mean better, same as more ram doesnt mean better (iOS example).
- rootusrootus 4y ago
  Yes, I'm on my second Tesla in fact. I have even earned the dubious honor of having been pulled over for suspected drunk driving because I used AP on a trip home from my MIL's house after Thanksgiving dinner. I was amused by how much trouble AP was having with mild corners on I-5, the state trooper behind me was less amused. Well, at least until he established that I was sober, then we had a good laugh about autopilot and he gave my kids sticker badges. A happy ending for everyone but I didn't activate AP again that night. Maybe the problem was that it was clear and dry? Ya think it would have been better with some road spray on the cameras? Subsequent to that, I ended up abandoning AP altogether because it was so jarring when it would decide an overpass looked like something it should slow down for. My wife suggested that she'd be happier if I just used my foot for the accelerator. For some reason we still ended up buying another one, but after the phantom braking on day one (empty freeway, overpass, apparently this is very much now "a thing"), I guess I'll just skip AP before even starting. I wonder if I could ask Tesla to deactivate AP altogether so I could have traditional cruise control?

jkeddo 4y ago

They are standing firm on the vision-only approach because it is the correct approach. FSD cannot be perfected unless the Tesla team puts 100% exclusive focus on perfecting vision based models that don't have lidar as a fallback. Tesla can only use lidar again for redundancy only after vision is fully solved problem. https://tvtropes.org/pmwiki/pmwiki.php/Main/BurningTheShips https://tvtropes.org/pmwiki/pmwiki.php/Main/BurningTheShips

seanmcdirmid 4y ago
Combing multiple sensory inputs is a hard problem in itself. Say Tesla uses LIDAR with vision...they can have a great vision experience, a great LIDAR experience, but fuses those two experiences together can create lots of problems that wind up being worse than either by itself.
r00fus 4y ago
So the beatings will continue until morale improves, eh? This is like saying we need to get to the moon using steam power and ONLY THEN we can improve by using rockets. What if it's completely infeasible to solely use vision for FSD? Even humans use audio processing to augment the vision.

mhb 4y ago

Tesla May Be Adding A Radar https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-may-be-adding-a-radar-which-is-the-crutch-they-definitely-need/ https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-...

jcrash 4y ago
Archived: https://archive.vn/20221212143745/https://www.forbes.com/sites/bradtempleton/2022/12/12/tesla-may-be-adding-a-radar-which-is-the-crutch-they-definitely-need/ https://archive.vn/20221212143745/https://www.forbes.com/sit...