5 ms·

Karpathy claimed they worked for years and could not reap the benefits from multiple sensors however hard they tried. He seemed really convinced and does not ge

by szundi 4y ago

Karpathy claimed they worked for years and could not reap the benefits from multiple sensors however hard they tried. He seemed really convinced and does not get to me as one who tells stuff just to justify cost reductions, like Elon sometimes is carried away.

muglug 4y ago
It’s also possible they weren’t able to see benefits given the processors that were available to them at the time. It’s also possible that they were so far behind Waymo in the journey to FSD that they weren’t yet at a point where multiple sensors would make a significant difference.
- alexose 4y ago
  This is my read on it as well. I listened to the same interview (it was definitely on the front page, if not at #1 for a while). I stepped away feeling like Karpathy had described a lot of good business reasons for not using other sensors, but not a lot of good technical reasons. Sensor fusion is hard, yes, but maybe not harder than perfectly re-projecting 2D pixel images into 3D vector space. Just my interpretation, but it felt to me like a hail mary because they were fully committed to being the first mover. Waiting around for LiDAR prices to come down would have meant that Waymo would have beat them.
impulser_ 4y ago
I always thought it came down to two reason, costs and looks. Telsa has to sell a car people want to drive daily. They can't have a bunch of lidar sensors on their cars no one would buy them even if they could drive themselves. Also they would most likely cost a lot more due to lidar sensors not being cheap compared to normal cameras. Unlike Waymo who doesn't care about selling cars to people to drive daily. No one going to care that the taxi they are taking looks ugly as long as it get them to the place they are going for cheaper. The cost is also a less of a factor due to them being able to produce an income by charging people to ride in them.
- rootusrootus 4y ago
  > They can't have a bunch of lidar sensors on their cars Why not? There are production cars with lidar now that isn't the big spinning thing on top of the car.
righttoolforjob 4y ago
Then something else was the bottleneck at the time. It is very easy to prove that some sensors in some situations will be able to perceive things that other sensors cannot. In those situations the additional sensors are crucial first steps. I would guess the bottleneck is shitty reliance on statistical machine learning with a long tail of unhandled edge cases. Each case very uncommon, but in aggregate a very important sum.
typon 4y ago
You are misunderstanding the situation. Karpathy claimed that sensor fusion of radar, sonar and vision isn’t working well. He made no such claim about Lidar. Lidar is the sensor that is the crucial difference between Waymo and Tesla's approach to self driving.
- stergios 4y ago
  The reason he claimed sensor fusion was not working well was due to vendor versioning. He claimed the same sensor from different manufacturing batches performed differently and thus needed to be re-characterized, which then has follow on effects in various math models. Multiply this by many sensors, and the need for replacement parts inventory for a decade or two and the problem becomes intractable was his claim on his most recent appearance on the LF podcast.
  - cbsmith 4y ago
    Almost sounds like a supply chain/manufacturing problem than a software problem.
ErikVandeWater 4y ago
Wasn't the Summon feature downgraded when they moved to a vision-only approach? One YT video I saw compared a version 1 and a more recent version and the vision-only wasn't able to do as much. Contradicts the idea additional sensors do not add value.
- fastball 4y ago
  I think the idea is less "additional sensors don't add value in various scenarios" and more "we are 100% certain a vision-only system can perform well on existing road infrastructure; we are not sure about sensor fusion systems".
hackmiester 4y ago
The problem is, in reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now very difficult to drive with any automation enabled, even simple cruise control, due to the limitations of vision-only driving. And this is leading to them slamming on brakes at inopportune times, blinding oncoming drivers with the bright headlights, etc. - issues that did not happen when other sensors were used.
- brandonagr2 4y ago
  In reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now better to drive with automation enabled than with the extraneous sensors, even simple cruise control is better and phantom braking is actually decreased vs spurious radar returns that existed previously, due to the limitations of low resolution radar. Auto high beams is now better at not blinding oncoming drivers with bright headlights, etc. - issues that happened all the time when other sensors were used previously.
  - papertokyo 4y ago
    How is your experience the direct opposite of the parent post, given the same change?
    - kiratp 4y ago
      Believe it or not, if you spend enough time with a Tesla, you quickly realize that actually detecting things is a solved problem. The thing they need to improve and are doing so rapidly is actual trajectory policy calculations. And that’s not going to get better with more sensors seeing the same things it already sees.
- kiratp 4y ago
  This claim is not true. As someone who drives a Tesla with the FSD beta, the vehicle has been getting progressively better since 2018. It’s drives smoother and brakes more predictably dive they stopped using the front radar.
upbeat_general 4y ago
If you're referring to the Lex Friedman interview, at the start of his answer, he mentions it was a cost-based decision. And that radar/ultrasonic wasn't worth it for them, due to the additional time it took. Not that it wasn't helpful, just more effort that could be better spent elsewhere.
- 10x_contrarian 4y ago
  He clearly states that extra sensors "contribute noise and entropy into everything. And they bloat stuff." https://www.youtube.com/watch?v=_W1JBAfV4Io https://www.youtube.com/watch?v=_W1JBAfV4Io Essentially that trying to utilize multiple sensors cripples any progress (given that resources will never be infinite).
  - pclmulqdq 4y ago
    It's too bad that none of their competitors, who all have better results, disagree.
  - Ono-Sendai 4y ago
    While technically true that extra sensors contribute noise, surely with the proper programming they should be helpful. You just need to weight the information from you extra sensors appropriately based on your confidence the signal is accurate. For example, if your lidar sensor is 99.999% percent sure there is an obstacle in front of you, surely it's helpful to take that information into account, even if it is a tiny bit uncertain/noisy.
  - c4ptnjack 4y ago
    Wouldn't that be more related to parallel processing power and throughout capabilities than feasibility? More data in any situation where bandwidth is already maximized will lead to entropy and noise. However, if the capabilities were there to process all of that data in low latency scenarios with headroom to spare, surely adding additional sensors and data points would lead to a more complete model of spatial awareness for the car. That's all hypothetical and reliant on ignoring the realities of running a business and tech development lol

by szundi 4y ago

muglug 4y ago

It’s also possible they weren’t able to see benefits given the processors that were available to them at the time. It’s also possible that they were so far behind Waymo in the journey to FSD that they weren’t yet at a point where multiple sensors would make a significant difference.

alexose 4y ago
This is my read on it as well. I listened to the same interview (it was definitely on the front page, if not at #1 for a while). I stepped away feeling like Karpathy had described a lot of good business reasons for not using other sensors, but not a lot of good technical reasons. Sensor fusion is hard, yes, but maybe not harder than perfectly re-projecting 2D pixel images into 3D vector space. Just my interpretation, but it felt to me like a hail mary because they were fully committed to being the first mover. Waiting around for LiDAR prices to come down would have meant that Waymo would have beat them.

impulser_ 4y ago

I always thought it came down to two reason, costs and looks. Telsa has to sell a car people want to drive daily. They can't have a bunch of lidar sensors on their cars no one would buy them even if they could drive themselves. Also they would most likely cost a lot more due to lidar sensors not being cheap compared to normal cameras. Unlike Waymo who doesn't care about selling cars to people to drive daily. No one going to care that the taxi they are taking looks ugly as long as it get them to the place they are going for cheaper. The cost is also a less of a factor due to them being able to produce an income by charging people to ride in them.

rootusrootus 4y ago
> They can't have a bunch of lidar sensors on their cars Why not? There are production cars with lidar now that isn't the big spinning thing on top of the car.

righttoolforjob 4y ago

Then something else was the bottleneck at the time. It is very easy to prove that some sensors in some situations will be able to perceive things that other sensors cannot. In those situations the additional sensors are crucial first steps. I would guess the bottleneck is shitty reliance on statistical machine learning with a long tail of unhandled edge cases. Each case very uncommon, but in aggregate a very important sum.

typon 4y ago

You are misunderstanding the situation. Karpathy claimed that sensor fusion of radar, sonar and vision isn’t working well. He made no such claim about Lidar. Lidar is the sensor that is the crucial difference between Waymo and Tesla's approach to self driving.

stergios 4y ago
The reason he claimed sensor fusion was not working well was due to vendor versioning. He claimed the same sensor from different manufacturing batches performed differently and thus needed to be re-characterized, which then has follow on effects in various math models. Multiply this by many sensors, and the need for replacement parts inventory for a decade or two and the problem becomes intractable was his claim on his most recent appearance on the LF podcast.
- cbsmith 4y ago
  Almost sounds like a supply chain/manufacturing problem than a software problem.

ErikVandeWater 4y ago

Wasn't the Summon feature downgraded when they moved to a vision-only approach? One YT video I saw compared a version 1 and a more recent version and the vision-only wasn't able to do as much. Contradicts the idea additional sensors do not add value.

fastball 4y ago
I think the idea is less "additional sensors don't add value in various scenarios" and more "we are 100% certain a vision-only system can perform well on existing road infrastructure; we are not sure about sensor fusion systems".

hackmiester 4y ago

The problem is, in reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now very difficult to drive with any automation enabled, even simple cruise control, due to the limitations of vision-only driving. And this is leading to them slamming on brakes at inopportune times, blinding oncoming drivers with the bright headlights, etc. - issues that did not happen when other sensors were used.

brandonagr2 4y ago
In reality, we have Tesla vehicles that sold with other sensors that are now disabled, and those cars are now better to drive with automation enabled than with the extraneous sensors, even simple cruise control is better and phantom braking is actually decreased vs spurious radar returns that existed previously, due to the limitations of low resolution radar. Auto high beams is now better at not blinding oncoming drivers with bright headlights, etc. - issues that happened all the time when other sensors were used previously.
- papertokyo 4y ago
  How is your experience the direct opposite of the parent post, given the same change?
  - kiratp 4y ago
    Believe it or not, if you spend enough time with a Tesla, you quickly realize that actually detecting things is a solved problem. The thing they need to improve and are doing so rapidly is actual trajectory policy calculations. And that’s not going to get better with more sensors seeing the same things it already sees.
kiratp 4y ago
This claim is not true. As someone who drives a Tesla with the FSD beta, the vehicle has been getting progressively better since 2018. It’s drives smoother and brakes more predictably dive they stopped using the front radar.

upbeat_general 4y ago

If you're referring to the Lex Friedman interview, at the start of his answer, he mentions it was a cost-based decision. And that radar/ultrasonic wasn't worth it for them, due to the additional time it took. Not that it wasn't helpful, just more effort that could be better spent elsewhere.

10x_contrarian 4y ago
He clearly states that extra sensors "contribute noise and entropy into everything. And they bloat stuff." https://www.youtube.com/watch?v=_W1JBAfV4Io https://www.youtube.com/watch?v=_W1JBAfV4Io Essentially that trying to utilize multiple sensors cripples any progress (given that resources will never be infinite).
- pclmulqdq 4y ago
  It's too bad that none of their competitors, who all have better results, disagree.
- Ono-Sendai 4y ago
  While technically true that extra sensors contribute noise, surely with the proper programming they should be helpful. You just need to weight the information from you extra sensors appropriately based on your confidence the signal is accurate. For example, if your lidar sensor is 99.999% percent sure there is an obstacle in front of you, surely it's helpful to take that information into account, even if it is a tiny bit uncertain/noisy.
- c4ptnjack 4y ago
  Wouldn't that be more related to parallel processing power and throughout capabilities than feasibility? More data in any situation where bandwidth is already maximized will lead to entropy and noise. However, if the capabilities were there to process all of that data in low latency scenarios with headroom to spare, surely adding additional sensors and data points would lead to a more complete model of spatial awareness for the car. That's all hypothetical and reliant on ignoring the realities of running a business and tech development lol