8 ms·

Karpathy provided additional context on the removal of LiDAR during his Lex Fridman Podcast appearance. This article condenses what he said: https://archive.is

by runjake 8mo ago

Karpathy provided additional context on the removal of LiDAR during his Lex Fridman Podcast appearance. This article condenses what he said: https://archive.is/PPiVG https://archive.is/PPiVG And here's one of Elon's mentions (he also has talked about it quite a bit in various spots). https://xcancel.com/elonmusk/status/1959831831668228450?s=20 https://xcancel.com/elonmusk/status/1959831831668228450?s=20 Edit: My personal view is that LiDAR and other sensors are extremely useful, but I worked on aircraft, not cars.

willio58 8mo ago
Based on that list it boils down to 2 things it seems: - cost (no longer a problem) - too much code needed and it bloats the data pipelines. Does anyone have any actual evidence of this being the case? Like yes, code would be needed, but why is that innately a bad thing? Bloated data pipelines feels like another hand-wave when I think if you do it right it’s fine. As proven by Waymo. Really curious if any Tesla engineers feel like this is still the best way forward or if it’s just a matter of having to listen to the big guy musk. I’ve always felt that relying on vision only would be a detriment because even humans with good vision get into circumstances where they get hurt because of temporary vision hindrances. Think heavy snow, heavy rain, heavy fog, even just when you crest a hill at a certain time of day and the sun flashes you
- atonse 8mo ago
  Just for the record though, Musk isn't blindly anti-LIDAR. He has said (and I think this is an objective fact) that all existing roads and driving are based on vision (which is what all humans do). So that should technically be sufficient. SpaceX uses LIDAR for their docking systems. I would argue that yes, we do use vision but we get that "lidar depth" from our stereo vision. And that used to be why I thought cameras weren't enough. But then look at all the work with gaussian splatting (where you can take multiple 2d samples and build a 3d world out of it). So you could probably get 80% there with just that. The ethos of many Musk companies (you'll hear this from many engineers that work there) is simplify, simplify, simplify. If something isn't needed, take it out. Question everything that might be needed. To me, LIDAR is just one of those things in that general pattern of "if it isn't absolutely needed, take it out" – and the fact that FSD works so well without it proves that it isn't required. It's probably a nice to have, but maybe not required.
  - dymk 8mo ago
    Humans aren't using only fixed vision for driving. This is such a tiresome thing to see repeated in every discussion about self driving. You're listening to the road and car sounds around you. You're feeling vibration on the road. You're feeling feedback on the steering wheel. You're using a combination of monocular and binocular depth perception - plus, your eyes are not a fixed focal length "cameras". You're moving your head to change the perspective you see the road at. Your inner ear is telling you about your acceleration and orientation.
    - kube-system 8mo ago
      And also, even with the suite of sensors that humans have, their vision perception is frequently inadequate and leads to crashes. If vision was good enough, "SMIDSY" wouldn't be such an infamous acronym in vehicle injury cases.
      - kelnos 8mo ago
        For those of us not aware of Australian cycling jargon, "SMIDSY" means "Sorry, Mate, I Didn't See You".
      - anthonypasq 8mo ago
        the issue is clearly attention not vision when it comes to humans. if we could actually process 100% of the visual information in our field of view, then accidents would probably go down a shit load.
        kube-system 8mo ago
        Humans have both issues. There are many human failures which are distinctly a vision issue and not attention related, e.g. misestimation of depth/speed, obscured or obstructed vision, optical focus issues, insufficient contrast or exposure, etc.
        arijun 8mo ago
- c7b 8mo ago
  My understanding is that there's more data processing required with cameras because you need to estimate distance from stereoscopic vision. And as it happens, the required chips for that have shot up in price because of the AI boom. But I think costs were just part of the reason why Elon decided against Lidar. Apparently, they interfere with each other once the market saturates and you have many such cars on the same streets at the same time. Haven't heard yet how the Lidar proponents are planning to address that.
  - jerlam 8mo ago
    How does Waymo handle it now? There are many videos of Waymo depots with dozens of cars not running into each other.
    - overfeed 8mo ago
      Lidar critics like to pretend that anti-collision is not a well-studied branch of Computer Science and telecoms. Wifi, Ethernet and cellphones all work well simultaneously, despite participants all sharing the same physical medium.
      - deleted 8mo ago
        [deleted]
      - c7b 8mo ago
        I'm not a Lidar critic. I'm really just curious how they're addressing it, or plan to.
    - ggreer 8mo ago
      It's rare, but sometimes they do hit each other: https://www.reddit.com/r/SelfDrivingCars/comments/1mdl5zn/two_waymo_cars_collided_in_phoenix_today/ https://www.reddit.com/r/SelfDrivingCars/comments/1mdl5zn/tw... https://www.reddit.com/r/waymo/comments/1pggtpu/two_waymos_made_contact_with_each_other_in_sf/ https://www.reddit.com/r/waymo/comments/1pggtpu/two_waymos_m...
- dzhiurgis 8mo ago
  If Waymo proven their model works, why the silly automaker is doing several orders of magnitude more autonomous miles?
  - AlotOfReading 8mo ago
    They aren't. Tesla has logged some 800k total miles with their robotaxi vehicles, including miles with safety drivers. Waymo has logged 200M driverless miles. That's 0.4% of the mileage, with the most generous possible framing.
AnotherGoodName 8mo ago
The points linked repeatedly focus on cost and complexity as justification, even explicitly stating musks desire to minimise components in Kaparthy’s list. They don’t focus on safety or effectiveness except to say that vision should be ‘sufficient’. Which is damning with faint praise imho. If that link was to try and argue that the removal of sensors makes perfect sense i have to point out that anyone that reads that would likely have their negative viewpoint hardened. It was done to reduce cost (back when the sensors were 1000’s) and out of a ridiculous desire by Musk for minimalism. It’s the same desire that removed the indicator stalk i might add.
- runjake 8mo ago
  To be clear, from a personal standpoint, I am pro-more sensors and sensor fusion. I assume Musk, et al are acting in best faith in trying to find the right compromises.
  - thfuran 8mo ago
    Why would you assume Musk is acting in good faith? That’s very much not his thing.
  - hn_acc1 8mo ago
    Oh, you sweet summer child..
kappi 8mo ago
Instead of betting on RADAR and LIDAR HW getting better and cost going down, they went with vision only approach. Everybody in this field knows the strengths and weakness of each system. Multi-modal sensor fusion is the way to go for L4 autonomy. There is no other way to reduce the risk. Vision only will never be able to achieve L4 in all the weather conditions. Tesla may try to demonstrate L4 in limited geography and in good weather conditions but it won't scale.
utopcell 8mo ago
From the article: Karpathy’s main points: Extra sensors add cost to the system, and more importantly complexity. They make the software task harder, and increase the cost of all the data pipelines. They add risk and complexity to the supply chain and manufacturing. Elon Musk pushes a philosophy of “the best part is no part” which can be seen throughout the car in things like doing everything through the touchscreen. This is an expression of this philosophy. Vision is necessary to the task (which almost all agree on) and it should also be sufficient as well. If it is sufficient, the cost of extra sensors and tools outweighs their benefit. Sensors change as parts change or become available and unavailable. They must be maintained and software adapted to these changes. They must also be calibrated to make fusion work properly. Having a fleet gathering more data is more important than having more sensors. Having to process LIDAR and radar produces a lot of bloat in the code and data pipelines. He predicts other companies will also drop these sensors in time. Mapping the world and keeping it up to date is much too expensive. You won’t change the world with this limitation, you need to focus on vision which is the most important. The roads are designed to be interpreted with vision.
- utopcell 8mo ago
  So the argument is pretty much: it should be sufficient to use vision only, and that it is too difficult / expensive to do otherwise.

by runjake 8mo ago

willio58 8mo ago

Based on that list it boils down to 2 things it seems: - cost (no longer a problem) - too much code needed and it bloats the data pipelines. Does anyone have any actual evidence of this being the case? Like yes, code would be needed, but why is that innately a bad thing? Bloated data pipelines feels like another hand-wave when I think if you do it right it’s fine. As proven by Waymo. Really curious if any Tesla engineers feel like this is still the best way forward or if it’s just a matter of having to listen to the big guy musk. I’ve always felt that relying on vision only would be a detriment because even humans with good vision get into circumstances where they get hurt because of temporary vision hindrances. Think heavy snow, heavy rain, heavy fog, even just when you crest a hill at a certain time of day and the sun flashes you

atonse 8mo ago
Just for the record though, Musk isn't blindly anti-LIDAR. He has said (and I think this is an objective fact) that all existing roads and driving are based on vision (which is what all humans do). So that should technically be sufficient. SpaceX uses LIDAR for their docking systems. I would argue that yes, we do use vision but we get that "lidar depth" from our stereo vision. And that used to be why I thought cameras weren't enough. But then look at all the work with gaussian splatting (where you can take multiple 2d samples and build a 3d world out of it). So you could probably get 80% there with just that. The ethos of many Musk companies (you'll hear this from many engineers that work there) is simplify, simplify, simplify. If something isn't needed, take it out. Question everything that might be needed. To me, LIDAR is just one of those things in that general pattern of "if it isn't absolutely needed, take it out" – and the fact that FSD works so well without it proves that it isn't required. It's probably a nice to have, but maybe not required.
- dymk 8mo ago
  Humans aren't using only fixed vision for driving. This is such a tiresome thing to see repeated in every discussion about self driving. You're listening to the road and car sounds around you. You're feeling vibration on the road. You're feeling feedback on the steering wheel. You're using a combination of monocular and binocular depth perception - plus, your eyes are not a fixed focal length "cameras". You're moving your head to change the perspective you see the road at. Your inner ear is telling you about your acceleration and orientation.
  - kube-system 8mo ago
    And also, even with the suite of sensors that humans have, their vision perception is frequently inadequate and leads to crashes. If vision was good enough, "SMIDSY" wouldn't be such an infamous acronym in vehicle injury cases.
    - kelnos 8mo ago
      For those of us not aware of Australian cycling jargon, "SMIDSY" means "Sorry, Mate, I Didn't See You".
    - anthonypasq 8mo ago
      the issue is clearly attention not vision when it comes to humans. if we could actually process 100% of the visual information in our field of view, then accidents would probably go down a shit load.
      - kube-system 8mo ago
        Humans have both issues. There are many human failures which are distinctly a vision issue and not attention related, e.g. misestimation of depth/speed, obscured or obstructed vision, optical focus issues, insufficient contrast or exposure, etc.
        arijun 8mo ago
c7b 8mo ago
My understanding is that there's more data processing required with cameras because you need to estimate distance from stereoscopic vision. And as it happens, the required chips for that have shot up in price because of the AI boom. But I think costs were just part of the reason why Elon decided against Lidar. Apparently, they interfere with each other once the market saturates and you have many such cars on the same streets at the same time. Haven't heard yet how the Lidar proponents are planning to address that.
- jerlam 8mo ago
  How does Waymo handle it now? There are many videos of Waymo depots with dozens of cars not running into each other.
  - overfeed 8mo ago
    Lidar critics like to pretend that anti-collision is not a well-studied branch of Computer Science and telecoms. Wifi, Ethernet and cellphones all work well simultaneously, despite participants all sharing the same physical medium.
    - deleted 8mo ago
      [deleted]
    - c7b 8mo ago
      I'm not a Lidar critic. I'm really just curious how they're addressing it, or plan to.
  - ggreer 8mo ago
    It's rare, but sometimes they do hit each other: https://www.reddit.com/r/SelfDrivingCars/comments/1mdl5zn/two_waymo_cars_collided_in_phoenix_today/ https://www.reddit.com/r/SelfDrivingCars/comments/1mdl5zn/tw... https://www.reddit.com/r/waymo/comments/1pggtpu/two_waymos_made_contact_with_each_other_in_sf/ https://www.reddit.com/r/waymo/comments/1pggtpu/two_waymos_m...
dzhiurgis 8mo ago
If Waymo proven their model works, why the silly automaker is doing several orders of magnitude more autonomous miles?
- AlotOfReading 8mo ago
  They aren't. Tesla has logged some 800k total miles with their robotaxi vehicles, including miles with safety drivers. Waymo has logged 200M driverless miles. That's 0.4% of the mileage, with the most generous possible framing.

AnotherGoodName 8mo ago

The points linked repeatedly focus on cost and complexity as justification, even explicitly stating musks desire to minimise components in Kaparthy’s list. They don’t focus on safety or effectiveness except to say that vision should be ‘sufficient’. Which is damning with faint praise imho. If that link was to try and argue that the removal of sensors makes perfect sense i have to point out that anyone that reads that would likely have their negative viewpoint hardened. It was done to reduce cost (back when the sensors were 1000’s) and out of a ridiculous desire by Musk for minimalism. It’s the same desire that removed the indicator stalk i might add.

runjake 8mo ago
To be clear, from a personal standpoint, I am pro-more sensors and sensor fusion. I assume Musk, et al are acting in best faith in trying to find the right compromises.
- thfuran 8mo ago
  Why would you assume Musk is acting in good faith? That’s very much not his thing.
- hn_acc1 8mo ago
  Oh, you sweet summer child..

kappi 8mo ago

Instead of betting on RADAR and LIDAR HW getting better and cost going down, they went with vision only approach. Everybody in this field knows the strengths and weakness of each system. Multi-modal sensor fusion is the way to go for L4 autonomy. There is no other way to reduce the risk. Vision only will never be able to achieve L4 in all the weather conditions. Tesla may try to demonstrate L4 in limited geography and in good weather conditions but it won't scale.

utopcell 8mo ago

From the article: Karpathy’s main points: Extra sensors add cost to the system, and more importantly complexity. They make the software task harder, and increase the cost of all the data pipelines. They add risk and complexity to the supply chain and manufacturing. Elon Musk pushes a philosophy of “the best part is no part” which can be seen throughout the car in things like doing everything through the touchscreen. This is an expression of this philosophy. Vision is necessary to the task (which almost all agree on) and it should also be sufficient as well. If it is sufficient, the cost of extra sensors and tools outweighs their benefit. Sensors change as parts change or become available and unavailable. They must be maintained and software adapted to these changes. They must also be calibrated to make fusion work properly. Having a fleet gathering more data is more important than having more sensors. Having to process LIDAR and radar produces a lot of bloat in the code and data pipelines. He predicts other companies will also drop these sensors in time. Mapping the world and keeping it up to date is much too expensive. You won’t change the world with this limitation, you need to focus on vision which is the most important. The roads are designed to be interpreted with vision.

utopcell 8mo ago
So the argument is pretty much: it should be sufficient to use vision only, and that it is too difficult / expensive to do otherwise.