5 ms·

Would Musk's argument about sensor fusion have made any sense back when they were doing lots of hand-coded C++? I've been thinking maybe vision-only was a reas

Would Musk's argument about sensor fusion have made any sense back when they were doing lots of hand-coded C++? I've been thinking maybe vision-only was a reasonable decision, back when lidar was expensive and the software was hand-coded. Now it doesn't, because lidar is cheaper and the software is and end-to-end neural net, and additional sensors are just more inputs to the network which will learn to use them. But Tesla is locked in because of the promises they made to early FSD buyers.

general1726 1y ago
> and the software is and end-to-end neural net, Well considering Musk record for adherence to reality, question is if you can really believe that or if Musk thinks that this is happening or it is not happening at all and Musk is just making it up.
- DennisP 1y ago
  Given what's happening in the rest of the AI field, it seems pretty likely that it's correct. Plus various Tesla engineers have said the same thing, Tesla does have a very large AI training cluster, and FSD quality made a big jump when they claimed to deploy end-to-end.
Jyaif 1y ago
You've got it exactly backwards. When the software was hand-coded, having Lidar and a high-res map was vital. But if you have a good enough AI, sensors that replicate the human senses are all what is needed. The real question is: when will we get good enough AI that can be applied to all cars?
- DennisP 1y ago
  The question isn't whether it can be done eventually, but whether it would get safer faster with extra sensors. Back in the day, Elon's specific objection to lidar was that it was too hard to code the sensor fusion part. With end-to-end there's no coding to deal with. And it's not like they've replicated the visual cortex. It's the same neural net technology everybody else is using. It can deal with any sort of sensors just fine.
- CyberDildonics 1y ago
  > But if you have a good enough AI, sensors that replicate the human senses are all what is needed. What is your example or evidence for this?
  - Jyaif 1y ago
    Concentrated and well-rested drivers with good eyesight not under the influence are able to drive very well.
    - DennisP 1y ago
      Yes, but we're a long way from putting AI hardware in cars that's as powerful as the human brain. Plus our cameras aren't as good as human eyes.
    - CyberDildonics 1y ago
      So human senses are all we need and we just have to solve artificial general intelligence and fully simulate a human brain? I wonder why no one has done that yet.
dham 1y ago
It's not "Musk's argument," it's Andrej Karpathy's. Also, if you've ever done any ML you would note that more data isn't always better. Plus there's the piece about which thing to believe when you get conflicting data. It's a lot more to it than what random Hacker News people are saying in this thread.
- DennisP 1y ago
  Conflicting data is an issue even when you just have lidar, and it's not hard to deal with. The cofounder of Waymo taught one of the first Udacity courses on this subject. He went through a small Python project that processed lidar point clouds for self-driving. The data is noisy, you get conflicting information from different points, and the code aggregates all that into the most likely 3D model of the world. Additional sensor inputs are just more of the same, and neural nets are pretty good at this sort of thing. They'd even learn which sensors are more reliable in different scenarios. As for "more data isn't always better," I've mostly seen that applied to training, not inference in real-time control systems. Even for training, it turned out people had been fooled by a local maximum, and once past that, more data really was better.
AlotOfReading 1y ago
Sensor fusion is pretty straightforward. You can think of it like sorting algorithms in CS. There's a bunch of standard techniques simple enough to teach undergrads that work fine in production, and enough technical depth beyond them to last the rest of your career. If you actually look back at the E2E tweet, musk only says that the NN replaced 300k lines of "control code". Control code usually doesn't encompass the entire AV software stack, but neither should it take 300k LOC. As far as I'm aware no one is 100% sure what they mean by E2E and if it's actually the standard meaning or something else that's been widely misinterpreted.
- maxlin 1y ago
  Their engineers, who obviously are on the bleeding edge, going out of their way to avoid sensor fusion issues says something quite different. I could believe "Straightforward" could maybe apply for something many tiers below in complexity and safety requirements to what they're doing. But adding non-agreeing, non-uniform information sources to the most capable real-world ML vision system not driven by human-engineered code? I don't care even if you said you had 70% of the experience their team has, what you say can't sound reasonable or caring for actually improving safety in numbers.
  - sidibe 1y ago
    They go out of their way to avoid sensor fusions because they are not allowed to use other sensors. Some people seem to have no limits to their credulity.
    - maxlin 1y ago
      Yes. They are disallowing themselves from using other sensors. To avoid complicating the software stack and ultimately to keep it higher performing, which will make it safer in any alotted time. This is really no different from good code practices with a complex system. Most HN readers should be familiar with rot in codebases quite comparable to "adding a few extra features to make it better" which just became a maintenance burden and take away from core features. If you knew the slightest of how Musk has actually stayed exactly the same since the beginnings of Tesla, you'd know his hard specs are always technology-based. People that know more have said that his capability of speccing systems relatively right deep in to the future is possibly his single greatest leadership feat. >Once Musk is near-certain about one technology pathway over another, he’s not afraid to put massive amounts of resources into that path, while still staying flexible enough in the case that a new emerging technology disrupts that particular path. Because he’s willing to make enormous (and seemingly risky) bets on these pathways, he’s able to outpace his competitors. https://www.quora.com/Is-Elon-Musk-all-that-hes-cracked-up-to-be/answer/Leigh-Christie?ch=10&oid=103055305&share=5c111a86&srid=hql6j8&target_type=answer https://www.quora.com/Is-Elon-Musk-all-that-hes-cracked-up-t...

general1726 1y ago

> and the software is and end-to-end neural net, Well considering Musk record for adherence to reality, question is if you can really believe that or if Musk thinks that this is happening or it is not happening at all and Musk is just making it up.

DennisP 1y ago
Given what's happening in the rest of the AI field, it seems pretty likely that it's correct. Plus various Tesla engineers have said the same thing, Tesla does have a very large AI training cluster, and FSD quality made a big jump when they claimed to deploy end-to-end.

Jyaif 1y ago

You've got it exactly backwards. When the software was hand-coded, having Lidar and a high-res map was vital. But if you have a good enough AI, sensors that replicate the human senses are all what is needed. The real question is: when will we get good enough AI that can be applied to all cars?

DennisP 1y ago
The question isn't whether it can be done eventually, but whether it would get safer faster with extra sensors. Back in the day, Elon's specific objection to lidar was that it was too hard to code the sensor fusion part. With end-to-end there's no coding to deal with. And it's not like they've replicated the visual cortex. It's the same neural net technology everybody else is using. It can deal with any sort of sensors just fine.
CyberDildonics 1y ago
> But if you have a good enough AI, sensors that replicate the human senses are all what is needed. What is your example or evidence for this?
- Jyaif 1y ago
  Concentrated and well-rested drivers with good eyesight not under the influence are able to drive very well.
  - DennisP 1y ago
    Yes, but we're a long way from putting AI hardware in cars that's as powerful as the human brain. Plus our cameras aren't as good as human eyes.
  - CyberDildonics 1y ago
    So human senses are all we need and we just have to solve artificial general intelligence and fully simulate a human brain? I wonder why no one has done that yet.

dham 1y ago

It's not "Musk's argument," it's Andrej Karpathy's. Also, if you've ever done any ML you would note that more data isn't always better. Plus there's the piece about which thing to believe when you get conflicting data. It's a lot more to it than what random Hacker News people are saying in this thread.

DennisP 1y ago
Conflicting data is an issue even when you just have lidar, and it's not hard to deal with. The cofounder of Waymo taught one of the first Udacity courses on this subject. He went through a small Python project that processed lidar point clouds for self-driving. The data is noisy, you get conflicting information from different points, and the code aggregates all that into the most likely 3D model of the world. Additional sensor inputs are just more of the same, and neural nets are pretty good at this sort of thing. They'd even learn which sensors are more reliable in different scenarios. As for "more data isn't always better," I've mostly seen that applied to training, not inference in real-time control systems. Even for training, it turned out people had been fooled by a local maximum, and once past that, more data really was better.

AlotOfReading 1y ago

Sensor fusion is pretty straightforward. You can think of it like sorting algorithms in CS. There's a bunch of standard techniques simple enough to teach undergrads that work fine in production, and enough technical depth beyond them to last the rest of your career. If you actually look back at the E2E tweet, musk only says that the NN replaced 300k lines of "control code". Control code usually doesn't encompass the entire AV software stack, but neither should it take 300k LOC. As far as I'm aware no one is 100% sure what they mean by E2E and if it's actually the standard meaning or something else that's been widely misinterpreted.

maxlin 1y ago
Their engineers, who obviously are on the bleeding edge, going out of their way to avoid sensor fusion issues says something quite different. I could believe "Straightforward" could maybe apply for something many tiers below in complexity and safety requirements to what they're doing. But adding non-agreeing, non-uniform information sources to the most capable real-world ML vision system not driven by human-engineered code? I don't care even if you said you had 70% of the experience their team has, what you say can't sound reasonable or caring for actually improving safety in numbers.
- sidibe 1y ago
  They go out of their way to avoid sensor fusions because they are not allowed to use other sensors. Some people seem to have no limits to their credulity.
  - maxlin 1y ago
    Yes. They are disallowing themselves from using other sensors. To avoid complicating the software stack and ultimately to keep it higher performing, which will make it safer in any alotted time. This is really no different from good code practices with a complex system. Most HN readers should be familiar with rot in codebases quite comparable to "adding a few extra features to make it better" which just became a maintenance burden and take away from core features. If you knew the slightest of how Musk has actually stayed exactly the same since the beginnings of Tesla, you'd know his hard specs are always technology-based. People that know more have said that his capability of speccing systems relatively right deep in to the future is possibly his single greatest leadership feat. >Once Musk is near-certain about one technology pathway over another, he’s not afraid to put massive amounts of resources into that path, while still staying flexible enough in the case that a new emerging technology disrupts that particular path. Because he’s willing to make enormous (and seemingly risky) bets on these pathways, he’s able to outpace his competitors. https://www.quora.com/Is-Elon-Musk-all-that-hes-cracked-up-to-be/answer/Leigh-Christie?ch=10&oid=103055305&share=5c111a86&srid=hql6j8&target_type=answer https://www.quora.com/Is-Elon-Musk-all-that-hes-cracked-up-t...