8 ms·

> We believe that a vision-only system is ultimately all that is needed for full autonomy. Our AI-based software architecture has been increasingly reliant on c

by anonu 6y ago

> We believe that a vision-only system is ultimately all that is needed for full autonomy. Our AI-based software architecture has been increasingly reliant on cameras, to the point where radar is becoming unnecessary earlier than expected. As a result, our FSD team is fully focused on evolving to a vision-based autonomous system and we are nearly ready to switch the US market to Tesla Vision. This is a huge gamble IMHO. If you have access to LIDAR and RADAR, why not use them?

perryizgr8 6y ago
> If you have access to LIDAR and RADAR, why not use them? I think LIDAR is a very expensive part (~$10,000). Also, it is difficult to integrate with the car's design such that it does not make the car ugly. Tesla probably does not want to do a separate self-driving testing fleet, instead they want to use the existing Teslas on road to get training data. So they cannot use LIDAR in all cars sold to users. I am sure if fully functional LIDAR units magically start selling for $100-$500 Tesla will start using them.
_ph_ 6y ago
It is a gamble, but also betting on LIDAR is a gamble, as it could be a pit to fall in if it turns out not to be beneficial. The big benefit of LIDAR is, it gives you very good 3D data about your environment. Unfortunately, it completely misses a lot of information which is essential for driving and actually a lot of our traffic rules are based on things which you cannot detected with LIDAR: traffic signs, traffic lights and so on (you could detect the sign itself with high precision, but not what it reads). This means, LIDAR could just be additional sensor data augmenting the camera information. But it adds a huge cognitive load when combining the data. Besides the costs of the LIDAR system itself, it adds to the computational costs. Once the 3D-data from the cameras only is "good enough", you might get better results by spending the available compute resources rather on improving the analysis of the visual data alone. And the 3D data Tesla seems to be getting from the cameras seems to be pretty good already. The big elephant in the room isn't to recognize a road sign correctly, it is to decide, whether it applies to the car or a different lane, road or whether it is actually a burger king logo :). There LIDAR might be just a distraction. It will be interesting to see how the autonomic car race will pan out. Especially, when competitors are using different strategies. That als ensures, that not everyone is going into the wrong direction.
jillesvangurp 6y ago
Because that adds cost. The game here is producing a car that is cheap to manufacture that can drive itself without having to have a data center in the trunk and a large lidar bump on the roof. The gamble here is on the software, not the hardware. All the accidents that happen are not a failure of sensors not sensing but of software not detecting what the sensors captured. We're talking cars crashing into things on a clear day. Cars not recognizing situations that a human would recognize from the captured video. Etc. They need to solve that anyway. Regardless of whether they have lidar or not. Other companies like Waymo pushing the Lidar agenda are also behind on their schedules to launch autonomous driving. But at least they have actual driver less cars on the road right now driving actual passengers around in very select markets. Either way, this is more or less a greenfield market without a lot of contenders shipping actual products yet. So, Tesla can afford to spend more time developing this market. If they nail this somehow, eventually, it will be a big growth driver for them.
- fundatus 6y ago
  Not sure if the cost and the "lidar bump" will still be arguments in the future. Heck, the latest iPhone has a lidar sensor integrated! Surely with a lot less capabilities than what you would need in a car - but it proves that it can be built small and cheap. And I would say that more data from sensors that sense different things will always result in better decisions by autonomous driving software.
audunw 6y ago
This is just my personal belief: but I really don't think an AI advanced enough to actually do self-driving will have trouble getting all the information it needs from cameras. It seems to me that accurately determining signs, construction work, pedestrians, bikes, and even the intention/direction of these, is way harder than just reconstructing a 3D space from the cameras. I think the neural nets will have to do this internally anyway to be able to predict the direction a pedestrian is about to walk. So I think LIDAR may be a benefit for limited self-driving in the short term. If it's a benefit in the long term remains to be seen, but there's good arguments that Teslas gamble will pay off, as they're allowed to use all their focus on the part of the AI that's most important for daily driving.
_rpd 6y ago
Possibly patent constrained?
nojito 6y ago
Because cars are currently being sold as FSD ready so you can't ever release LIDAR/RADAR after the fact.
- LeoPanthera 6y ago
  Current Teslas are already fitted with radar, though Tesla wants to remove it from future models.
blamazon 6y ago
Unnecessary cost and complexity are the reasons Tesla has quoted before. One less sensor to buy, or replace when it breaks. I’m not convinced. There’s good reasons they’re the lone player in the self driving space with this vision only play.
- moduspol 6y ago
  > When radar and vision disagree, which one do you believe? Vision has much more precision, so better to double down on vision than do sensor fusion. [1] https://twitter.com/elonmusk/status/1380796939151704071 https://twitter.com/elonmusk/status/1380796939151704071
  - CamperBob2 6y ago
    How much precision do you need to keep from running into a truck?
  - nimithryn 6y ago
    As I’ve commented above, “conflicting sensors” doesn’t make much sense as an argument. If sensors A and B conflicted, you could just ignore sensor B and get performance at least as good as just having sensor A. Does sensor B have enough other benefits to outweigh the costs of adding it? I don’t know, but that’s the actual crux of this problem.
    - jiofih 6y ago
      Sensor B “enhancing” sensor A is the exact same situation as a sensor conflict. How do you tell them apart? You’re oversimplifying the issue as if it was a matter of simple addition.
      - nimithryn 6y ago
        I don’t see how “enhancing” and “conflict” are the same. As gkop pointed out, the two sensor types may have different ranges of view, or may agree about situations where just sensor A may be uncertain.
feifan 6y ago
I'm not entirely convinced it's a good idea, but one convincing argument I've seen is that sensor fusion is hard and it's not a priori clear which sensor to believe when they provide conflicting information. It's entirely possible that they get better predictive accuracy from vision-only data then in the analogous situations with vision+radar data.
- voqv 6y ago
  The industry has been sensor fusion for decades. Training Autopilot neural networks to hallucinate road parts is easy but doing uncertainty estimation on simple radar measurements is hard? Also you still have to fuse the data from the multiple cameras. I'm almost doubting my entire knowledge in the field now when all of a sudden everyone just accepts that "sensor fusion is hard" statement.
  - feifan 6y ago
    I'd imagine sensor fusion in this context isn't a singular problem right? It seems entirely plausible that for Tesla's system, it's straightforward to integrate cameras where the resolution is known and data is homogenous in a certain way, and that it's hard to integrate and use radar data into that system.
    - voqv 6y ago
      The Model 3 has a ARS-4B radar and these things usually already do perception for you, so the output is a list of vehicle velocity and pose. Given what I know about Autopilot , I don't see how is that hard to fuse. And I doubt the supplier of the radar (Continental) gave Tesla raw data access. I just think there's more to the story than we're getting. Impossible to tell for an outsider though.
- nimithryn 6y ago
  I don't find this argument convincing at all. Just ignore the lidar in situations where lidar and vision conflict and you have the original decision. They probably started without lidar due to the cost (Tesla is one of few companies actually selling cars to customers, who are sensitive to price). The price of lidar has come down, but I would guess it requires a fair amount of expensive rearchitecture to add it in now, and they haven't bitten the bullet yet.
ravedave5 6y ago
like 90%+ of living beings on the planet use eyes to navigate. Seems to work for them. None use LIDAR or RADAR, only a few use SONAR.
- ra7 6y ago
  Computers sometimes need additional aid to compensate for the lack of human brain.
- joakleaf 6y ago
  By that reasoning, shouldn’t the fastest living beings move around on wheels instead of legs? Just because something works for evolution/nature, it doesn’t mean that there isn’t a simpler/better engineering alternative. We have spend hopelessly many years trying to build machines that walk — Now Boston Dynamics seem to be there. But it is really complicated compared to wheels, and their machines are not running at 2-300 km/h.
  - vardump 6y ago
    Birds do exist and can reach speeds up to 200 mph (320 km/h).
    - vardump 6y ago
      Someone disputes that birds can fly fast and do visual navigation?
      - addison-lee 6y ago
        https://birdsarentreal.com/ https://birdsarentreal.com/

by anonu 6y ago

perryizgr8 6y ago

> If you have access to LIDAR and RADAR, why not use them? I think LIDAR is a very expensive part (~$10,000). Also, it is difficult to integrate with the car's design such that it does not make the car ugly. Tesla probably does not want to do a separate self-driving testing fleet, instead they want to use the existing Teslas on road to get training data. So they cannot use LIDAR in all cars sold to users. I am sure if fully functional LIDAR units magically start selling for $100-$500 Tesla will start using them.

_ph_ 6y ago

It is a gamble, but also betting on LIDAR is a gamble, as it could be a pit to fall in if it turns out not to be beneficial. The big benefit of LIDAR is, it gives you very good 3D data about your environment. Unfortunately, it completely misses a lot of information which is essential for driving and actually a lot of our traffic rules are based on things which you cannot detected with LIDAR: traffic signs, traffic lights and so on (you could detect the sign itself with high precision, but not what it reads). This means, LIDAR could just be additional sensor data augmenting the camera information. But it adds a huge cognitive load when combining the data. Besides the costs of the LIDAR system itself, it adds to the computational costs. Once the 3D-data from the cameras only is "good enough", you might get better results by spending the available compute resources rather on improving the analysis of the visual data alone. And the 3D data Tesla seems to be getting from the cameras seems to be pretty good already. The big elephant in the room isn't to recognize a road sign correctly, it is to decide, whether it applies to the car or a different lane, road or whether it is actually a burger king logo :). There LIDAR might be just a distraction. It will be interesting to see how the autonomic car race will pan out. Especially, when competitors are using different strategies. That als ensures, that not everyone is going into the wrong direction.

jillesvangurp 6y ago

Because that adds cost. The game here is producing a car that is cheap to manufacture that can drive itself without having to have a data center in the trunk and a large lidar bump on the roof. The gamble here is on the software, not the hardware. All the accidents that happen are not a failure of sensors not sensing but of software not detecting what the sensors captured. We're talking cars crashing into things on a clear day. Cars not recognizing situations that a human would recognize from the captured video. Etc. They need to solve that anyway. Regardless of whether they have lidar or not. Other companies like Waymo pushing the Lidar agenda are also behind on their schedules to launch autonomous driving. But at least they have actual driver less cars on the road right now driving actual passengers around in very select markets. Either way, this is more or less a greenfield market without a lot of contenders shipping actual products yet. So, Tesla can afford to spend more time developing this market. If they nail this somehow, eventually, it will be a big growth driver for them.

fundatus 6y ago
Not sure if the cost and the "lidar bump" will still be arguments in the future. Heck, the latest iPhone has a lidar sensor integrated! Surely with a lot less capabilities than what you would need in a car - but it proves that it can be built small and cheap. And I would say that more data from sensors that sense different things will always result in better decisions by autonomous driving software.

audunw 6y ago

This is just my personal belief: but I really don't think an AI advanced enough to actually do self-driving will have trouble getting all the information it needs from cameras. It seems to me that accurately determining signs, construction work, pedestrians, bikes, and even the intention/direction of these, is way harder than just reconstructing a 3D space from the cameras. I think the neural nets will have to do this internally anyway to be able to predict the direction a pedestrian is about to walk. So I think LIDAR may be a benefit for limited self-driving in the short term. If it's a benefit in the long term remains to be seen, but there's good arguments that Teslas gamble will pay off, as they're allowed to use all their focus on the part of the AI that's most important for daily driving.

_rpd 6y ago

Possibly patent constrained?

nojito 6y ago

Because cars are currently being sold as FSD ready so you can't ever release LIDAR/RADAR after the fact.

LeoPanthera 6y ago
Current Teslas are already fitted with radar, though Tesla wants to remove it from future models.

blamazon 6y ago

Unnecessary cost and complexity are the reasons Tesla has quoted before. One less sensor to buy, or replace when it breaks. I’m not convinced. There’s good reasons they’re the lone player in the self driving space with this vision only play.

moduspol 6y ago
> When radar and vision disagree, which one do you believe? Vision has much more precision, so better to double down on vision than do sensor fusion. [1] https://twitter.com/elonmusk/status/1380796939151704071 https://twitter.com/elonmusk/status/1380796939151704071
- CamperBob2 6y ago
  How much precision do you need to keep from running into a truck?
- nimithryn 6y ago
  As I’ve commented above, “conflicting sensors” doesn’t make much sense as an argument. If sensors A and B conflicted, you could just ignore sensor B and get performance at least as good as just having sensor A. Does sensor B have enough other benefits to outweigh the costs of adding it? I don’t know, but that’s the actual crux of this problem.
  - jiofih 6y ago
    Sensor B “enhancing” sensor A is the exact same situation as a sensor conflict. How do you tell them apart? You’re oversimplifying the issue as if it was a matter of simple addition.
    - nimithryn 6y ago
      I don’t see how “enhancing” and “conflict” are the same. As gkop pointed out, the two sensor types may have different ranges of view, or may agree about situations where just sensor A may be uncertain.

feifan 6y ago

I'm not entirely convinced it's a good idea, but one convincing argument I've seen is that sensor fusion is hard and it's not a priori clear which sensor to believe when they provide conflicting information. It's entirely possible that they get better predictive accuracy from vision-only data then in the analogous situations with vision+radar data.

voqv 6y ago
The industry has been sensor fusion for decades. Training Autopilot neural networks to hallucinate road parts is easy but doing uncertainty estimation on simple radar measurements is hard? Also you still have to fuse the data from the multiple cameras. I'm almost doubting my entire knowledge in the field now when all of a sudden everyone just accepts that "sensor fusion is hard" statement.
- feifan 6y ago
  I'd imagine sensor fusion in this context isn't a singular problem right? It seems entirely plausible that for Tesla's system, it's straightforward to integrate cameras where the resolution is known and data is homogenous in a certain way, and that it's hard to integrate and use radar data into that system.
  - voqv 6y ago
    The Model 3 has a ARS-4B radar and these things usually already do perception for you, so the output is a list of vehicle velocity and pose. Given what I know about Autopilot , I don't see how is that hard to fuse. And I doubt the supplier of the radar (Continental) gave Tesla raw data access. I just think there's more to the story than we're getting. Impossible to tell for an outsider though.
nimithryn 6y ago
I don't find this argument convincing at all. Just ignore the lidar in situations where lidar and vision conflict and you have the original decision. They probably started without lidar due to the cost (Tesla is one of few companies actually selling cars to customers, who are sensitive to price). The price of lidar has come down, but I would guess it requires a fair amount of expensive rearchitecture to add it in now, and they haven't bitten the bullet yet.

ravedave5 6y ago

like 90%+ of living beings on the planet use eyes to navigate. Seems to work for them. None use LIDAR or RADAR, only a few use SONAR.

ra7 6y ago
Computers sometimes need additional aid to compensate for the lack of human brain.
joakleaf 6y ago
By that reasoning, shouldn’t the fastest living beings move around on wheels instead of legs? Just because something works for evolution/nature, it doesn’t mean that there isn’t a simpler/better engineering alternative. We have spend hopelessly many years trying to build machines that walk — Now Boston Dynamics seem to be there. But it is really complicated compared to wheels, and their machines are not running at 2-300 km/h.
- vardump 6y ago
  Birds do exist and can reach speeds up to 200 mph (320 km/h).
  - vardump 6y ago
    Someone disputes that birds can fly fast and do visual navigation?
    - addison-lee 6y ago
      https://birdsarentreal.com/ https://birdsarentreal.com/