6 ms·

Something I’ve been curious about with computer-vision-only versus LIDAR: snow. Now, I am naïve about the real intricacies of computer vision. But won’t snow b

Something I’ve been curious about with computer-vision-only versus LIDAR: snow. Now, I am naïve about the real intricacies of computer vision. But won’t snow be quite nasty for camera-only systems? It’s almost the physical definition of noise, as it’s random and reflective. Is there any research or experience with computer vision and dealing with a snowstorm? Because even human drivers aren’t great in that kind of low-vision scenario. (Why yes, I did grow up in the Midwest.)

artursapek 6y ago
I think the argument is that if a human with two eyeballs can do it, a computer with a bunch of cameras should be able to as well. I can drive in the snow.
- perardi 6y ago
  That’s…a supposition more than an argument.
  - fastball 6y ago
    How is that not an argument? A human can drive in the snow, and has a sensing package which effectively consists of two cameras. The hard part (when you only have two cameras) is that the processor needs to be exceedingly sophisticated, e.g. a human brain. People like Waymo have decided the best way to decrease the brain requirement is to add more sophisticated sensing that humans are fundamentally incapable of. Tesla's strategy is more to just build a better digital driving brain with machine learning + lots of miles, because arguably all sensing mechanisms have their shortcomings. That being said, they still have a lot more than two cameras in their FSD package – just not LIDAR. They still have radar and other things as well. Only time will tell which approach is better. Probably in the end the best approach will be to combine the two: a brain that is sophisticated enough it can drive with just cameras, but you throw LIDAR and such in their too to go the extra mile.
    - Skunkleton 6y ago
      > A human can drive in the snow, and has a sensing package which effectively consists of two cameras. Two cameras, and a huge amount of mostly not understood wetware interpreting their signals. Our current approaches to ML may be a local maxima. The true path to AGI be somewhere as yet undiscovered. Lidar certainly seems like a good tool to use given our current ML capabilities.
      - fastball 6y ago
        Our current ML capabilities involve throwing as much training data at the problem as humanly possible. No LIDAR means more training data, as has been made clear by the amount of miles driven by Waymo vehicles vs Teslas in Autopilot.
    - Barrin92 6y ago
      it's a strange philosophy. Isn't the point of building machines that they can do things that humans can't? It's like building drones for warfare and turning the heat-seeking off because humans don't have that capacity. A generally intelligent driving system is going to benefit from better sensory inputs just like a human would, it's not like that makes the system dumber. The whole dichotomy between sensory inputs and processing seems strange to begin with, there's not necessary a clear line, or a reason why one should be prioritized over the other.
      - fastball 6y ago
        We are not talking about generally intelligent driving systems though. You have to realize that most SOTA self-driving systems are using a lot of NNs. NNs in turn want lots of training data. That training is not fully generic – in other words, if you train a NN using only cameras, giving LIDAR to that trained network (i.e. for inference) will not really do anything. Having large LIDAR arrays on the top of vehicles is expensive and slow to deploy, which directly translates to being able to generate less training data. So if your goal is to have the best possible self-driving vehicles 100 years from now, absolutely you should be using as much sophisticated sensing equipment you can get your hands on, and slowly train your nets for the next century. If your goal is to get out something that is better than a human driver as fast as possible, then it is not clear that requiring all vehicles to have LIDAR on them is the best way to go.
- hilbertseries 6y ago
  This is a terrible argument. You’re basically saying that computers can do anything humans can, which so far has not been the case.
  - jsight 6y ago
    Saying that computer vision can view the world with snow present is hardly the same as claiming that computers can do literally anything a human can. On the contrary, having computer vision capable of handling snow is a necessary precondition for self-driving, regardless of what other sensors are present.
  - Dylan16807 6y ago
    Not "anything". Just taking noisy camera data and turning it into a de-noised picture.
    - tylerhou 6y ago
      > Just taking noisy camera data and turning it into a de-noised picture. I think this is a huge simplification of the actual problem. Not only do you have to denoise the data, you have to use the data to generate models of 3D objects in space. Maybe you could argue that exact 3D models are not necessary, but IMO any neural network that can drive at Level 4/5 needs to approximate 3D space in some layer. This is incredibly difficult; we have an entire region of the brain dedicated to for this exact purpose (the visual cortex). This region has the benefit of millions of years of evolution. We also have the advantage that we can predict 3D models based on everyday experience; we don't know how to encode that knowledge into a neural network efficiently. That's not even mentioning the latency requirements that driving in the real world would impose on such a network.
      - Dylan16807 6y ago
        You have to do all that hard stuff with or without snow. The difficulty imposed by snowfall itself is just de-noising.
  - Sohcahtoa82 6y ago
- grumple 6y ago
  Humans certainly do drive in the snow, but not well. I've driven without being able to see the road (very slowly). This is not a safe situation. This is actually a case where I think computers (with LIDAR or other advanced sensors) will be able to outperform humans and it won't be close. I agree with the posters that say the anti-LIDAR stance is not going to work out in the long run.
  - AlotOfReading 6y ago
    Musk is not only anti-LIDAR, he's anti-HD mapping. Imagine if you had a centimeter-accurate map overlaid on your windscreen. You'd probably be able to operate the vehicle right up to the limits of the conditions. It definitely won't be close in many situations.
- alacombe 6y ago
  > I can drive in the snow More often than not, snow driving is CHALLENGING. And it's not for a lack of experience, as I'm doing multiple roadtrips through the BC Coquihalla Highway. More often than now, it's more of a matter of best attempt at driving in the path made by other car disregarding lines on the road than driving by the book. The utter HELL being driving at night with a blizzard, this is a level-up experience for sure.
  - ghaff 6y ago
    And before anyone says, well don't do that: 1. Many people have no choice but to drive in bad snow conditions for any number of reasons including that they're emergency workers. 2.) Winter conditions catch you. Even though I don't commute and basically do everything to avoid bad winter weather conditions, I've still been caught in pretty bad conditions a couple times over the past few years.
    - alacombe 6y ago
      > And before anyone says, well don't do that: Fuck them... Is that how much a pussy society we have become ? Avoid dangerous conditions because "it might be dangerous" ? If a freaking Tesla can't handle what my 06 SUV can handle without issue, and what people were handling ~forever, ... just as much as an Apple laptop can't handle what a then-IBM was handling, then the damn "car"/"laptop" is nothing but a piece of shit. Urbanus homo sapiens sapiens is really a pussy...
      - mewpmewp2 6y ago
        That's a ridiculous statement. Having a self driving car that works in only certain conditions can still be a benefit for the society. It won't benefit you, but it will benefit thousands of others.
        alacombe 6y ago
        In the mean time, every Texan EV owner is stuck because their brand new 50k Tesla can drive in the snow.
andy_ppp 6y ago
How fast do humans drive in heavy snow? I think most people drive extremely slowly to allow themselves to filter out the snow noise by taking in more frames, this will be possible with AI too. It's worth noting there are many accidents caused by people driving in snow and that AI could be hugely safer at some point. And also having to drive manually when it is snowing is probably fine for v1. I would worry more about bicycles, they can be extremely tricky if the rider doesn't understand how to explain their presence on the road to drivers.
- mjevans 6y ago
  I drive slower in the snow because of limited traction and the dangers of other drivers around me. If I had some button that made spikes or another traction method pop out of my tires and gave me great grip I probably would drive full speed in the snow; because the main issue would have been solved.
jsight 6y ago
Its a hard problem, but LIDAR has similar issues. Snow can cause noise for other sensors as well. In each case it seems like these issues are not insurmountable. I don't expect that any self-driving systems will start out with handling all weather conditions. Even in the long term they may handle fewer conditions than humans.
rorykoehler 6y ago
I suspect it will be able to drive as well as the best human drivers in snow. Why would it be any different? We use our eyes which are basically cameras.

artursapek 6y ago

I think the argument is that if a human with two eyeballs can do it, a computer with a bunch of cameras should be able to as well. I can drive in the snow.

perardi 6y ago
That’s…a supposition more than an argument.
- fastball 6y ago
  How is that not an argument? A human can drive in the snow, and has a sensing package which effectively consists of two cameras. The hard part (when you only have two cameras) is that the processor needs to be exceedingly sophisticated, e.g. a human brain. People like Waymo have decided the best way to decrease the brain requirement is to add more sophisticated sensing that humans are fundamentally incapable of. Tesla's strategy is more to just build a better digital driving brain with machine learning + lots of miles, because arguably all sensing mechanisms have their shortcomings. That being said, they still have a lot more than two cameras in their FSD package – just not LIDAR. They still have radar and other things as well. Only time will tell which approach is better. Probably in the end the best approach will be to combine the two: a brain that is sophisticated enough it can drive with just cameras, but you throw LIDAR and such in their too to go the extra mile.
  - Skunkleton 6y ago
    > A human can drive in the snow, and has a sensing package which effectively consists of two cameras. Two cameras, and a huge amount of mostly not understood wetware interpreting their signals. Our current approaches to ML may be a local maxima. The true path to AGI be somewhere as yet undiscovered. Lidar certainly seems like a good tool to use given our current ML capabilities.
    - fastball 6y ago
      Our current ML capabilities involve throwing as much training data at the problem as humanly possible. No LIDAR means more training data, as has been made clear by the amount of miles driven by Waymo vehicles vs Teslas in Autopilot.
  - Barrin92 6y ago
    it's a strange philosophy. Isn't the point of building machines that they can do things that humans can't? It's like building drones for warfare and turning the heat-seeking off because humans don't have that capacity. A generally intelligent driving system is going to benefit from better sensory inputs just like a human would, it's not like that makes the system dumber. The whole dichotomy between sensory inputs and processing seems strange to begin with, there's not necessary a clear line, or a reason why one should be prioritized over the other.
    - fastball 6y ago
      We are not talking about generally intelligent driving systems though. You have to realize that most SOTA self-driving systems are using a lot of NNs. NNs in turn want lots of training data. That training is not fully generic – in other words, if you train a NN using only cameras, giving LIDAR to that trained network (i.e. for inference) will not really do anything. Having large LIDAR arrays on the top of vehicles is expensive and slow to deploy, which directly translates to being able to generate less training data. So if your goal is to have the best possible self-driving vehicles 100 years from now, absolutely you should be using as much sophisticated sensing equipment you can get your hands on, and slowly train your nets for the next century. If your goal is to get out something that is better than a human driver as fast as possible, then it is not clear that requiring all vehicles to have LIDAR on them is the best way to go.
hilbertseries 6y ago
This is a terrible argument. You’re basically saying that computers can do anything humans can, which so far has not been the case.
- jsight 6y ago
  Saying that computer vision can view the world with snow present is hardly the same as claiming that computers can do literally anything a human can. On the contrary, having computer vision capable of handling snow is a necessary precondition for self-driving, regardless of what other sensors are present.
- Dylan16807 6y ago
  Not "anything". Just taking noisy camera data and turning it into a de-noised picture.
  - tylerhou 6y ago
    > Just taking noisy camera data and turning it into a de-noised picture. I think this is a huge simplification of the actual problem. Not only do you have to denoise the data, you have to use the data to generate models of 3D objects in space. Maybe you could argue that exact 3D models are not necessary, but IMO any neural network that can drive at Level 4/5 needs to approximate 3D space in some layer. This is incredibly difficult; we have an entire region of the brain dedicated to for this exact purpose (the visual cortex). This region has the benefit of millions of years of evolution. We also have the advantage that we can predict 3D models based on everyday experience; we don't know how to encode that knowledge into a neural network efficiently. That's not even mentioning the latency requirements that driving in the real world would impose on such a network.
    - Dylan16807 6y ago
      You have to do all that hard stuff with or without snow. The difficulty imposed by snowfall itself is just de-noising.
- Sohcahtoa82 6y ago
grumple 6y ago
Humans certainly do drive in the snow, but not well. I've driven without being able to see the road (very slowly). This is not a safe situation. This is actually a case where I think computers (with LIDAR or other advanced sensors) will be able to outperform humans and it won't be close. I agree with the posters that say the anti-LIDAR stance is not going to work out in the long run.
- AlotOfReading 6y ago
  Musk is not only anti-LIDAR, he's anti-HD mapping. Imagine if you had a centimeter-accurate map overlaid on your windscreen. You'd probably be able to operate the vehicle right up to the limits of the conditions. It definitely won't be close in many situations.
alacombe 6y ago
> I can drive in the snow More often than not, snow driving is CHALLENGING. And it's not for a lack of experience, as I'm doing multiple roadtrips through the BC Coquihalla Highway. More often than now, it's more of a matter of best attempt at driving in the path made by other car disregarding lines on the road than driving by the book. The utter HELL being driving at night with a blizzard, this is a level-up experience for sure.
- ghaff 6y ago
  And before anyone says, well don't do that: 1. Many people have no choice but to drive in bad snow conditions for any number of reasons including that they're emergency workers. 2.) Winter conditions catch you. Even though I don't commute and basically do everything to avoid bad winter weather conditions, I've still been caught in pretty bad conditions a couple times over the past few years.
  - alacombe 6y ago
    > And before anyone says, well don't do that: Fuck them... Is that how much a pussy society we have become ? Avoid dangerous conditions because "it might be dangerous" ? If a freaking Tesla can't handle what my 06 SUV can handle without issue, and what people were handling ~forever, ... just as much as an Apple laptop can't handle what a then-IBM was handling, then the damn "car"/"laptop" is nothing but a piece of shit. Urbanus homo sapiens sapiens is really a pussy...
    - mewpmewp2 6y ago
      That's a ridiculous statement. Having a self driving car that works in only certain conditions can still be a benefit for the society. It won't benefit you, but it will benefit thousands of others.
      - alacombe 6y ago
        In the mean time, every Texan EV owner is stuck because their brand new 50k Tesla can drive in the snow.

andy_ppp 6y ago

How fast do humans drive in heavy snow? I think most people drive extremely slowly to allow themselves to filter out the snow noise by taking in more frames, this will be possible with AI too. It's worth noting there are many accidents caused by people driving in snow and that AI could be hugely safer at some point. And also having to drive manually when it is snowing is probably fine for v1. I would worry more about bicycles, they can be extremely tricky if the rider doesn't understand how to explain their presence on the road to drivers.

mjevans 6y ago
I drive slower in the snow because of limited traction and the dangers of other drivers around me. If I had some button that made spikes or another traction method pop out of my tires and gave me great grip I probably would drive full speed in the snow; because the main issue would have been solved.

jsight 6y ago

Its a hard problem, but LIDAR has similar issues. Snow can cause noise for other sensors as well. In each case it seems like these issues are not insurmountable. I don't expect that any self-driving systems will start out with handling all weather conditions. Even in the long term they may handle fewer conditions than humans.

rorykoehler 6y ago

I suspect it will be able to drive as well as the best human drivers in snow. Why would it be any different? We use our eyes which are basically cameras.