7 ms·

The Camera Is the Lidar

200 points by nthuser 8y ago

dllu 8y ago
Relevant discussion with some comments from Angus on reddit: https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the_camera_is_the_lidar/ https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the...
thanatropism 8y ago
Is it just me, or are we seeing yet another impressive leap in Computer Vision that's soon going to be hyped as an incremental step into Skynet?
- 21 8y ago
  Unless you think Skynet is impossible, all impressive leaps are an incremental step towards Skynet.
  - wpietri 8y ago
    Not necessarily. My expectation is that Skynet is highly unlikely, a side branch we probably won't take. Think of the 1920s-1950s version of robots, for example. They were machines shaped like people and that acted like people. In retrospect, they seem not scary but silly. The human shape isn't particularly useful or easy to build; our most common robots are vacuums shaped like hockey pucks. Skynet is another "what if machines acted like people" fairy tale. It makes sense if you imagine yourself as a computer that wakes up; we wake up all the time, so it seems normal to us. But self-awareness and self-preservation are biological systems that evolved over very long time scales. Those are intricate systems, again not really useful or easy to build. And also not likely to randomly occur. It could be that we'll build those kinds of systems, of course. But I think it will take a long time to get them right, and then it's not really the skynet story, it's the mad scientist with the robot army story.
- qualitative 8y ago
  None of this technology is new. It's been done to death as terrain following guidance systems for cruise missiles, now reapplied in a civilian context. It's technology that already exists, but must be reinvented in a non-military context from scratch, since the tech transfer between weapons systems and civilian applications is likely locked up in policy. So, we know that this technology exists, and is proven, but we have to reinvent the wheel, because reasons. The reason we see this interminable slow motion public struggle to bring it to consumer applications, is likely because there are no controls in place that can actually prevent "contemporaneous discovery" wink, wink.
zerealshadowban 8y ago
This is so brilliant, I'm considering completely switching the type of software/hardware work I've been doing for the last decade.
amelius 8y ago
The problem is that deep learning should not be allowed in safety critical systems, because (1) the accuracy is always less than 100% even in known test situations, and (2) we don't know how it works and under what conditions it breaks down.
- MrQuincle 8y ago
  Delaying the roll out of algorithms if they achieve superhuman performance in avoiding accidents might not be the moral high ground...
- 21 8y ago
  The status quo is neural nets already are allowed in safety critical systems - humans. And with the amount of drivers snapchatting behind the wheel, I'd rather take my chances with a self-driving car.
  - jfoutz 8y ago
    > neural nets already are allowed in safety critical systems - humans. I’d really like to see a demonstration that human behavior is just neural nets. Sadly, I think that’s still an open question.
  - sorokod 8y ago
    Nothing you do will prevent a snapchatting driver from crushing into you.At that point I am sure you will observe with interest how a neural network deals with the situation.
- dllu 8y ago
  Every self driving car company is using neural networks. If neural nets are going to be used anyway, I'd rather have centimeter level range information per pixel than not. By the way, although the article focuses on deep learning, there are many applications that don't involve deep learning. For example, although you can run the deep neural network based SuperPoint on the intensity data, you can also run any classical feature extraction algorithm such as SIFT, SURF, ORB, BRISK, FAST, AGAST, etc. Doing so provides an elegant solution to the problem of localizing in a geometrically sparse but visually rich environment, such as a smooth but well-illuminated tunnel.
bsaul 8y ago
Something i don't understand : i've seen those kind of videos (i think it was nvidia or maybe waymo) where a signal (either camera or lidar or both) is processed in real-time and displays boxes around cars, street lights, etc. It seemed to be that detouring real-life objects from sensors in real-time has been working for a long time now. What does this bring that's new ?
- dllu 8y ago
  Estimating depth and 3D bounding boxes using cameras only, while possible, is much less accurate than using actual range data from a lidar. On the other hand if you fuse lidar and camera data such as with Waymo and others, there may be issues with the sensors being out of sync (as they run at different framerates, and the lidar continually spins) or physically offset (leading to parallax issues). Dealing with such issues is very difficult. Having a single sensor output both accurate range information and camera data makes it much nicer to work with.
barbegal 8y ago
I'm sure this works well in bright light but I'm sceptical that this can perform at all well on overcast days or at night. The OS-1 device spins at 10Hz and the LIDAR samples 2048 points over one 360 degree revolution. This means each column of pixels is sampled at 1/20480 of a second. To sample that fast requires a lot of light which is fine on a sunny day but on a cloudy day you can have 100 time less light. And at night you would have no ambient near infrared light at all.
- deleted 8y ago
  [deleted]
- post_break 8y ago
  Could you have a camera without the IR filter and then use IR leds to make up the difference? You would have to worry about other vehicles doing to same and blinding the cameras somehow but that might be a solution.
haney 8y ago
So I'm not up on the Lidar industry but $12k for a sensor seems really expensive, but then again from my casual observations Lidar is really expensive. Is there a physical / first principles reason this is true or is it just really new technology?
- newnewpdro 8y ago
  Presumably the price is currently dominated by the effects of low volume production. The market is r&d for new products. When those start going to mass production, then this part can also go to mass production, bringing the price down.
- wpietri 8y ago
  I'm not either, but I think there's no reason for them not to charge a lot of money. I expect goal is to advance their tech as rapidly as possible while claiming mindshare. The people buying this stuff are generally swimming in investor money, perceived value is often related to price, and it's much easier to lower prices than raise them. So as long as they're selling enough units to get useful real-world feedback that supports their development, in their shoes I'd basically gouge people.
- Animats 8y ago
  Each unit has 16 or 64 little laser rangefinders in it. They all have to have uniform response (which, from their pictures, Ouster hasn't achieved yet) and be lined up properly. Eventually somebody will develop high volume ways to do that, but in the prototype stage they're probably hand assembled. Image orthicons for TV cameras once cost $10,000 each, and color cameras needed three of them, plus another $50K or so of electronics to drive them. Today a cell phone camera costs about $10.
- deepnotderp 8y ago
  Equivalent Velodyne lidar (the ones commonly used) cost 64K each. And no, it's my understanding that it's just a matter of volume.
  - mhb 8y ago
Animats 8y ago
The hardware is described, vaguely, at [1]. It's a rotating drum scanner with 16 or 64 lasers. $12,000 for the 64-laser model. $24,000 for the upcoming model with 200m range. Still a long way from a useful auto part. There are about a dozen companies in this space now. Nobody has the price down yet. Continental, the European auto parts company, is the most likely winner. Quanergy made a lot of noise but didn't ship much.[2] There's a conference on automotive LIDAR this month in Detroit.[3] Many of the exhibitors are major semiconductor packaging companies, with various approaches to putting lots of little LIDAR units in a convenient package at a reasonable price. [1] https://www.ouster.io/faq/ https://www.ouster.io/faq/ [2] https://news.ycombinator.com/item?id=17755183 https://news.ycombinator.com/item?id=17755183 [3] http://www.automotivelidar.com/ http://www.automotivelidar.com/
- deepnotderp 8y ago
  Why Continental? Flash lidar seems to be a fundamentally broken concept to me wrt range.
  - Animats 8y ago
    One might think that, except that Advanced Scientific Concepts, which Continental bought, has had it working for a decade.[1] Their units work fine, but are expensive. They're mostly sold to DoD and used for space applications. The Space-X Dragon spacecraft uses one for docking. There's a tradeoff between field of view and range. Automotive systems will probably include a long-range narrow field of view unit and a shorter range wide field of view unit. Flash LIDAR has some advantages. No moving parts. Can be fabbed by semiconductor processes. The one big laser is separate from the sensor array, which helps with cooling. Also, you can spread the outgoing beam, which helps with eye safety. (Eye safety involves how much energy is in an eye iris sized, 1/4" or so, cross section of the beam. If the beam is spread out, energy density is lower.) [1] https://ieeexplore.ieee.org/document/7268968/ https://ieeexplore.ieee.org/document/7268968/
    - deepnotderp 8y ago
      What's the power output for the unit in the paper?
flyinglizard 8y ago
Why is this better than separate LIDAR and camera? Because you're collecting NIR ambient light, your optics are wideband. Meaning that daylight would have a more pronounced negative effect on system range (easier to saturate the photocells). It's also low resolution (as most LIDARs are), and there is no color segmentation data. In an automotive application, I can't see a justification to unify both visual and LIDAR into a single sensor, rather than having an extrinsically calibrated array of sensors. You can improve the calibration out of the data over time if you're very concerned about system stability. It seems like a nice party trick, but the vehicle LIDAR game focuses on solid state long range units, as this will be what gets into mass production. The visual band imagers in the car are a given for many other reasons anyway.
- deepnotderp 8y ago
  It's better because there's no need for calibration, you always have perfect calibration. Solid state lidar has issues. The cofounder of Ouster, Angus Pacala previously cofounded Quanergy, a solid state lidar startup.
  - flyinglizard 8y ago
    Solid state LIDAR certainly has issues - but someone is going to solve those and this is what will get into automotive, definitely not $10k units with moving parts. There was an announcement on a cooperation between BMW and Innoviz (an Israeli maker of solid state LIDARs) with Magna being their OEM sponsor. I'm not sure calibration is that big of a deal for this application. Sensors are going to be calibrated and tested in the factory or at a module level regardless, and the accuracy requirements in automotive are much lower than consumer products using similar technology. You can't overcome not having colors (traffic lights, anyone?), limited ranging distance or sensor saturation due to ambient conditions.
  - jimduk 8y ago
    Agreed, spent time last year on a project fusing lidar and rotating LWIR (thermal) with some smart people and calibration took significant effort, mechanically, in electronic timing, and in algorithmic fusing. This looks like a nice step forward.
kumarvvr 8y ago
A semi-off topic question. Is it not possible to get an accurate depth map based on a two camera stereoscopic setup? Like human eyes? Perhaps combine it with video processing to isolate objects at different depths.
- Someone1234 8y ago
  It is and works great, see Subaru
- kyrra 8y ago
  I was watching a talk from Cruise that mentions this. The main problem with cameras is dynamic range. Dealing with different lighting conditions that can change quickly is hard (the sun is really good at washing out colors). Lidar doesn't care about the current lighting conditions. https://youtu.be/s-8cYj_eh8E?t=22m39s https://youtu.be/s-8cYj_eh8E?t=22m39s
  - sjwright 8y ago
    Also heavy rain would be a problem for regular cameras. Not just seeing through the airborne droplets, but also (at a guess far more significantly) the water directly in contact with the windscreen causing severe random distortions.
    - nikofeyn 8y ago
      but heavy rain and also snow are also problems for lidar.
    - sbr464 8y ago
      I built a hacky prototype, combining: - FLIR thermal camera - 3 different small cameras manually set at different settings, models chosen for their qualities handling light levels. Those 4 live feeds were fed into a small black magic design quad layout device, that turned them into a single hd feed via hardware/real-time. That was fed into a hardware capture, that stacked the quad arrangement, applied some other filters and did hardware compression. At that point almost no latency was introduced but had a nice working base video feed. That was fed into the Linux box for processing. The quad device created a sort of super hdr video, and the thermal layer took it to the next level. All of the cameras had drawbacks, but combined they were minimized.
sytelus 8y ago
The blog post is very badly written to figure out what really these folks are doing and what sets them apart. Here’s my guess: they are detecting laser bounces using modified camera and using that camera to generate visual image at the same time as point cloud. It’s still a mechanical moving lidar at 10hz, range is 120m and resolution of 2048 horizontal beams per 360-degree and 16 beams verticle looks pretty good, although not completely out of the league.
akeck 8y ago
Could hundreds of self driving cars cruising around a city with LIDAR affect the eyes of pedestrians?
- dllu 8y ago
  It would not affect the eyes of pedestrians. The Ouster OS-1 in the article, as well as all other automotive lidars that I know of, are class 1 laser eye-safe, meaning that it is safe even if you put your eye right up to it for hours. The power also decreases dramatically once you get far away from it, since the laser beams spend most of their time pointed in different directions, and the collimation is not perfect.
barrystaes 8y ago
I get really excited about this technology, yet the not-made-here-syndrome force is strong in this one. I wonder if theres an EU equivalent?
- dllu 8y ago
  There are many EU lidar companies. SICK, Ibeo, Pepperl+Fuchs, Osram, Innoluce, Blickfeld, and so on. However, none of those matches the capabilities of the Ouster OS-1 exactly.

200 points by nthuser 8y ago

dllu 8y ago

Relevant discussion with some comments from Angus on reddit: https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the_camera_is_the_lidar/ https://www.reddit.com/r/SelfDrivingCars/comments/9c60pe/the...

thanatropism 8y ago

Is it just me, or are we seeing yet another impressive leap in Computer Vision that's soon going to be hyped as an incremental step into Skynet?

21 8y ago
Unless you think Skynet is impossible, all impressive leaps are an incremental step towards Skynet.
- wpietri 8y ago
  Not necessarily. My expectation is that Skynet is highly unlikely, a side branch we probably won't take. Think of the 1920s-1950s version of robots, for example. They were machines shaped like people and that acted like people. In retrospect, they seem not scary but silly. The human shape isn't particularly useful or easy to build; our most common robots are vacuums shaped like hockey pucks. Skynet is another "what if machines acted like people" fairy tale. It makes sense if you imagine yourself as a computer that wakes up; we wake up all the time, so it seems normal to us. But self-awareness and self-preservation are biological systems that evolved over very long time scales. Those are intricate systems, again not really useful or easy to build. And also not likely to randomly occur. It could be that we'll build those kinds of systems, of course. But I think it will take a long time to get them right, and then it's not really the skynet story, it's the mad scientist with the robot army story.
qualitative 8y ago
None of this technology is new. It's been done to death as terrain following guidance systems for cruise missiles, now reapplied in a civilian context. It's technology that already exists, but must be reinvented in a non-military context from scratch, since the tech transfer between weapons systems and civilian applications is likely locked up in policy. So, we know that this technology exists, and is proven, but we have to reinvent the wheel, because reasons. The reason we see this interminable slow motion public struggle to bring it to consumer applications, is likely because there are no controls in place that can actually prevent "contemporaneous discovery" wink, wink.

zerealshadowban 8y ago

This is so brilliant, I'm considering completely switching the type of software/hardware work I've been doing for the last decade.

amelius 8y ago

The problem is that deep learning should not be allowed in safety critical systems, because (1) the accuracy is always less than 100% even in known test situations, and (2) we don't know how it works and under what conditions it breaks down.

MrQuincle 8y ago
Delaying the roll out of algorithms if they achieve superhuman performance in avoiding accidents might not be the moral high ground...
21 8y ago
The status quo is neural nets already are allowed in safety critical systems - humans. And with the amount of drivers snapchatting behind the wheel, I'd rather take my chances with a self-driving car.
- jfoutz 8y ago
  > neural nets already are allowed in safety critical systems - humans. I’d really like to see a demonstration that human behavior is just neural nets. Sadly, I think that’s still an open question.
- sorokod 8y ago
  Nothing you do will prevent a snapchatting driver from crushing into you.At that point I am sure you will observe with interest how a neural network deals with the situation.
dllu 8y ago
Every self driving car company is using neural networks. If neural nets are going to be used anyway, I'd rather have centimeter level range information per pixel than not. By the way, although the article focuses on deep learning, there are many applications that don't involve deep learning. For example, although you can run the deep neural network based SuperPoint on the intensity data, you can also run any classical feature extraction algorithm such as SIFT, SURF, ORB, BRISK, FAST, AGAST, etc. Doing so provides an elegant solution to the problem of localizing in a geometrically sparse but visually rich environment, such as a smooth but well-illuminated tunnel.

bsaul 8y ago

Something i don't understand : i've seen those kind of videos (i think it was nvidia or maybe waymo) where a signal (either camera or lidar or both) is processed in real-time and displays boxes around cars, street lights, etc. It seemed to be that detouring real-life objects from sensors in real-time has been working for a long time now. What does this bring that's new ?

dllu 8y ago
Estimating depth and 3D bounding boxes using cameras only, while possible, is much less accurate than using actual range data from a lidar. On the other hand if you fuse lidar and camera data such as with Waymo and others, there may be issues with the sensors being out of sync (as they run at different framerates, and the lidar continually spins) or physically offset (leading to parallax issues). Dealing with such issues is very difficult. Having a single sensor output both accurate range information and camera data makes it much nicer to work with.

barbegal 8y ago

I'm sure this works well in bright light but I'm sceptical that this can perform at all well on overcast days or at night. The OS-1 device spins at 10Hz and the LIDAR samples 2048 points over one 360 degree revolution. This means each column of pixels is sampled at 1/20480 of a second. To sample that fast requires a lot of light which is fine on a sunny day but on a cloudy day you can have 100 time less light. And at night you would have no ambient near infrared light at all.

deleted 8y ago
[deleted]
post_break 8y ago
Could you have a camera without the IR filter and then use IR leds to make up the difference? You would have to worry about other vehicles doing to same and blinding the cameras somehow but that might be a solution.

haney 8y ago

So I'm not up on the Lidar industry but $12k for a sensor seems really expensive, but then again from my casual observations Lidar is really expensive. Is there a physical / first principles reason this is true or is it just really new technology?

newnewpdro 8y ago
Presumably the price is currently dominated by the effects of low volume production. The market is r&d for new products. When those start going to mass production, then this part can also go to mass production, bringing the price down.
wpietri 8y ago
I'm not either, but I think there's no reason for them not to charge a lot of money. I expect goal is to advance their tech as rapidly as possible while claiming mindshare. The people buying this stuff are generally swimming in investor money, perceived value is often related to price, and it's much easier to lower prices than raise them. So as long as they're selling enough units to get useful real-world feedback that supports their development, in their shoes I'd basically gouge people.
Animats 8y ago
Each unit has 16 or 64 little laser rangefinders in it. They all have to have uniform response (which, from their pictures, Ouster hasn't achieved yet) and be lined up properly. Eventually somebody will develop high volume ways to do that, but in the prototype stage they're probably hand assembled. Image orthicons for TV cameras once cost $10,000 each, and color cameras needed three of them, plus another $50K or so of electronics to drive them. Today a cell phone camera costs about $10.
deepnotderp 8y ago
Equivalent Velodyne lidar (the ones commonly used) cost 64K each. And no, it's my understanding that it's just a matter of volume.
- mhb 8y ago

Animats 8y ago

The hardware is described, vaguely, at [1]. It's a rotating drum scanner with 16 or 64 lasers. $12,000 for the 64-laser model. $24,000 for the upcoming model with 200m range. Still a long way from a useful auto part. There are about a dozen companies in this space now. Nobody has the price down yet. Continental, the European auto parts company, is the most likely winner. Quanergy made a lot of noise but didn't ship much.[2] There's a conference on automotive LIDAR this month in Detroit.[3] Many of the exhibitors are major semiconductor packaging companies, with various approaches to putting lots of little LIDAR units in a convenient package at a reasonable price. [1] https://www.ouster.io/faq/ https://www.ouster.io/faq/ [2] https://news.ycombinator.com/item?id=17755183 https://news.ycombinator.com/item?id=17755183 [3] http://www.automotivelidar.com/ http://www.automotivelidar.com/

deepnotderp 8y ago
Why Continental? Flash lidar seems to be a fundamentally broken concept to me wrt range.
- Animats 8y ago
  One might think that, except that Advanced Scientific Concepts, which Continental bought, has had it working for a decade.[1] Their units work fine, but are expensive. They're mostly sold to DoD and used for space applications. The Space-X Dragon spacecraft uses one for docking. There's a tradeoff between field of view and range. Automotive systems will probably include a long-range narrow field of view unit and a shorter range wide field of view unit. Flash LIDAR has some advantages. No moving parts. Can be fabbed by semiconductor processes. The one big laser is separate from the sensor array, which helps with cooling. Also, you can spread the outgoing beam, which helps with eye safety. (Eye safety involves how much energy is in an eye iris sized, 1/4" or so, cross section of the beam. If the beam is spread out, energy density is lower.) [1] https://ieeexplore.ieee.org/document/7268968/ https://ieeexplore.ieee.org/document/7268968/
  - deepnotderp 8y ago
    What's the power output for the unit in the paper?

flyinglizard 8y ago

Why is this better than separate LIDAR and camera? Because you're collecting NIR ambient light, your optics are wideband. Meaning that daylight would have a more pronounced negative effect on system range (easier to saturate the photocells). It's also low resolution (as most LIDARs are), and there is no color segmentation data. In an automotive application, I can't see a justification to unify both visual and LIDAR into a single sensor, rather than having an extrinsically calibrated array of sensors. You can improve the calibration out of the data over time if you're very concerned about system stability. It seems like a nice party trick, but the vehicle LIDAR game focuses on solid state long range units, as this will be what gets into mass production. The visual band imagers in the car are a given for many other reasons anyway.

deepnotderp 8y ago
It's better because there's no need for calibration, you always have perfect calibration. Solid state lidar has issues. The cofounder of Ouster, Angus Pacala previously cofounded Quanergy, a solid state lidar startup.
- flyinglizard 8y ago
  Solid state LIDAR certainly has issues - but someone is going to solve those and this is what will get into automotive, definitely not $10k units with moving parts. There was an announcement on a cooperation between BMW and Innoviz (an Israeli maker of solid state LIDARs) with Magna being their OEM sponsor. I'm not sure calibration is that big of a deal for this application. Sensors are going to be calibrated and tested in the factory or at a module level regardless, and the accuracy requirements in automotive are much lower than consumer products using similar technology. You can't overcome not having colors (traffic lights, anyone?), limited ranging distance or sensor saturation due to ambient conditions.
- jimduk 8y ago
  Agreed, spent time last year on a project fusing lidar and rotating LWIR (thermal) with some smart people and calibration took significant effort, mechanically, in electronic timing, and in algorithmic fusing. This looks like a nice step forward.

kumarvvr 8y ago

A semi-off topic question. Is it not possible to get an accurate depth map based on a two camera stereoscopic setup? Like human eyes? Perhaps combine it with video processing to isolate objects at different depths.

Someone1234 8y ago
It is and works great, see Subaru
kyrra 8y ago
I was watching a talk from Cruise that mentions this. The main problem with cameras is dynamic range. Dealing with different lighting conditions that can change quickly is hard (the sun is really good at washing out colors). Lidar doesn't care about the current lighting conditions. https://youtu.be/s-8cYj_eh8E?t=22m39s https://youtu.be/s-8cYj_eh8E?t=22m39s
- sjwright 8y ago
  Also heavy rain would be a problem for regular cameras. Not just seeing through the airborne droplets, but also (at a guess far more significantly) the water directly in contact with the windscreen causing severe random distortions.
  - nikofeyn 8y ago
    but heavy rain and also snow are also problems for lidar.
  - sbr464 8y ago
    I built a hacky prototype, combining: - FLIR thermal camera - 3 different small cameras manually set at different settings, models chosen for their qualities handling light levels. Those 4 live feeds were fed into a small black magic design quad layout device, that turned them into a single hd feed via hardware/real-time. That was fed into a hardware capture, that stacked the quad arrangement, applied some other filters and did hardware compression. At that point almost no latency was introduced but had a nice working base video feed. That was fed into the Linux box for processing. The quad device created a sort of super hdr video, and the thermal layer took it to the next level. All of the cameras had drawbacks, but combined they were minimized.

sytelus 8y ago

The blog post is very badly written to figure out what really these folks are doing and what sets them apart. Here’s my guess: they are detecting laser bounces using modified camera and using that camera to generate visual image at the same time as point cloud. It’s still a mechanical moving lidar at 10hz, range is 120m and resolution of 2048 horizontal beams per 360-degree and 16 beams verticle looks pretty good, although not completely out of the league.

akeck 8y ago

Could hundreds of self driving cars cruising around a city with LIDAR affect the eyes of pedestrians?

dllu 8y ago
It would not affect the eyes of pedestrians. The Ouster OS-1 in the article, as well as all other automotive lidars that I know of, are class 1 laser eye-safe, meaning that it is safe even if you put your eye right up to it for hours. The power also decreases dramatically once you get far away from it, since the laser beams spend most of their time pointed in different directions, and the collimation is not perfect.

barrystaes 8y ago

I get really excited about this technology, yet the not-made-here-syndrome force is strong in this one. I wonder if theres an EU equivalent?

dllu 8y ago
There are many EU lidar companies. SICK, Ibeo, Pepperl+Fuchs, Osram, Innoluce, Blickfeld, and so on. However, none of those matches the capabilities of the Ouster OS-1 exactly.