10 ms·

Look inside iPad Pro 11s Lidar Scanner

148 points by _Microft 6y ago

throwaway122378 6y ago
What’s the consumer application for this?
ynniv 6y ago
This is the mass market application of PrimeSense's Time-of-flight sensor. They were first known by the Structured-light sensor used in the Xbox Kinect camera, but were later bought by Apple and used for Face ID. ToF is a more difficult technology than Structured-light, but the spatial and temporal resolutions are much higher. Rotating Lidar is also ToF, with lower spatial and temporal resolution. https://en.wikipedia.org/wiki/PrimeSense https://en.wikipedia.org/wiki/PrimeSense https://en.wikipedia.org/wiki/Time-of-flight_camera https://en.wikipedia.org/wiki/Time-of-flight_camera https://en.wikipedia.org/wiki/Structured-light_3D_scanner https://en.wikipedia.org/wiki/Structured-light_3D_scanner
- bigcorp-slave 6y ago
  Hi, I work in this field professionally - you are correct up until the tradeoffs. It is not the case that the spatial resolution is higher - in fact, semiconductor manufacturers have struggled to get ToF sensors much above VGA in mobile form factors. In general, ToF systems have a considerably lower maximum precision than structured light of the same resolution, especially at close range - typically structured light systems have quadratic error with range, but the blessing that that curse gives you is that at close ranges, precision goes up superlinearly with horizontal resolution. This is one reason that essentially all industrial 3D scanning is based on structured light. The other is multipath inference, which is specific to time of flight systems (and you can see the effects if you look at a ToF’s results near a corner - the exact corner itself will be correct, but the walls immediately adjacent to it will be pushed out away from the camera). Temporal resolution is more debatable, because ToF is a lot more conducive to “extremely short, extremely bright flash” than structured light is. But for example, there are systems that run structured light (specifically, single-shot dot pattern structured light, like that seen in TrueDepth or Kinect), or its two-camera equivalent (active stereo) at exposure times of 1ms or less. It is all about the optics stack and sensitivity of the camera. So I don’t agree that temporal resolution is a compelling advantage either. The main advantages of ToF are that it can be built in a single module (vs two for structured light), it does not require significant calibration in the field when the device is dropped or otherwise deformed, and it is easier to recover depth with decent edge quality. In general the software investment required for a good quality depth map is lower, though in this case Apple has been chewing on this for many years. Another potential advantage is outdoor performance at range - while historically this has been a significant weakness for ToFs, more modern ToFs adopted techniques to improve this, such as deeper pixel wells, brighter and shorter duration exposures, and built-in ambient light compensation. These are hard to do with structured light without manufacturing your own custom camera module. Finally - and I suspect this is why Apple ultimately picked it for their rear depth solution - because time of flight is a single module, it can be worked into the camera region on the rear of the device without having to have a separate opening for the illuminator and camera. The quadratic drop in accuracy with range that I mentioned above can be offset not just by resolution but by the distance between camera and illuminator - for a rear mounted device, the temptation is to make that baseline large, but this would put another hole in the back on the other side of the camera bump. I don’t see Apple going for that.
  - biglidar 6y ago
    You're wrong.
    - bigcorp-slave 6y ago
      This is the kind of high quality comment I come to hacker news for. Brave of you to make an account just to say this.
    - jjtheblunt 6y ago
      What part?
      - bigcorp-slave 6y ago
        The only part that I think could be debated is my assertion that it’s single module. While technically the illuminator and receiver are different components manufactured by different vendors, they are integrated into one module well before final assembly - and generally when you buy ToFs from semiconductor manufacturers, you get a single mostly rigid part that has everything attached to it. I suppose I could also imagine someone challenging my assertion that industrial 3D scanning uses structured light, because there are some vendors that use -temporal structured light-, where you flash different patterns to resolve different spatial frequencies on a stationary part. But beyond that I am as confused as you are.
        derefr 6y ago
        > where you flash different patterns to resolve different spatial frequencies on a stationary part Oh, that's neat! Is the output from the sensor, when combined into a time-series, essentially a frequency-domain image of the part, such that you just apply an inverse-FFT and get a picture out?
  - xfs 6y ago
    Can you comment on the tradeoffs between indirect TOF (phase) and direct TOF (time), what made Apple opt for direct TOF here, is it Microsoft's patents?
- rsp1984 6y ago
  Primesense never made a ToF sensor. When they were bought by Apple they had a Structured Light sensor on the market and apparently had a MEMS-mirror LiDAR sensor in the labs. Despite the name, ToF sensor typically means using modulated flood illumination and phase shift to compute the depth. It's conceptually very simple but suffers multi-path errors and poor efficiency. Real Time-of-Flight sensors (that measure depth by timing light) are typically called LiDAR. These typically collect many pointwise independent measurements. The LiDAR inside the iPad Pro and the iPhone Pro is solid state (no moving parts, not even a MEMS mirror). It was developed by Sony and they are now selling it to Apple.
  - ynniv 6y ago
    You are right regarding Sony. I thought the Primesense Carmine was ToF, but it was only a miniaturization of their SL. There weren't many high quality comments when I posted, so I filled in what I could remember from years ago. It's nice to see some current industry knowledge!
Geee 6y ago
So, what's the difference between this and those "real" LiDARS that are used in autonomous cars and robots? Such as https://velodynelidar.com https://velodynelidar.com which cost thousands of dollars.
- choppaface 6y ago
  Range and robustness to atmospheric conditions and sunlight.
johnflan 6y ago
I wonder will Apple move from structured light to dToF for Face ID, I’d imagine that would mean increased accuracy and a reduction in the component count.
wetwiper 6y ago
> Apple adopted the term to describe a new sensor that measures depth Not to be _that_ person, but someone should maybe inform the author that use of LiDAR has been around for quite some time already... the "sensor that measures depth" is pretty much a description of what LiDAR is used for and this isnt a new application of the term. And sensors to do this already exist so no "newness" in this field. I mean sure, if they were talking about it being included for consumer usage in everyday mobile devices, then maybe thats a new thing? But the wording of that phrase seems to want to credit Apple for something newly coined or invented, which isnt the case
- amelius 6y ago
  You can't win this fight. Soon, LiDAR will be Apple's invention. And besides, it already has a lowercase "i" in the name. It's similar to how Apple invented the tablet and the smartphone (though, curiously, they don't have any broad patents on these concepts).
  - HugoDaniel 6y ago
    In the meanwhile they also somehow managed to file patents for rectangular products with four evenly rounded corners[1]. My kitchen cutting board might fit the bill. [1] https://arstechnica.com/gadgets/2012/11/apple-awarded-design-patent-for-actual-rounded-rectangle/ https://arstechnica.com/gadgets/2012/11/apple-awarded-design...
    - threeseed 6y ago
      This is only true if you are lazy enough to not bother to read the patent or really understand how design patents work at all. Because you can't patent a rounded rectangle nor is that what Apple did. They patented the very specific design for the iPad.
      - CamperBob2 6y ago
        Regardless, it makes no sense to allow anyone to "own" rudimentary geometric design elements.
mmackh 6y ago
I’ve been using LiDAR to scan rooms and it has been incredibly accurate. Issues remaining are mirrors and water.
- 2sk21 6y ago
  Same here. I used an app called RoomScan Lidar on my iPad Pro to obtain very accurate plans of some rooms in my house that we are planning to renovate. Lidar is a real game changer for such applications.
  - abakker 6y ago
    I’m curious what the file types you get out of this system are? I’d very much like a 3D scanning solution that could accurately capture things like molding or other smaller architectural details. Is this system good enough on small parts?
    - mdorazio 6y ago
      No, the resolution isn't even close to good enough to get that kind of detail at distance. You would need a much more expensive (and larger) device or perhaps some approach using photogrammetry and proper lighting. To your first question, it spits out a point cloud that is then converted to a vertex-based mesh [1]. It's not in an easily exported format by default, but you could convert it into an OBJ (I'm sure someone already has an app or code snippet to do this).
arminiusreturns 6y ago
I don't want lidars mapping everything all the time, especially on systems I don't have root on... add one more sensor to the surveillance engine list I suppose.
- oliveshell 6y ago
  If that’s genuinely how you feel, you’re best off not carrying any sort of mobile phone at all. (Or at the very least, keep yours in a faraday bag until you need to use it.)
  - saurik 6y ago
    FWIW, it is also how I feel, and is similar to my thoughts on microphones and location logging and pretty much everything surrounding social networks; I still use iPhones and Facebook as the alternatives--not for any legitimate reason correlated with this property (it isn't a tradeoff or anything)--are much less effective due to vertical supply chain monopolization and network effects from the major oligopolies in hardware and service providers which would make me less effective, ironically, at my attempts to educate people about the downsides of this technology and achieve regulation on its usage and deployment... to me the real question is why you _don't_ think this is a problem :(.
    - JKCalhoun 6y ago
      > I still use iPhones and Facebook iPhones I think you can legitimately assume are at least trying to allow you control of your privacy. But Facebook? Not only are they actively trying to scrape your personal data, they're also so easy to drop entirely.
  - arminiusreturns 6y ago
    Fun fact, its not that hard to block those sensors and still use the phone... This sort of servile apathy is pathetic to me.
    - arminiusreturns 6y ago
k__ 6y ago
Do any other smartphone shops plan to add lidar to their android models?
- llampx 6y ago
  Many Android smartphones are available with "depth sensors" which tend to be useless. I'm not sure what the difference is between those and the iPhone lidar sensor but it has to be officially supported by Android if it's to be useful.
- solarkraft 6y ago
  IIRC LG had a "laser" on one of their G phones a few years ago to improve autofocus speed.
- maven29 6y ago
  The Note 10 5G and the Huawei P30 pro were the first with ToF on the rear in a mass market phone, I believe.
- thenthenthen 6y ago
  I had an Asus Zenphone AR. Used it to 3d scan environments, worked quite well, until the screen died (they all have this defect)
thdrdt 6y ago
For me it is a little confusing article, but it sound like Sony now has a chip that can emit infrared and in the same pixel measure the direct time of flight. Am I right? So props to Sony for making this possible in a, I assume, much smaller chip.
- djrogers 6y ago
  No, that’s not quite the case. FTA: ——- In EE Times’ interview, Sylvain Hallereau, senior technology and cost analyst at System Plus, explained that iPad Pro 11’s “LiDAR scanner” consists of an emitter — a vertical cavity surface emitting laser (VCSEL) from Lumentum, and a receptor — near infrared (NIR) CMOS image sensor that does direct measurement of time of flight, developed by Sony. —- So an emitter from Lumentum and a sensor from Sony.
lordnacho 6y ago
Is that the thing that lets you measure stuff with the camera? There's an app on my ipad that does that, though I'm not sure it works using one of these sensors. Bought it earlier this year. The app lets you point at stuff to measure the distance between things and seems to work.
- _Microft 6y ago
  The distance and area measuring tool seems to use the camera and IMU (inertial measurement unit = "accelerometer") as it also works on older iPhones which have no lidar sensor.
  - read_if_gay_ 6y ago
    AFAIK it’s augmented with lidar data if avilable. It feels more stable on my iPad with lidar than on my iPhone without.
someguydave 6y ago
Does anyone have specifications for the field of view that the iphone lidar sensor operates on?
nullc 6y ago
What's the state of the art in low cost (like .. ipad pro pricing or less) sensors for depth that could be strapped to an embedded Linux system? Are the best options still just stereo cameras or structured light solutions?

148 points by _Microft 6y ago

What’s the consumer application for this?

This is the mass market application of PrimeSense's Time-of-flight sensor. They were first known by the Structured-light sensor used in the Xbox Kinect camera, but were later bought by Apple and used for Face ID. ToF is a more difficult technology than Structured-light, but the spatial and temporal resolutions are much higher. Rotating Lidar is also ToF, with lower spatial and temporal resolution. https://en.wikipedia.org/wiki/PrimeSense https://en.wikipedia.org/wiki/PrimeSense https://en.wikipedia.org/wiki/Time-of-flight_camera https://en.wikipedia.org/wiki/Time-of-flight_camera https://en.wikipedia.org/wiki/Structured-light_3D_scanner https://en.wikipedia.org/wiki/Structured-light_3D_scanner

bigcorp-slave 6y ago
Hi, I work in this field professionally - you are correct up until the tradeoffs. It is not the case that the spatial resolution is higher - in fact, semiconductor manufacturers have struggled to get ToF sensors much above VGA in mobile form factors. In general, ToF systems have a considerably lower maximum precision than structured light of the same resolution, especially at close range - typically structured light systems have quadratic error with range, but the blessing that that curse gives you is that at close ranges, precision goes up superlinearly with horizontal resolution. This is one reason that essentially all industrial 3D scanning is based on structured light. The other is multipath inference, which is specific to time of flight systems (and you can see the effects if you look at a ToF’s results near a corner - the exact corner itself will be correct, but the walls immediately adjacent to it will be pushed out away from the camera). Temporal resolution is more debatable, because ToF is a lot more conducive to “extremely short, extremely bright flash” than structured light is. But for example, there are systems that run structured light (specifically, single-shot dot pattern structured light, like that seen in TrueDepth or Kinect), or its two-camera equivalent (active stereo) at exposure times of 1ms or less. It is all about the optics stack and sensitivity of the camera. So I don’t agree that temporal resolution is a compelling advantage either. The main advantages of ToF are that it can be built in a single module (vs two for structured light), it does not require significant calibration in the field when the device is dropped or otherwise deformed, and it is easier to recover depth with decent edge quality. In general the software investment required for a good quality depth map is lower, though in this case Apple has been chewing on this for many years. Another potential advantage is outdoor performance at range - while historically this has been a significant weakness for ToFs, more modern ToFs adopted techniques to improve this, such as deeper pixel wells, brighter and shorter duration exposures, and built-in ambient light compensation. These are hard to do with structured light without manufacturing your own custom camera module. Finally - and I suspect this is why Apple ultimately picked it for their rear depth solution - because time of flight is a single module, it can be worked into the camera region on the rear of the device without having to have a separate opening for the illuminator and camera. The quadratic drop in accuracy with range that I mentioned above can be offset not just by resolution but by the distance between camera and illuminator - for a rear mounted device, the temptation is to make that baseline large, but this would put another hole in the back on the other side of the camera bump. I don’t see Apple going for that.
- biglidar 6y ago
  You're wrong.
  - bigcorp-slave 6y ago
    This is the kind of high quality comment I come to hacker news for. Brave of you to make an account just to say this.
  - jjtheblunt 6y ago
    What part?
    - bigcorp-slave 6y ago
      The only part that I think could be debated is my assertion that it’s single module. While technically the illuminator and receiver are different components manufactured by different vendors, they are integrated into one module well before final assembly - and generally when you buy ToFs from semiconductor manufacturers, you get a single mostly rigid part that has everything attached to it. I suppose I could also imagine someone challenging my assertion that industrial 3D scanning uses structured light, because there are some vendors that use -temporal structured light-, where you flash different patterns to resolve different spatial frequencies on a stationary part. But beyond that I am as confused as you are.
      - derefr 6y ago
        > where you flash different patterns to resolve different spatial frequencies on a stationary part Oh, that's neat! Is the output from the sensor, when combined into a time-series, essentially a frequency-domain image of the part, such that you just apply an inverse-FFT and get a picture out?
- xfs 6y ago
  Can you comment on the tradeoffs between indirect TOF (phase) and direct TOF (time), what made Apple opt for direct TOF here, is it Microsoft's patents?
rsp1984 6y ago
Primesense never made a ToF sensor. When they were bought by Apple they had a Structured Light sensor on the market and apparently had a MEMS-mirror LiDAR sensor in the labs. Despite the name, ToF sensor typically means using modulated flood illumination and phase shift to compute the depth. It's conceptually very simple but suffers multi-path errors and poor efficiency. Real Time-of-Flight sensors (that measure depth by timing light) are typically called LiDAR. These typically collect many pointwise independent measurements. The LiDAR inside the iPad Pro and the iPhone Pro is solid state (no moving parts, not even a MEMS mirror). It was developed by Sony and they are now selling it to Apple.
- ynniv 6y ago
  You are right regarding Sony. I thought the Primesense Carmine was ToF, but it was only a miniaturization of their SL. There weren't many high quality comments when I posted, so I filled in what I could remember from years ago. It's nice to see some current industry knowledge!

Geee 6y ago

So, what's the difference between this and those "real" LiDARS that are used in autonomous cars and robots? Such as https://velodynelidar.com https://velodynelidar.com which cost thousands of dollars.

choppaface 6y ago
Range and robustness to atmospheric conditions and sunlight.

johnflan 6y ago

I wonder will Apple move from structured light to dToF for Face ID, I’d imagine that would mean increased accuracy and a reduction in the component count.

wetwiper 6y ago

> Apple adopted the term to describe a new sensor that measures depth Not to be _that_ person, but someone should maybe inform the author that use of LiDAR has been around for quite some time already... the "sensor that measures depth" is pretty much a description of what LiDAR is used for and this isnt a new application of the term. And sensors to do this already exist so no "newness" in this field. I mean sure, if they were talking about it being included for consumer usage in everyday mobile devices, then maybe thats a new thing? But the wording of that phrase seems to want to credit Apple for something newly coined or invented, which isnt the case

amelius 6y ago
You can't win this fight. Soon, LiDAR will be Apple's invention. And besides, it already has a lowercase "i" in the name. It's similar to how Apple invented the tablet and the smartphone (though, curiously, they don't have any broad patents on these concepts).
- HugoDaniel 6y ago
  In the meanwhile they also somehow managed to file patents for rectangular products with four evenly rounded corners[1]. My kitchen cutting board might fit the bill. [1] https://arstechnica.com/gadgets/2012/11/apple-awarded-design-patent-for-actual-rounded-rectangle/ https://arstechnica.com/gadgets/2012/11/apple-awarded-design...
  - threeseed 6y ago
    This is only true if you are lazy enough to not bother to read the patent or really understand how design patents work at all. Because you can't patent a rounded rectangle nor is that what Apple did. They patented the very specific design for the iPad.
    - CamperBob2 6y ago
      Regardless, it makes no sense to allow anyone to "own" rudimentary geometric design elements.

mmackh 6y ago

I’ve been using LiDAR to scan rooms and it has been incredibly accurate. Issues remaining are mirrors and water.

2sk21 6y ago
Same here. I used an app called RoomScan Lidar on my iPad Pro to obtain very accurate plans of some rooms in my house that we are planning to renovate. Lidar is a real game changer for such applications.
- abakker 6y ago
  I’m curious what the file types you get out of this system are? I’d very much like a 3D scanning solution that could accurately capture things like molding or other smaller architectural details. Is this system good enough on small parts?
  - mdorazio 6y ago
    No, the resolution isn't even close to good enough to get that kind of detail at distance. You would need a much more expensive (and larger) device or perhaps some approach using photogrammetry and proper lighting. To your first question, it spits out a point cloud that is then converted to a vertex-based mesh [1]. It's not in an easily exported format by default, but you could convert it into an OBJ (I'm sure someone already has an app or code snippet to do this).

arminiusreturns 6y ago

I don't want lidars mapping everything all the time, especially on systems I don't have root on... add one more sensor to the surveillance engine list I suppose.

oliveshell 6y ago
If that’s genuinely how you feel, you’re best off not carrying any sort of mobile phone at all. (Or at the very least, keep yours in a faraday bag until you need to use it.)
- saurik 6y ago
  FWIW, it is also how I feel, and is similar to my thoughts on microphones and location logging and pretty much everything surrounding social networks; I still use iPhones and Facebook as the alternatives--not for any legitimate reason correlated with this property (it isn't a tradeoff or anything)--are much less effective due to vertical supply chain monopolization and network effects from the major oligopolies in hardware and service providers which would make me less effective, ironically, at my attempts to educate people about the downsides of this technology and achieve regulation on its usage and deployment... to me the real question is why you _don't_ think this is a problem :(.
  - JKCalhoun 6y ago
    > I still use iPhones and Facebook iPhones I think you can legitimately assume are at least trying to allow you control of your privacy. But Facebook? Not only are they actively trying to scrape your personal data, they're also so easy to drop entirely.
- arminiusreturns 6y ago
  Fun fact, its not that hard to block those sensors and still use the phone... This sort of servile apathy is pathetic to me.
  - arminiusreturns 6y ago

k__ 6y ago

Do any other smartphone shops plan to add lidar to their android models?

llampx 6y ago
Many Android smartphones are available with "depth sensors" which tend to be useless. I'm not sure what the difference is between those and the iPhone lidar sensor but it has to be officially supported by Android if it's to be useful.
solarkraft 6y ago
IIRC LG had a "laser" on one of their G phones a few years ago to improve autofocus speed.
maven29 6y ago
The Note 10 5G and the Huawei P30 pro were the first with ToF on the rear in a mass market phone, I believe.
thenthenthen 6y ago
I had an Asus Zenphone AR. Used it to 3d scan environments, worked quite well, until the screen died (they all have this defect)

thdrdt 6y ago

For me it is a little confusing article, but it sound like Sony now has a chip that can emit infrared and in the same pixel measure the direct time of flight. Am I right? So props to Sony for making this possible in a, I assume, much smaller chip.

djrogers 6y ago
No, that’s not quite the case. FTA: ——- In EE Times’ interview, Sylvain Hallereau, senior technology and cost analyst at System Plus, explained that iPad Pro 11’s “LiDAR scanner” consists of an emitter — a vertical cavity surface emitting laser (VCSEL) from Lumentum, and a receptor — near infrared (NIR) CMOS image sensor that does direct measurement of time of flight, developed by Sony. —- So an emitter from Lumentum and a sensor from Sony.

lordnacho 6y ago

Is that the thing that lets you measure stuff with the camera? There's an app on my ipad that does that, though I'm not sure it works using one of these sensors. Bought it earlier this year. The app lets you point at stuff to measure the distance between things and seems to work.

_Microft 6y ago
The distance and area measuring tool seems to use the camera and IMU (inertial measurement unit = "accelerometer") as it also works on older iPhones which have no lidar sensor.
- read_if_gay_ 6y ago
  AFAIK it’s augmented with lidar data if avilable. It feels more stable on my iPad with lidar than on my iPhone without.

someguydave 6y ago

Does anyone have specifications for the field of view that the iphone lidar sensor operates on?

nullc 6y ago

What's the state of the art in low cost (like .. ipad pro pricing or less) sensors for depth that could be strapped to an embedded Linux system? Are the best options still just stereo cameras or structured light solutions?