7 ms·

Generating aerial imagery with your iPhone's Lidar sensor

267 points by jakecopp 4y ago

jakecopp 4y ago
I wrote this guide on creating orthorectified imagery using just an iPhone, using a demo on George St, Sydney! Usually this is easier with drones, but there are a lot of places you can't fly drones. I hope this guide comes in helpful, especially if you're contributing to OpenStreetMap. I've found you can strap an iPhone to the front of a bicycle and if you go slowly the 3D models/LiDAR point clouds are quite good (it has a 5m range). There are drift issues if you try to complete loops though.
- ramraj07 4y ago
  Sites down btw
  - jakecopp 4y ago
    Thanks, fixing now Archive at https://web.archive.org/web/20230314002734/https://jakecoppinger.com/2023/03/generating-aerial-imagery-with-your-iphones-lidar-sensor/ https://web.archive.org/web/20230314002734/https://jakecoppi...
    - jakecopp 4y ago
      Fixed.
- Chilko 4y ago
  Hi, this sounds really interesting but I am unable to read the guide as I am getting a 524 Timeout error.
lucb1e 4y ago
Note this is not aerial imagery shot from an aircraft with an Apple phone, this is someone with an Apple phone on their head cycling around the neighborhood and then viewing the 3d model from above. No less cool, but I was rather confused for a lot of the article, wondering what in the world the range on this sensor is (usually you get a few meters on normal-priced sensors, not hundreds of meters, and the picture showed entire streets being captured). The article actually mentions the maximum range is 5 metres after you set it to low confidence.
- jakecopp 4y ago
  Thanks for this - I'll update it to try and be more clear! This capture was just holding an iPhone in my hand - but you can place it on the front of a bicycle too. The Apple ARKit part handles the SLAM [1] of fusing the camera/LiDAR/gyro and possibly GPS. [1]: https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping https://en.wikipedia.org/wiki/Simultaneous_localization_and_...
  - lucb1e 4y ago
    > I'll update it to try and be more clear! Not an issue of yours, it's all a matter of context -- and how closely one reads the article while trying to get an idea of what it's about. I just happen to be in the OSM ecosystem deep enough to be very familiar with the word aerial (as opposed to other types of area photographies), but not deep enough into the Apple sphere to know the capabilities of these devices and realize what the title clearly must mean. It might just be a me-issue :)
  - trafficante 4y ago
    This is really cool, thanks for sharing. Do you know the functional limit of the iPhone’s LiDAR? Ie: could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever?
    - jakecopp 4y ago
      > could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever? Yep that should be possible - as long as you flow below 5 metres though! If you have a drone and are somewhere that is legal & safe to fly, photogrammetry (3d model from just visual photos) will likely give a better result than relying on the iPhone LiDAR stitching. Set the drone image self timer to 2 seconds, fly around for a bit and then upload all the photos into WebODM: https://opendronemap.org/webodm/ https://opendronemap.org/webodm/ You can run it on your local machine (see instructions in blog post) or hosted for a fee. It creates brilliant models and is a lot of fun!
      - 4y ago
willio58 4y ago
Ever since iPhone introduced Lidar, I've wondered why Teslas got rid of it. They said it was due to additional cost and complexity in models. Of course Waymo heavily uses Lidar but they have huge sensors they mount on cars to do so. Why is that lidar so different? What's stopping Tesla from adding the same sensors the iphone has, but all over the car and stitching it together like it does video from cameras? Is it just to do with distance? Refresh rate? Would love someone who knows this to teach me!
- SomeHacker44 4y ago
  AFAIK Tesla never had lidar. They had and will again have simple radar.
- Melatonic 4y ago
  Lidar I believe can be heavily affected by weather and might also require a lot of expensive sensors. It works great for VFX work when you need a precise model of an area. But I would guess the refresh rate is also not amazing. I live near Waymo and see the cars all the time - they do have huge things on top but the weather in Santa Monica is also very good. Not sure if I have seen them ever driving in the rain?
  - zxexz 4y ago
    Interesting. I’ve used a Velodyne VLP-32 in reasonably heavy rain with reasonable performance after some error correction. I was on foot and not driving, and probably only for a few hours total. But it seems to me that a big company with serious engineering talent should be able to make it work.
    - throwbadubadu 4y ago
      Yes.. any other car manufacturer I know of (likely not complete) is considering also Lidar sensors... they are still pricey and huge, but there are also MEMS-like improvements (likely incorrect terminology) and also costs get continously lower. At the same time advancements also happen at the multi-model side, not fusing different deep learning outputs afterwards but right in the one model (not judging if that is a good idea... but the steps taking currently seem to get bigger and better). Lastly, also camera has degraded performance under certain weather conditions.. some even say our eyes can be prone to that! :D So that's just all the challenges that need to be taken. Yes, humans can drive with their eyes only, but is that perfect? Sometimes we e.g. would also benefit greatly from radar/lidar, as e.g. in dark conditions our judgement of distances is really flawed. Refresh rate of some of this already aged car lidars was like 10 Hz, good enough I'd say? And it is truly amazing what details one can get out of those when getting close and move around in those point clouds.
bilsbie 4y ago
I’m questioning my reading comprehension but what’s it telling me to do?
pimlottc 4y ago
Feedback: on mobile, I thought the call-to-subscribe block was the end of the page. I didn’t even see the table of contents the first time.
- jakecopp 4y ago
  Thanks for that - I've heard that feedback once before. I've changed the h2 to an h3. Do you have any tips for how I could make it feel less like the end of the page?
- kejaed 4y ago
  Same here. I’d put the table of contents at the very top of the page, so that I know to expect the whole thing. I’d make the div for the subscribe block a different colour to make it stand out from the rest of the page. An href link with “continue article…” that jumped down to the next section could help too. FWIW I almost left the page and missed a really good read because of this issue as well. Glad I didn’t!
  - jakecopp 4y ago
    Thanks! I've moved the table of contents to be just below the images (so the image and intro is above the fold), and "read below" text to the signup. I'll improve the background/design of the subscribe block when I have some free time.
jonas21 4y ago
Have you tried Scaniverse for the capture step? It does a decent job at correcting for drift and avoiding misalignments for these sorts of scans. For example: https://sketchfab.com/3d-models/babakawa-urban-design-project-performance4-e5481c77ed4e4bceba1d48afad79b99a https://sketchfab.com/3d-models/babakawa-urban-design-projec... https://sketchfab.com/3d-models/holiday-villa-iphone-3d-scan-bcd1727b23ba468080d63d6c3f59e00b https://sketchfab.com/3d-models/holiday-villa-iphone-3d-scan... https://sketchfab.com/3d-models/street-art-on-rue-chapleau-montreal-2266ed5e7ebe464387e88f60cabd2ad4 https://sketchfab.com/3d-models/street-art-on-rue-chapleau-m... https://sketchfab.com/3d-models/day-124-colored-sidewalk-in-sf-1scanaday-fbda0dcb14a74f489d80c8765907129f https://sketchfab.com/3d-models/day-124-colored-sidewalk-in-...
- jakecopp 4y ago
  I haven’t yet, thanks for the tip. I’ll have a look into it!
deleted 4y ago
[deleted]
graderjs 4y ago
This is very cool. Would it theoretically be possible to "open source" a Google Streetview type of thing by getting "mappers" to contribute their walks around the city, with some app taking photos / lidar? In a similar way that distributed projects like SETI split the work among lots of machines (that maybe not the best analogy~~ha!)?
- jakecopp 4y ago
  Similar alternatives exist - see Mapillary [1] (owned by Meta now, but they support OpenStreetMap and allow usage for that [2]) or KartaView [3] (owned by Grab). I'm not aware of an OSM/open place to store/contribute pointclouds though - please let me know if you come across one! [1]: https://mapillary.com https://mapillary.com [2]: https://www.mapillary.com/osm https://www.mapillary.com/osm [3]: https://kartaview.org/ https://kartaview.org/
  - graderjs 4y ago
    Oh I was just asking a throwaway improvising question, I'm not going to come across one or look for it! But cool project. Anyone everyone else tho.. haha! :) ;p
subroutine 4y ago
Does it do well with large elevation changes over long distances, or is the lidar mapping all relative to the current location of the phone?
dgreensp 4y ago
I recently 3D scanned an area of my property (about 250 feet long) with my iPhone in order to make a map. Polycam was the winning app (Scaniverse and SiteScape did not work as well). Do not underestimate the sheer amount of walking around and aiming your phone that this technique requires. A 5m range is not much, and in some cases the effective range seems to be less than that. If you just want a to-scale image of an area, you can stop after "Capturing the model" and take a screenshot of your captured scan, but it's still a lot of work.
crtified 4y ago
Super cool concept. Nothing against the iPhone, but it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices. It's worth noting (again, without trying to be a grinch - merely trying to add to the overall picture) that there are actual dedicated lidar scanners on the retail market, of higher spec and capability, within a similar price range as the Apple devices involved here. Of course, you don't get an awesome iPhone with those! :)
- jakecopp 4y ago
  > it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices. I agree! I've heard of "cheap" lidar scanners (Intel Lidar Camera L515 for $375 apparently [1], or [2]) but the act of turning real time depth measurements from a UART output into a 3D model (or even just a coloured pointcloud) is no simple task! The Intel one does appear to have been discontinued: https://www.therobotreport.com/intel-issues-end-of-life-notice-realsense-lidar/ https://www.therobotreport.com/intel-issues-end-of-life-noti... Something that may scratch the itch though - I've thought it would be cool to build a bicycle mounted, microcontroller-controlled servo mount for an iPhone which moves it back and forth to increase the the width of scanned area (I assume a sine wave). [1]: https://old.reddit.com/r/robotics/comments/liudco/where_to_get_cheap_but_effective_lidar_sensor/gn5andm/ https://old.reddit.com/r/robotics/comments/liudco/where_to_g... [2]: https://www.leishenlidar.com/product/m10-low-cost-tof-lidar-scanner-outdoor/ https://www.leishenlidar.com/product/m10-low-cost-tof-lidar-...
hebrox 4y ago
Looking at https://map.openaerialmap.org/#/151.20735257863998,-33.86550701678302,18/latest/640db28cc2655600066c2a44?_k=sefsq2 https://map.openaerialmap.org/#/151.20735257863998,-33.86550... I'm only seeing a 2D option. Is there maybe also a 2.5D or 3D viewer?
rsp1984 4y ago
This is getting very close to actual field survey work and IMO the best app, by a mile, to carry out the scanning part in this kind of setting is Dot3D. It's basically been built for larger-scale capture and reliable loop-closure. Here's a scan of several km (!) of length all done with Dot3D on an iPad [1]. You can also explore the point cloud on the web. Also, digging a little deeper, here's a study comparing the accuracy and loop-closure capabilities of several 3D scanning apps. It's spot on. [2] [1] https://twitter.com/netch87/status/1632392369143435264 https://twitter.com/netch87/status/1632392369143435264 [2] https://rafaels-blog.weebly.com/blog/why-loop-closure-is-crucial-to-3d-scanning-accuracy-an-iphone-study https://rafaels-blog.weebly.com/blog/why-loop-closure-is-cru...
- danielvf 4y ago
  Link 2 is clearly content marketing. It's a blog with only one post. Though it claims to be a comparison of tools, it only mentions the name of Dot3D, refers to all other tools by number, so you can't look at them, and is written by someone on the Dot3D team.
  - FinnKuhn 4y ago
    Also two of the three last posts of the account you replied to are just tweets from the Dot3D founder...
- traspler 4y ago
  300$/y is a bit much for hobbyist street scanning though... The linked content marketing blog post shows Scaniverse to be nearly as good, so that's nice to know as an alternative.

267 points by jakecopp 4y ago

I wrote this guide on creating orthorectified imagery using just an iPhone, using a demo on George St, Sydney! Usually this is easier with drones, but there are a lot of places you can't fly drones. I hope this guide comes in helpful, especially if you're contributing to OpenStreetMap. I've found you can strap an iPhone to the front of a bicycle and if you go slowly the 3D models/LiDAR point clouds are quite good (it has a 5m range). There are drift issues if you try to complete loops though.

ramraj07 4y ago
Sites down btw
- jakecopp 4y ago
  Thanks, fixing now Archive at https://web.archive.org/web/20230314002734/https://jakecoppinger.com/2023/03/generating-aerial-imagery-with-your-iphones-lidar-sensor/ https://web.archive.org/web/20230314002734/https://jakecoppi...
  - jakecopp 4y ago
    Fixed.
Chilko 4y ago
Hi, this sounds really interesting but I am unable to read the guide as I am getting a 524 Timeout error.

lucb1e 4y ago

Note this is not aerial imagery shot from an aircraft with an Apple phone, this is someone with an Apple phone on their head cycling around the neighborhood and then viewing the 3d model from above. No less cool, but I was rather confused for a lot of the article, wondering what in the world the range on this sensor is (usually you get a few meters on normal-priced sensors, not hundreds of meters, and the picture showed entire streets being captured). The article actually mentions the maximum range is 5 metres after you set it to low confidence.

jakecopp 4y ago
Thanks for this - I'll update it to try and be more clear! This capture was just holding an iPhone in my hand - but you can place it on the front of a bicycle too. The Apple ARKit part handles the SLAM [1] of fusing the camera/LiDAR/gyro and possibly GPS. [1]: https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping https://en.wikipedia.org/wiki/Simultaneous_localization_and_...
- lucb1e 4y ago
  > I'll update it to try and be more clear! Not an issue of yours, it's all a matter of context -- and how closely one reads the article while trying to get an idea of what it's about. I just happen to be in the OSM ecosystem deep enough to be very familiar with the word aerial (as opposed to other types of area photographies), but not deep enough into the Apple sphere to know the capabilities of these devices and realize what the title clearly must mean. It might just be a me-issue :)
- trafficante 4y ago
  This is really cool, thanks for sharing. Do you know the functional limit of the iPhone’s LiDAR? Ie: could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever?
  - jakecopp 4y ago
    > could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever? Yep that should be possible - as long as you flow below 5 metres though! If you have a drone and are somewhere that is legal & safe to fly, photogrammetry (3d model from just visual photos) will likely give a better result than relying on the iPhone LiDAR stitching. Set the drone image self timer to 2 seconds, fly around for a bit and then upload all the photos into WebODM: https://opendronemap.org/webodm/ https://opendronemap.org/webodm/ You can run it on your local machine (see instructions in blog post) or hosted for a fee. It creates brilliant models and is a lot of fun!
    - 4y ago

willio58 4y ago

Ever since iPhone introduced Lidar, I've wondered why Teslas got rid of it. They said it was due to additional cost and complexity in models. Of course Waymo heavily uses Lidar but they have huge sensors they mount on cars to do so. Why is that lidar so different? What's stopping Tesla from adding the same sensors the iphone has, but all over the car and stitching it together like it does video from cameras? Is it just to do with distance? Refresh rate? Would love someone who knows this to teach me!

SomeHacker44 4y ago
AFAIK Tesla never had lidar. They had and will again have simple radar.
Melatonic 4y ago
Lidar I believe can be heavily affected by weather and might also require a lot of expensive sensors. It works great for VFX work when you need a precise model of an area. But I would guess the refresh rate is also not amazing. I live near Waymo and see the cars all the time - they do have huge things on top but the weather in Santa Monica is also very good. Not sure if I have seen them ever driving in the rain?
- zxexz 4y ago
  Interesting. I’ve used a Velodyne VLP-32 in reasonably heavy rain with reasonable performance after some error correction. I was on foot and not driving, and probably only for a few hours total. But it seems to me that a big company with serious engineering talent should be able to make it work.
  - throwbadubadu 4y ago
    Yes.. any other car manufacturer I know of (likely not complete) is considering also Lidar sensors... they are still pricey and huge, but there are also MEMS-like improvements (likely incorrect terminology) and also costs get continously lower. At the same time advancements also happen at the multi-model side, not fusing different deep learning outputs afterwards but right in the one model (not judging if that is a good idea... but the steps taking currently seem to get bigger and better). Lastly, also camera has degraded performance under certain weather conditions.. some even say our eyes can be prone to that! :D So that's just all the challenges that need to be taken. Yes, humans can drive with their eyes only, but is that perfect? Sometimes we e.g. would also benefit greatly from radar/lidar, as e.g. in dark conditions our judgement of distances is really flawed. Refresh rate of some of this already aged car lidars was like 10 Hz, good enough I'd say? And it is truly amazing what details one can get out of those when getting close and move around in those point clouds.

bilsbie 4y ago

I’m questioning my reading comprehension but what’s it telling me to do?

pimlottc 4y ago

Feedback: on mobile, I thought the call-to-subscribe block was the end of the page. I didn’t even see the table of contents the first time.

jakecopp 4y ago
Thanks for that - I've heard that feedback once before. I've changed the h2 to an h3. Do you have any tips for how I could make it feel less like the end of the page?
kejaed 4y ago
Same here. I’d put the table of contents at the very top of the page, so that I know to expect the whole thing. I’d make the div for the subscribe block a different colour to make it stand out from the rest of the page. An href link with “continue article…” that jumped down to the next section could help too. FWIW I almost left the page and missed a really good read because of this issue as well. Glad I didn’t!
- jakecopp 4y ago
  Thanks! I've moved the table of contents to be just below the images (so the image and intro is above the fold), and "read below" text to the signup. I'll improve the background/design of the subscribe block when I have some free time.

jonas21 4y ago

Have you tried Scaniverse for the capture step? It does a decent job at correcting for drift and avoiding misalignments for these sorts of scans. For example: https://sketchfab.com/3d-models/babakawa-urban-design-project-performance4-e5481c77ed4e4bceba1d48afad79b99a https://sketchfab.com/3d-models/babakawa-urban-design-projec... https://sketchfab.com/3d-models/holiday-villa-iphone-3d-scan-bcd1727b23ba468080d63d6c3f59e00b https://sketchfab.com/3d-models/holiday-villa-iphone-3d-scan... https://sketchfab.com/3d-models/street-art-on-rue-chapleau-montreal-2266ed5e7ebe464387e88f60cabd2ad4 https://sketchfab.com/3d-models/street-art-on-rue-chapleau-m... https://sketchfab.com/3d-models/day-124-colored-sidewalk-in-sf-1scanaday-fbda0dcb14a74f489d80c8765907129f https://sketchfab.com/3d-models/day-124-colored-sidewalk-in-...

jakecopp 4y ago
I haven’t yet, thanks for the tip. I’ll have a look into it!

deleted 4y ago

[deleted]

graderjs 4y ago

This is very cool. Would it theoretically be possible to "open source" a Google Streetview type of thing by getting "mappers" to contribute their walks around the city, with some app taking photos / lidar? In a similar way that distributed projects like SETI split the work among lots of machines (that maybe not the best analogy~~ha!)?

jakecopp 4y ago
Similar alternatives exist - see Mapillary [1] (owned by Meta now, but they support OpenStreetMap and allow usage for that [2]) or KartaView [3] (owned by Grab). I'm not aware of an OSM/open place to store/contribute pointclouds though - please let me know if you come across one! [1]: https://mapillary.com https://mapillary.com [2]: https://www.mapillary.com/osm https://www.mapillary.com/osm [3]: https://kartaview.org/ https://kartaview.org/
- graderjs 4y ago
  Oh I was just asking a throwaway improvising question, I'm not going to come across one or look for it! But cool project. Anyone everyone else tho.. haha! :) ;p

subroutine 4y ago

Does it do well with large elevation changes over long distances, or is the lidar mapping all relative to the current location of the phone?

dgreensp 4y ago

I recently 3D scanned an area of my property (about 250 feet long) with my iPhone in order to make a map. Polycam was the winning app (Scaniverse and SiteScape did not work as well). Do not underestimate the sheer amount of walking around and aiming your phone that this technique requires. A 5m range is not much, and in some cases the effective range seems to be less than that. If you just want a to-scale image of an area, you can stop after "Capturing the model" and take a screenshot of your captured scan, but it's still a lot of work.

crtified 4y ago

Super cool concept. Nothing against the iPhone, but it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices. It's worth noting (again, without trying to be a grinch - merely trying to add to the overall picture) that there are actual dedicated lidar scanners on the retail market, of higher spec and capability, within a similar price range as the Apple devices involved here. Of course, you don't get an awesome iPhone with those! :)

jakecopp 4y ago
> it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices. I agree! I've heard of "cheap" lidar scanners (Intel Lidar Camera L515 for $375 apparently [1], or [2]) but the act of turning real time depth measurements from a UART output into a 3D model (or even just a coloured pointcloud) is no simple task! The Intel one does appear to have been discontinued: https://www.therobotreport.com/intel-issues-end-of-life-notice-realsense-lidar/ https://www.therobotreport.com/intel-issues-end-of-life-noti... Something that may scratch the itch though - I've thought it would be cool to build a bicycle mounted, microcontroller-controlled servo mount for an iPhone which moves it back and forth to increase the the width of scanned area (I assume a sine wave). [1]: https://old.reddit.com/r/robotics/comments/liudco/where_to_get_cheap_but_effective_lidar_sensor/gn5andm/ https://old.reddit.com/r/robotics/comments/liudco/where_to_g... [2]: https://www.leishenlidar.com/product/m10-low-cost-tof-lidar-scanner-outdoor/ https://www.leishenlidar.com/product/m10-low-cost-tof-lidar-...

hebrox 4y ago

Looking at https://map.openaerialmap.org/#/151.20735257863998,-33.86550701678302,18/latest/640db28cc2655600066c2a44?_k=sefsq2 https://map.openaerialmap.org/#/151.20735257863998,-33.86550... I'm only seeing a 2D option. Is there maybe also a 2.5D or 3D viewer?

rsp1984 4y ago

This is getting very close to actual field survey work and IMO the best app, by a mile, to carry out the scanning part in this kind of setting is Dot3D. It's basically been built for larger-scale capture and reliable loop-closure. Here's a scan of several km (!) of length all done with Dot3D on an iPad [1]. You can also explore the point cloud on the web. Also, digging a little deeper, here's a study comparing the accuracy and loop-closure capabilities of several 3D scanning apps. It's spot on. [2] [1] https://twitter.com/netch87/status/1632392369143435264 https://twitter.com/netch87/status/1632392369143435264 [2] https://rafaels-blog.weebly.com/blog/why-loop-closure-is-crucial-to-3d-scanning-accuracy-an-iphone-study https://rafaels-blog.weebly.com/blog/why-loop-closure-is-cru...

danielvf 4y ago
Link 2 is clearly content marketing. It's a blog with only one post. Though it claims to be a comparison of tools, it only mentions the name of Dot3D, refers to all other tools by number, so you can't look at them, and is written by someone on the Dot3D team.
- FinnKuhn 4y ago
  Also two of the three last posts of the account you replied to are just tweets from the Dot3D founder...
traspler 4y ago
300$/y is a bit much for hobbyist street scanning though... The linked content marketing blog post shows Scaniverse to be nearly as good, so that's nice to know as an alternative.