6 ms·

AR with SceneKit and Metal

148 points by emllnd 6y ago

aabajian 6y ago
I think AR kit is the most under-utilized API available on iOS. Combined with the U1 chip, you could make a real-world tagging-app. Post a note on your favorite trail or maybe overlay In n Out's secret menu on their real menu. How about AR Zoom backgrounds?
- sipjca 6y ago
  I think you're spot on. The documentation for AR Kit is top notch, and includes TONS of sample code that is easily understandable with minimal iOS development knowledge. Perhaps I'm still too new to AR Kit, but to me it doesn't look entirely straightforward (more backend lifting) to do someone of the things you are mentioning. I hope Apple releases some updated Geolocation API's to better enable the experiences you mention. Kai Faust's [1] work with Infopop [2] is definitely worth playing around with [1]: (https://twitter.com/kaifaust https://twitter.com/kaifaust) [2]: (https://apps.apple.com/us/app/infopop/id1530246692 https://apps.apple.com/us/app/infopop/id1530246692)
  - emeraldpanda 6y ago
    The building blocks for location based AR are pretty much all there: - Azure Spatial Anchors (iOS, Android, HoloLens, ROS) [1] - Google Cloud Anchors (iOS, Android) [2] - Apple Geo Anchors (iOS & not all locations so not ready for use yet) [3] Integration with these SDKs is pretty easy, and Azure Spatial Anchors and Google Cloud Anchors are production-ready. Minecraft Earth for example used Azure Spatial Anchors for shared multiplayer AR experiences. Infopop is nice, but doesn't have multiplayer or long-term persistence, so not really useful as-is. But wouldn't be surprised if it eventually uses one of these SDKs to provide better multiplayer + shared relocalization. You could probably throw together a rough version of Infopop in a week if sufficiently motivated, ARKit does a lot of the heavy lifting there. [1]: (https://docs.microsoft.com/en-us/azure/spatial-anchors/overview https://docs.microsoft.com/en-us/azure/spatial-anchors/overv...) [2]: (https://developers.google.com/ar/develop/java/cloud-anchors/overview-android https://developers.google.com/ar/develop/java/cloud-anchors/...) [3]: (https://developer.apple.com/documentation/arkit/argeoanchor https://developer.apple.com/documentation/arkit/argeoanchor)
    - sipjca 6y ago
      Thanks for sharing the links to Google Cloud Anchors and Azure! Quickly looked through the docs of both, excited to try them out and see how well they work :) I used Apple's Geo Anchors, but found it pretty limiting for my use case (labelling rock climbs outdoors), since it rely's heavily on street level data it seems
      - emeraldpanda 6y ago
        Yep, they're both pretty comparable in their API surface, and more general purpose than Apple Geo Anchors, as neither are tightly coupled with GPS. Rather you create an anchor and then save it to the cloud. On your own you can tie it to a GPS location if you'd like. Though Azure Spatial Anchors does have more functionality than Google Cloud Anchors (Coarse Relocalization + find nearby anchors in the anchor graph).
- wlesieutre 6y ago
  I haven't tried it myself, but Reality Mixer is a neat application of person segmentation that I saw recently: https://twitter.com/reality_mixer https://twitter.com/reality_mixer Impressive how well that works.
  - emllnd 6y ago
    Thanks for sharing, good to know someone has built this!
- germinalphrase 6y ago
  Clearly, we need a ‘Wikiworld’ AR app for adding shared metadata to the world.
- nradov 6y ago
  William Gibson predicted those use cases in his 1994 novel Virtual Light. He had artists creating works of art in physical locations that were only visible to people wearing AR goggles. https://www.penguinrandomhouse.com/books/59659/virtual-light-by-william-gibson/ https://www.penguinrandomhouse.com/books/59659/virtual-light...
edge17 6y ago
nice post, worth knowing though that I have heard through the grapevine that SceneKit is to be deprecated in favor of RealityKit. SceneKit also has abhorrent documentation that is just plain wrong in places.
- emllnd 6y ago
  Seems logical that it would happen at some point. I don’t think RealityKit covers enough of the semi-common use cases in its current state that they could deprecate SceneKit just yet. But it’s on the horizon definitely.
masonhensley 6y ago
Very cool, I've been starting to use ARKit from a different angle. Usecase - test out 3d designs of replacement classic car parts I'm manufacturing as a side project. While crude right now, I think this sort of stuff is a game changer - I'm regretting not picking up a lidar enabled 12 pro instead of a 12 now to do better scanning and placement. I guess I'll pick up an ipad pro when it's upgrade time on that front. Here are two quick videos I threw together: https://youtu.be/M8dgrf7GBow https://youtu.be/M8dgrf7GBow https://www.youtube.com/watch?v=7K0-vK2wafA https://www.youtube.com/watch?v=7K0-vK2wafA Workflow: - build 2d models in solvespace (https://solvespace.com https://solvespace.com) - export dxf to fusion 360 - convert dxf to sheet metal - bend sheet metal in fustion360 - export obj from fusion360 - save obj file to icloud drive - open obj file in AR Viewer app (https://apps.apple.com/us/app/ar-viewer-augmented-reality/id1253207471 https://apps.apple.com/us/app/ar-viewer-augmented-reality/id...) Sidenote: There still isn't a replacement for physically printing the parts before I have them laser cut out of metal. I 3d print models and test fit pieces as intermediate steps too. Really excited to see ios developers digging into ARKit more, I'm excited to see what else is possible here.
- wlesieutre 6y ago
  If you got the iPhone after November 10th they have an extended holiday return period https://www.apple.com/shop/help/returns_refund https://www.apple.com/shop/help/returns_refund
  - masonhensley 6y ago
    Yep, I'm in that window, but availability of the 12 pro is pretty bad and it's not that painful in the grand scheme of life to warrant hunting one down haha. Regret, but not that big of a regret :)
    - wlesieutre 6y ago
      Well you've got another month to think about it. Personally LIDAR on the phone sounds a lot more useful to me than the iPad, and I say that having a 2020 iPad Pro. Knew I was going to upgrade my SE to a 12 Mini with no LIDAR (based on rumors) so I sprung for the new iPad instead of looking for 2018s on sale. Nice to have one device with LIDAR to play around with, but it's super awkward to wave an iPad around to scan things. More than anything else, I think it was pushed out as a developer kit to get the hardware in people's hands and apps ready to go before it hit the mass market with the iPhone 12 Pro.
doomlaser 6y ago
Unity has become a strong platform choice for AR development as it wraps both ARKit and ARCore and is cross-platform with iPhones and Android phones, as well as webgl. Pokemon Go, probably the AR app with the most market penetration, is Unity.
- emllnd 6y ago
  Agree, in most cases Unity is probably a better tool for AR application work than using the platform APIs directly.
- deleted 6y ago
  [deleted]
keyle 6y ago
So I'm an indie game dev and I have experimented quite a bit with the previous version of AR from Apple and the biggest issue was of game design. Making a game where people have to sit in front of a table, or stand, or hold the device up, is very restrictive and instantly makes you niche. You get more of a potential slice of the market making a cool 2D puzzle. The tech was cool, worked reasonably well until -A- you moved a lot -B- low light. But unfortunately the market is about playing on the toilet, in public transport; or playing late at night in bed, in the dark. None of which really worked with AR. As far as I see this going, it's still is looking for a "killer app" (I hate that term). My best guess is we'll see this used a lot more in a business environment and even then, glasses are a far better approach (hint hint, hold on to your whiskers).
- k__ 6y ago
  Doesn't LIDAR solve these technical problems?
  - doomlaser 6y ago
    It will once it rolls out onto more phones. Right now it is only available on iPhone 12 Pro models
    - AceJohnny2 6y ago
      Where the use-case currently is more photogrammetry https://en.wikipedia.org/wiki/Photogrammetry https://en.wikipedia.org/wiki/Photogrammetry
- definitelyhuman 6y ago
  Totally agree, AR in games is more of a toy. The progress with lidar is... progress - but still not great even in basic scenes, helps with low light. Visual issues break the immersion and pull the user from the experience, which is supposed to be the point. AR as a tech is interesting, and has practical business uses as you point out - but has a long way to go to be useful for games, let alone an engaging centerpiece. Like everything else, it’ll improve steadily, and I’m excited for where it could go
HugoDaniel 6y ago
Is SceneKit still alive? I had the idea that it was abandoned by Apple.
lankinen 6y ago
Great article and great conversation here. I wonder if you have some good communities where you spend time. I wish I could read much more of these. I created a forum but so far I have been the only one posting https://augmented-reality-forum.tribe.so/post/hello-world-5fb802a39c71e71022d9cd0d https://augmented-reality-forum.tribe.so/post/hello-world-5f...
- emllnd 6y ago
  Hi lankinen, maybe try some big AR subreddits like /r/augmentedreality, /r/AR_MR_XR or /r/virtuality. Could also google for some slack or discord channels. I don’t necessarily frequent specific communities too much, more just try to find stuff via googling about various interests and projects, and talking to friends and colleagues.
  - emeraldpanda 6y ago
    The HoloDevelopers Slack is very active. HoloLens focused, but also channels for other AR topics: https://holodevelopersslack.azurewebsites.net/ https://holodevelopersslack.azurewebsites.net/
Razengan 6y ago
I highly recommend watching the anime Dennou Coil [0] to see what the world could be like if AR glasses were as common as smartphones today: • Everybody is connected to the same "AR space" and sees the same AR objects. • Different people may have different view/interact access to different objects. • Some form of tactile feedback is assumed. In Apple's case we will assume this to be provided via the Apple Watch or a hypothetical AR gloves accessory. • Many things that are currently physical could be replaced with virtual versions: screens, keyboards, signage, fashion, pets. Many developers have already noticed the gradual introduction of APIs and hardware like LIDAR that are so obviously meant to support a future glasses product, but I wonder if anyone's connected the dots between features like the spatial audio recently introduced to AirPods. Apple's AR strategy is very likely going to be divided across their different products: iPhone (processing), Glasses (display), Watch (motion control and tactile feedback), and of course AirPods. Many of their users already own most of those devices. No other company is so perfectly positioned to finally deliver ubiquitous AR. [0] https://myanimelist.net/anime/2164/Dennou_Coil https://myanimelist.net/anime/2164/Dennou_Coil
- saagarjha 6y ago
  The interesting bit about Dennou Coil is the one corporation that essentially controls a significant portion of the protagonists' lives ;)
- autoexec 6y ago
  Also endless data collection of literally everything in every persons FOV, ads aggressively plastered over everything in your life at all times, subtle censorship of anything corporations or government would rather you not see, etc. As with every technology we have right now it will be used against you. It doesn't have to be that way, but that's the world we live in now.
hyperdimension 6y ago
I'll be waiting for an AR app to overlay billboards with 'OBEY', 'CONSUME', etc. personally.

148 points by emllnd 6y ago

I think AR kit is the most under-utilized API available on iOS. Combined with the U1 chip, you could make a real-world tagging-app. Post a note on your favorite trail or maybe overlay In n Out's secret menu on their real menu. How about AR Zoom backgrounds?

sipjca 6y ago
I think you're spot on. The documentation for AR Kit is top notch, and includes TONS of sample code that is easily understandable with minimal iOS development knowledge. Perhaps I'm still too new to AR Kit, but to me it doesn't look entirely straightforward (more backend lifting) to do someone of the things you are mentioning. I hope Apple releases some updated Geolocation API's to better enable the experiences you mention. Kai Faust's [1] work with Infopop [2] is definitely worth playing around with [1]: (https://twitter.com/kaifaust https://twitter.com/kaifaust) [2]: (https://apps.apple.com/us/app/infopop/id1530246692 https://apps.apple.com/us/app/infopop/id1530246692)
- emeraldpanda 6y ago
  The building blocks for location based AR are pretty much all there: - Azure Spatial Anchors (iOS, Android, HoloLens, ROS) [1] - Google Cloud Anchors (iOS, Android) [2] - Apple Geo Anchors (iOS & not all locations so not ready for use yet) [3] Integration with these SDKs is pretty easy, and Azure Spatial Anchors and Google Cloud Anchors are production-ready. Minecraft Earth for example used Azure Spatial Anchors for shared multiplayer AR experiences. Infopop is nice, but doesn't have multiplayer or long-term persistence, so not really useful as-is. But wouldn't be surprised if it eventually uses one of these SDKs to provide better multiplayer + shared relocalization. You could probably throw together a rough version of Infopop in a week if sufficiently motivated, ARKit does a lot of the heavy lifting there. [1]: (https://docs.microsoft.com/en-us/azure/spatial-anchors/overview https://docs.microsoft.com/en-us/azure/spatial-anchors/overv...) [2]: (https://developers.google.com/ar/develop/java/cloud-anchors/overview-android https://developers.google.com/ar/develop/java/cloud-anchors/...) [3]: (https://developer.apple.com/documentation/arkit/argeoanchor https://developer.apple.com/documentation/arkit/argeoanchor)
  - sipjca 6y ago
    Thanks for sharing the links to Google Cloud Anchors and Azure! Quickly looked through the docs of both, excited to try them out and see how well they work :) I used Apple's Geo Anchors, but found it pretty limiting for my use case (labelling rock climbs outdoors), since it rely's heavily on street level data it seems
    - emeraldpanda 6y ago
      Yep, they're both pretty comparable in their API surface, and more general purpose than Apple Geo Anchors, as neither are tightly coupled with GPS. Rather you create an anchor and then save it to the cloud. On your own you can tie it to a GPS location if you'd like. Though Azure Spatial Anchors does have more functionality than Google Cloud Anchors (Coarse Relocalization + find nearby anchors in the anchor graph).
wlesieutre 6y ago
I haven't tried it myself, but Reality Mixer is a neat application of person segmentation that I saw recently: https://twitter.com/reality_mixer https://twitter.com/reality_mixer Impressive how well that works.
- emllnd 6y ago
  Thanks for sharing, good to know someone has built this!
germinalphrase 6y ago
Clearly, we need a ‘Wikiworld’ AR app for adding shared metadata to the world.
nradov 6y ago
William Gibson predicted those use cases in his 1994 novel Virtual Light. He had artists creating works of art in physical locations that were only visible to people wearing AR goggles. https://www.penguinrandomhouse.com/books/59659/virtual-light-by-william-gibson/ https://www.penguinrandomhouse.com/books/59659/virtual-light...

edge17 6y ago

nice post, worth knowing though that I have heard through the grapevine that SceneKit is to be deprecated in favor of RealityKit. SceneKit also has abhorrent documentation that is just plain wrong in places.

emllnd 6y ago
Seems logical that it would happen at some point. I don’t think RealityKit covers enough of the semi-common use cases in its current state that they could deprecate SceneKit just yet. But it’s on the horizon definitely.

masonhensley 6y ago

Very cool, I've been starting to use ARKit from a different angle. Usecase - test out 3d designs of replacement classic car parts I'm manufacturing as a side project. While crude right now, I think this sort of stuff is a game changer - I'm regretting not picking up a lidar enabled 12 pro instead of a 12 now to do better scanning and placement. I guess I'll pick up an ipad pro when it's upgrade time on that front. Here are two quick videos I threw together: https://youtu.be/M8dgrf7GBow https://youtu.be/M8dgrf7GBow https://www.youtube.com/watch?v=7K0-vK2wafA https://www.youtube.com/watch?v=7K0-vK2wafA Workflow: - build 2d models in solvespace (https://solvespace.com https://solvespace.com) - export dxf to fusion 360 - convert dxf to sheet metal - bend sheet metal in fustion360 - export obj from fusion360 - save obj file to icloud drive - open obj file in AR Viewer app (https://apps.apple.com/us/app/ar-viewer-augmented-reality/id1253207471 https://apps.apple.com/us/app/ar-viewer-augmented-reality/id...) Sidenote: There still isn't a replacement for physically printing the parts before I have them laser cut out of metal. I 3d print models and test fit pieces as intermediate steps too. Really excited to see ios developers digging into ARKit more, I'm excited to see what else is possible here.

wlesieutre 6y ago
If you got the iPhone after November 10th they have an extended holiday return period https://www.apple.com/shop/help/returns_refund https://www.apple.com/shop/help/returns_refund
- masonhensley 6y ago
  Yep, I'm in that window, but availability of the 12 pro is pretty bad and it's not that painful in the grand scheme of life to warrant hunting one down haha. Regret, but not that big of a regret :)
  - wlesieutre 6y ago
    Well you've got another month to think about it. Personally LIDAR on the phone sounds a lot more useful to me than the iPad, and I say that having a 2020 iPad Pro. Knew I was going to upgrade my SE to a 12 Mini with no LIDAR (based on rumors) so I sprung for the new iPad instead of looking for 2018s on sale. Nice to have one device with LIDAR to play around with, but it's super awkward to wave an iPad around to scan things. More than anything else, I think it was pushed out as a developer kit to get the hardware in people's hands and apps ready to go before it hit the mass market with the iPhone 12 Pro.

doomlaser 6y ago

Unity has become a strong platform choice for AR development as it wraps both ARKit and ARCore and is cross-platform with iPhones and Android phones, as well as webgl. Pokemon Go, probably the AR app with the most market penetration, is Unity.

emllnd 6y ago
Agree, in most cases Unity is probably a better tool for AR application work than using the platform APIs directly.
deleted 6y ago
[deleted]

keyle 6y ago

So I'm an indie game dev and I have experimented quite a bit with the previous version of AR from Apple and the biggest issue was of game design. Making a game where people have to sit in front of a table, or stand, or hold the device up, is very restrictive and instantly makes you niche. You get more of a potential slice of the market making a cool 2D puzzle. The tech was cool, worked reasonably well until -A- you moved a lot -B- low light. But unfortunately the market is about playing on the toilet, in public transport; or playing late at night in bed, in the dark. None of which really worked with AR. As far as I see this going, it's still is looking for a "killer app" (I hate that term). My best guess is we'll see this used a lot more in a business environment and even then, glasses are a far better approach (hint hint, hold on to your whiskers).

k__ 6y ago
Doesn't LIDAR solve these technical problems?
- doomlaser 6y ago
  It will once it rolls out onto more phones. Right now it is only available on iPhone 12 Pro models
  - AceJohnny2 6y ago
    Where the use-case currently is more photogrammetry https://en.wikipedia.org/wiki/Photogrammetry https://en.wikipedia.org/wiki/Photogrammetry
definitelyhuman 6y ago
Totally agree, AR in games is more of a toy. The progress with lidar is... progress - but still not great even in basic scenes, helps with low light. Visual issues break the immersion and pull the user from the experience, which is supposed to be the point. AR as a tech is interesting, and has practical business uses as you point out - but has a long way to go to be useful for games, let alone an engaging centerpiece. Like everything else, it’ll improve steadily, and I’m excited for where it could go

HugoDaniel 6y ago

Is SceneKit still alive? I had the idea that it was abandoned by Apple.

lankinen 6y ago

Great article and great conversation here. I wonder if you have some good communities where you spend time. I wish I could read much more of these. I created a forum but so far I have been the only one posting https://augmented-reality-forum.tribe.so/post/hello-world-5fb802a39c71e71022d9cd0d https://augmented-reality-forum.tribe.so/post/hello-world-5f...

emllnd 6y ago
Hi lankinen, maybe try some big AR subreddits like /r/augmentedreality, /r/AR_MR_XR or /r/virtuality. Could also google for some slack or discord channels. I don’t necessarily frequent specific communities too much, more just try to find stuff via googling about various interests and projects, and talking to friends and colleagues.
- emeraldpanda 6y ago
  The HoloDevelopers Slack is very active. HoloLens focused, but also channels for other AR topics: https://holodevelopersslack.azurewebsites.net/ https://holodevelopersslack.azurewebsites.net/

Razengan 6y ago

I highly recommend watching the anime Dennou Coil [0] to see what the world could be like if AR glasses were as common as smartphones today: • Everybody is connected to the same "AR space" and sees the same AR objects. • Different people may have different view/interact access to different objects. • Some form of tactile feedback is assumed. In Apple's case we will assume this to be provided via the Apple Watch or a hypothetical AR gloves accessory. • Many things that are currently physical could be replaced with virtual versions: screens, keyboards, signage, fashion, pets. Many developers have already noticed the gradual introduction of APIs and hardware like LIDAR that are so obviously meant to support a future glasses product, but I wonder if anyone's connected the dots between features like the spatial audio recently introduced to AirPods. Apple's AR strategy is very likely going to be divided across their different products: iPhone (processing), Glasses (display), Watch (motion control and tactile feedback), and of course AirPods. Many of their users already own most of those devices. No other company is so perfectly positioned to finally deliver ubiquitous AR. [0] https://myanimelist.net/anime/2164/Dennou_Coil https://myanimelist.net/anime/2164/Dennou_Coil

saagarjha 6y ago
The interesting bit about Dennou Coil is the one corporation that essentially controls a significant portion of the protagonists' lives ;)
autoexec 6y ago
Also endless data collection of literally everything in every persons FOV, ads aggressively plastered over everything in your life at all times, subtle censorship of anything corporations or government would rather you not see, etc. As with every technology we have right now it will be used against you. It doesn't have to be that way, but that's the world we live in now.

hyperdimension 6y ago

I'll be waiting for an AR app to overlay billboards with 'OBEY', 'CONSUME', etc. personally.