11 ms·
Apple enters the augmented reality fray with ARKit for iOS
- tuyguntn 9y agoHow does it compare to Google Tango? Tango needs special hardware, but ARKit is not. Even if it required, there would be a definitely device (iPhone or iPad). Traction wise Apple could win, but what about performance and accuracy wise?
- asadlionpk 9y agoFor now it's more like how Snapchat's latest world filters work (try them out). Basic environment tracking, good enough for a smooth experience, considering you keep your camera steady. Tango takes the tracking to the next level. Maybe future iPhones will have that kind of IMU and other sensors, allowing Apple to expand their AR API.
- Eridrus 9y agoI was pretty impressed with their live demo. I tried Pokemon Go briefly, and their "AR" was a complete joke compared to this.
- paul9290 9y agoI thought we might be seeing an AR app built into the Camera or Map app. Nope just a kit for developers to build games only or does the ARkit allow access to the camera app? I'm dying to see innovation from Apple like in the camera app when you open it there's an option to view how things look in the past(100 years ago what did this spot in New York City look like.. where you can take a pic of yourself in the past). I'm sure there's oodles that can be done with an AR Camera. Apple and the others to me seem to be slow to innovate.
- k-mcgrady 9y agoI might be wrong but it sounds like 3rd party devs can build those things using ARKit.
- aylmao 9y agoYeah, and that's the whole point of it. There's too many applications of AR for Apple to build alone.
- paul9290 9y agoHmmm but their actual camera app? What will happen is a developer will build the app I mentioned above (could exist already) and then Apple will later build it into the camera app. I've never seen an app maker build functionality into any of the iPhone built in apps.
- Someone 9y agoLook harder. iOS supports custom keyboards and app extensions that extend iMessage (https://developer.apple.com/documentation/messages https://developer.apple.com/documentation/messages)
- gbanfalvi 9y agoI don't think Apple offers support to extend their camera app (there is support for extending the photos app, however). What you _can_ do, (and it's quite easy), is to create your own app, capturing video from the camera. Then you can overlay any AR content you wish onto it.
- paul9290 9y agoYeah that's what I was saying.
- Someone 9y agoEven ignoring that you'll need multiple photos to do the 3D reconstruction you need for augmented reality, that 100 year old photos may be lacking in resolution, and giving a few decades of leeway either way for that "100 years ago" I doubt even Google has enough data to have a reasonable chance of having a photo for random locations you might point your camera at.
- paul9290 9y agoDoes not have to be exact and a real photo from that time in that spot or a tad far away and or an arial shot can be used to create renderings.
- mendeza 9y agoI think this is a good thing they are not baking AR into the camera like Facebook is doing with the Camera Effects Platform. This allows developers to invent future "camera" apps that could be better to use than the regular camera.
- huangc10 9y agoHow does AR work without integration with the camera? They showed a demo of dev AR camera view where the camera identified "surface".
- eridius 9y agoAR requires camera access, but I'm rather confused as to why you think it would be integrated with the built-in Camera app. How would that work?
- paul9290 9y agoOpen the camera app and at the bottom click AR... an overlay appears with options like.. - Time Machine(idea noted above) - Daylight View (if currently nighttime) - and other AR apps devs or Apple creates.
- eridius 9y agoApple's Extensions operate over IPC. Trying to add overlays onto a live camera over IPC would probably introduce unacceptable lag. Especially if you also need to handle touch events too. And of course you wouldn't be able to actually do much besides have an overlay, e.g. transitioning to other screens, showing other UI controls etc. Plus this would significantly complicate the already-complicated Camera app interface. And what would be the point? You can just launch a 3rd party AR app and it will use the camera APIs to show you the live camera feed, exactly like the built-in Camera app does.
- htormey 9y agoARKit was one of the most interesting features announced this WWDC. Can't wait to play around with some demo code and see what it's like.
- synaesthesisx 9y agoThey're paving the way for their revolutionary consumer AR eyewear (which may be delayed now apparently). Although maybe we'll see tech from some of their recent AR acquisitions (PrimeSense/Metaio) integrated into the upcoming iPhone, possibly in the form of an advanced depth/surface mapping sensor of sorts. What's fascinating is Apple has a large team focusing on HCI & AR - what "pinch-to-zoom" and multitouch did for smartphones is what Apple will need to successfully pull off the "big bang" for consumer AR.
- scarlac 9y ago> which may be delayed now apparently I'm curious, what have you read/seen that makes you think this unknown product is delayed before being announced?
- threeseed 9y agoThere was an AMA on Reddit from a Foxconn employee who looks to be pretty spot on about other products: https://www.reddit.com/r/apple/comments/6ezhwm/iama_foxconn_insider_with_information_on_next_12/ https://www.reddit.com/r/apple/comments/6ezhwm/iama_foxconn_... Rough bill of materials of Project Mirrorshades - Apple Iris. Note Iris is Siri backwards. Kopin NED Acetate frame Polarisized or prescription lens with Zeiss smart optics Bone conduction modules Microphones (noise cancellation) Light sensor Accelerometer for step tracking and head movement app navigation Magnetometer for navigation Capacitive Pavel Ceramic battery Apple chipset Charging circuit BL5 Induction module Crystal, champagne and black. We only saw transparent reader lens but I understand they can procure polarised or prescription lenses Cellulose acetate Injection mold frames into an aluminum mold. Colors are added and tumbled for finish. P3 frame design Two sizes, men's and woman's Usually 4x the BOM cost; it would indicate around $600 mark. Crystal is just clear. But clear without artefacts is difficult.
- sprafa 9y agoI've heard about this being real from someone who works in a very very large corp. They are working with Apple building the first apps for the device. He said it was not much better than Hololens at the moment.
- izacus 9y agoThere's an AR fray? Google threw their weight behind VR on mobile, who else is in the AR space?
- mattbettinson 9y agoMicrosoft?
- cjCamel 9y agoHololens. Probably the most powerful AR device available atm.
- aylmao 9y agoMicrosoft, Snapchat, Google, Facebook. Facebook made a big deal in F8 out of adding cameras to all its applications. They're in the AR game big time. https://www.nytimes.com/2017/04/18/technology/mark-zuckerberg-sees-augmented-reality-ecosystem-in-facebook.html?_r=0 https://www.nytimes.com/2017/04/18/technology/mark-zuckerber...
- DonHopkins 9y agoMagic Leap is conspicuously but rightly missing from your list, because they're a fraud that's nowhere near releasing a real product, and whose bullshit is besmirching and dragging down the reputation of AR. https://www.youtube.com/watch?v=w8J5BWL8oJY https://www.youtube.com/watch?v=w8J5BWL8oJY
- wtfishackernews 9y agoGoogle's Tango is AR
- moron4hire 9y agoGoogle has at least three AR products: Glass is still in business for enterprise, Tango tablets and phones are finally out, and future versions of Daydream are supposed to be capable of inside-out tracking via an AR tracking system. Microsoft has the HoloLens, which technology they have also cannibalized for a series of headsets they are dubbing "Mixed Reality" for the depth camera pass-through they are performing. HTC is supposedly developing their own AR/inside-out system. Leap Motion, Intel, Qualcomm, Meta, and Magic Leap are still considered in the running, with varying levels of believability. And then there are a slew of different companies that are doing AR solutions with either the standard webcam (e.g. Facebook, Snap, Vuforia, Wikitude) or their own, standalone camera (e.g. Zed, Occipital).
- runesoerensen 9y agoMore information and docs here https://developer.apple.com/arkit/ https://developer.apple.com/arkit/
- stesch 9y ago.
- craigching 9y agoI thought 6S+ ran A9?
- bilal4hmed 9y agoIt does
- wsinks 9y agoThank you for this! Perfect link.
- Stanleyc23 9y agoThere was also a very brief mention of a "depth api" before they started talking about the actual ARKit. is that endpoint documented anywhere?
- huangc10 9y agohere? https://developer.apple.com/documentation/arkit https://developer.apple.com/documentation/arkit
- Stanleyc23 9y agonah I think its something for normal photos, not immersive content. > Camera: Updated video and image compression, improved Portrait Mode with support for low light, plus a new Depth API for developers. from https://venturebeat.com/2017/06/05/apple-announces-ios-11/ https://venturebeat.com/2017/06/05/apple-announces-ios-11/
- photojosh 9y agoI had a look at the documentation at [1], which says "The AVCaptureDepthDataOutput class captures and delivers depth data in a stream (similar to how the AVCaptureVideoDataOutput delivers video data)." So you can now access a stream of depth data. I'd put money on ARToolkit using this stream when it's available. For now, it would only be from the iPhone 7+. [1] https://developer.apple.com/documentation/avfoundation/avdepthdata https://developer.apple.com/documentation/avfoundation/avdep...
- Stanleyc23 9y agoah great find.
- MBCook 9y agoAR kit isn't limited to the 7+ ( they demoed it on an iPad) space so clearly that's not REQUIRED. I wonder if they're using the second camera for depth on the 7+ when it's available. Perhaps that data isn't granular enough (or fast enough) to be useful when doing AR compared to the normal camera picture plus the motion sensor data.
- roymurdock 9y agoStudied the market for AR development solutions recently. A few thoughts: - Probably using a lot of Metaio tech, a leading AR dev solutions provider Apple acquired/shuttered in May 2015 - Unfortunate naming. Sounds extremely similar to ARToolkit [1], the leading open source/free AR dev solution. - Markerless is the future of AR, so good on Apple for getting this markerless tech out to compete with Google Tango and Microsoft Hololens. The status quo is currently marker-based recognition using "tags" or QR codes. It's also much harder than marker-based, or "on body" (Snapchat, Facebook AR Studio) recognition. I'd say Google Tango has some of the most impressive SDKs out there for tracking, area learning, and depth perception, but it does require specialized smartphone hardware from Lenovo or Asus, which significantly limits its utility and mkt penetration at this point. Google was just too early with Glass and got burned, but it needs a stronger hardware platform, which is something that Apple can deliver. - The Unreal-engine AR video demo was cool and the graphics seemed decent for real-time rendering on an iPad, but the real future of AR applications (read: $$$) will come from industrial applications on wearable devices or head mounted displays such as the Hololens, Vuzix, Daqri, or Epson Moverio devices. Examples of industrial/enterprise AR applications: Remote help, complex assembly, pick & pack, line monitoring, materials handling, systems training, etc. This is the market Microsoft and PTC (Vuforia, ThingWorx) are targeting, not the consumer gaming/advertising markets, which use more "basic" forms of AR. This is not to say Pokemon Go/Snapchat do not generate a lot of revenue, but it's very debatable if they can be considered true AR applications. - Other vendors in the AR development solutions space include: Aurasma (HPE), Blippar, Catchoom, EON Reality, Kudan, Pikkart, Wikitude, among others. - In conclusion, Apple is currently stuck between a rock and a hard place. It's somewhat late to market with ARKit, will have to compete with Facebook, Google, and Snapchat for consumer AR-oriented developer mindshare, and will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family. - The best way for Apple to alleviate this jam is to release its own AR glasses/headset, which it is widely rumored to be developing. [1] https://artoolkit.org/ https://artoolkit.org/
- mendeza 9y agoFor AR, to win, you do not have to be the first one to market to win, its who can make the best platform for developers will win. As an AR developer, previous platforms like Vuforia, ARToolkit, and even Tango are either limiting or very hard to work with.
- huangc10 9y agoAs a developer, this was the most exciting thing for me from today's WWDC opening speech. Can't wait to try this out myself. The demo they did with the AR game was really neat.
- satysin 9y agoIt seemed quite mature for a v1 release. Then again it is hard to judge from an Apple demo but it did look nice.
- huangc10 9y agoI think there must have always been some way to do AR (Pokemon GO and other games) but I believe it always required a team to accomplish this and possibly work with engineers at Apple. As an independent developer, I hope the AR kit will make it much easier for me to create something fun and possibly integrate with Siri and other kits. This is what I'm hoping for :D I know VR is hot right now, but I've always had the thought AR > VR.
- satysin 9y agoYeah basically until now if you wanted to do AR you would need to role your own or use a third party library. Having ARKit as part of iOS will make building an AR app much easier.
- nerfhammer 9y agoWasn't Pokemon GO more like pseudo-AR? Like it just superimposed an animation with an alpha channel shadow but which otherwise had nothing to do with whatever the camera was looking at?
- MBCook 9y agoYes. It's AR in that they overlaid something over the real world. That's why the little bit during the demo was interesting. They showed Pokémon go using AR but it was actually doing it correctly, where the Pokémon stayed in position as they moved the camera around. I was wondering what Pokémon Go would do on android. Will this feature be iOS only or will they implement their own version on android?
- agumonkey 9y agoBarely related but what happened to HealthKit ? are ambitious Apple Kits failing ?
- dpkonofa 9y agoHealthKit is still going. They announced some big wins with HealthKit on the Apple Watch but they weren't included in the keynote. More than likely, there wasn't enough time considering the keynote without any HealthKit announcements was already longer than normal.
- agumonkey 9y agoThanks. Last month some medtech guy reported about different efforts and said HealthKit vanished. The fact Apple didn't mention made me think he was right. Now I'll wait for more.
- aSig 9y agoIs anyone able to find the ARKit sample code that was mentioned during the keynote?
- enos_feedler 9y agoDoes anyone have resources for "State of the art" algorithms and research that is used to do the sensor fusion (camera frames + motion sensor data)? I found this but not sure its the most up to date. I'm also looking for more detail in terms of code: http://user.it.uu.se/~thosc112/pubpdf/holsgs2006.pdf http://user.it.uu.se/~thosc112/pubpdf/holsgs2006.pdf
- steinomri 9y agoIt seems that ARkit is actually using SLAM as the core technology (in parts, the same technology is being used in autonomous cars and navigating drones). Getting it to work on mobile phones is quite challenging and had been the subject of much research in recent years, some good starting points (with code): ORB-SLAM - https://github.com/raulmur/ORB_SLAM2 https://github.com/raulmur/ORB_SLAM2, https://www.youtube.com/watch?v=T-9PYCKhDLM https://www.youtube.com/watch?v=T-9PYCKhDLM DSO - https://github.com/JakobEngel/dso https://github.com/JakobEngel/dso, https://www.youtube.com/watch?v=C6-xwSOOdqQ https://www.youtube.com/watch?v=C6-xwSOOdqQ VINS - https://github.com/HKUST-Aerial-Robotics/VINS-Mobile https://github.com/HKUST-Aerial-Robotics/VINS-Mobile, https://www.youtube.com/watch?v=0mTXnIfFisI&feature=youtu.be https://www.youtube.com/watch?v=0mTXnIfFisI&feature=youtu.be (the last one being released just a few days ago...) Of course there are a lot more resources out there and a lot of the cutting edge research is involved with deep learning techniques (starting with e.g. https://github.com/alexgkendall/caffe-posenet https://github.com/alexgkendall/caffe-posenet). You may also want to watch out for this conference http://cvpr2017.thecvf.com/ http://cvpr2017.thecvf.com/ scheduled for the end of this month and is usually a good source of cutting edge research...
- quantumwannabe 9y agoI find it funny that Apple is just now releasing an AR SDK as when the iPhone first came out there were a lot of AR apps available. Most of them were pretty boring, just games like shooting spaceships or POI explorers, but they didn't look that much more primitive than the apps demoed today. I wonder why they didn't jump on AR back then.
- MBCook 9y agoI remember playing with those on my 3G or my 4. It was pretty impressive what people were able to do with those little phones. I'm sure part of it is just the ability to do useful things, part of it may simply be that they have such had room on the chips that they can do it without draining battery life horribly. I remember those old apps clearly used a lot of processing power.