9 ms·
Apple introduces spatial video capture on iPhone 15 Pro
- deleted 3y ago[deleted]
- ticklemyelmo 3y agoWith less than an inch of separation, is the sense of depth even perceptible?
- abracadaniel 3y agoEarly reviews indicate it is, as some reviewers have had access to spatial video taken from a phone, but I’m not sure if those were ideal conditions or just ad-hoc.
- Reubend 3y agoPure speculation: when combined with the LIDAR depth sensor, the two cameras probably don't need as much physical separation to accurately create a depth map. The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
- TaylorAlexander 3y agoI was wondering about the use of the lidar sensor. Notably they do not say they are using it, but maybe they just wanted to keep it simple? Idk seems weird not to use lidar but also seems weird not to mention it if they are using it.
- aYsY4dDQ2NrcNzA 3y ago3D movies have existed for decades, without adjustable viewing angles.
- mlyle 3y agoBut if you have eyes 50mm apart, and source material from cameras 15mm apart (plus other depth information), you'll need to in-paint a small amount where your eyes could see "around" something and the cameras can't.
- coldtea 3y agoOr you can make do with 15mm-apart worth of "around"? It's still more "spatial"/3D than a regular (single lens) image. Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
- mlyle 3y ago> Or you can make do with 15mm-apart worth of "around"? You need to move the close objects further apart in left/right than they are in the camera. Then you need to fill the newly empty areas with something. > Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video. This doesn't affect anything.
- aYsY4dDQ2NrcNzA 3y agoI was responding to > The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
- mlyle 3y agoYes-- so to even create a fixed view with viewports that are further apart than the real cameras are, you have ot inpaint hidden detail.
- jauntywundrkind 3y agoMy understanding is that very few consumer lidar sensors work well in daylight. It's hard to send out & detect significantly meaningful pulses of light, when there's sunlight all around. I have an Intel L515 which is pretty remarkable in that sometimes you can get some depth finding outdoors. This is just a hobby item for me, I'm not an expert, but this launched as a fairly impressively long range & capable $350 USB3 system, and it seems like the market hasn't much comparable to it. Phones certainly I'd expect to be significantly worse.
- coldtea 3y ago>My understanding is that very few consumer lidar sensors work well in daylight. It's hard to send out & detect significantly meaningful pulses of light, when there's sunlight all around. Aren't many "self driving car" sensors lidar? This would imply they can work in daylight - perhaps they don't necessarily depend on light on the sunlight spectrum? (Or perhaps you don't consuder them consumer? Though those cars are consumer products, they're not made for military or industrial use)
- jauntywundrkind 3y agoMany cars are lidar,, but they use much stronger, bigger, and higher power lasers, on very expensive and precise rotator assemblies. The L515 I mentioned was somewhat advanced at least for it's day because it used MEMS to steer its light source. That gave it leading class performance/size but it's still big and kinda hot-ish. Maybe we can keep scaling that kind of system performance to smaller sizes but even this package was pretty cutting edge & gave much better falloff than many competing systems, and was still largely an indoor sensor.
- coldtea 3y ago>The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from. It's for spatial video, not for holographic video. When you see a 3d movie in a cinema, it's not like you can look at it from widely different anges and go peek from the side or behind the actors or whatever...
- laserlight 3y agoGiven that iPhone cameras are ~2.5 mm apart, there needs to be some amount of in-painting when building the stereo image, so that it looks like it is taken with cameras that are ~6.5 mm apart.
- golergka 3y agoI would expect some ML magic in the image processing pipeline that would make it pop out.
- klausa 3y agoOne of those lenses is a Ultra-Wide though, with a _very_ different FoV than the other one.
- brookst 3y agoIt uses a crop from the center. Not sure if that crop has same FOV as the other lens though. I’d expect so?
- 0xDEF 3y agoOne of the two cameras is the ultra-wide camera so it gets some additional parallax and visual information than just the separation to the other camera.
- ryandamm 3y agoThat’s not how parallax works. The wider field of view of the ultra wide camera will show some of the scene that the other camera doesn’t see, but over overlapping parts of the scene the parallax is a strict function of the location of the two lenses’ entrance windows.
- ynniv 3y agoThey're probably fusing both lenses with the Lidar and some other tricks to reliably compute a dense surface. That would explain their suggestion not to move the camera very much, as that would cause a large portion of the mesh to be rebuilt. A blogger exported what appears to be two side by side videos, so maybe the view really is narrow or reconstruction happens at playback. There might also be Lidar data in there that he didn't notice. Apple bought C3 Technologies a decade ago, and they use this technique to fuse photos from low flying charters to produce the 3d view in Apple Maps. [ Paper: https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstract https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstra... ] [ Coverage: https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3d-mapping-company-c3-technologies-looking-to-take-ios-maps-to-the-next-level/ https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3... ] [ Similar: https://web.stanford.edu/class/ee367/Winter2021/projects/report_blankenberg.pdf https://web.stanford.edu/class/ee367/Winter2021/projects/rep... ]
- varjag 3y agoAs the two cameras have very different focal lengths you get pronounced parallax effect that can be exploited in post.
- ricardobeat 3y agoTwo focal lengths at the same physical distance to the subject have exactly the same perspective (i.e. if you crop them to the same area they will look the same). There is no extra information to be had from that.
- varjag 3y agoThe depth information that can be obtained from differences in angular position/size of objects within cameras' FOV. There's a reason a photo taken with a 28mm doesn't look the same as with a 50 a few steps back.
- FPGAhacker 3y agoA cropped 28mm is indistinguishable from a 50mm or any other larger size, relative photo site receptor size notwithstanding.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- simondw 3y agoBut one of the lenses isn't a few steps back from the other.
- ricardobeat 3y ago> a few steps back Exactly. The steps back change perspective, not the lenses. That’s what I was trying to say above. In the iPhone both lenses are at the same distance to the subject.
- krackers 3y agoAre spatial videos only viewable on a vision pro?
- TaylorAlexander 3y agoLast part of the article. It states that they are viewed and shared as normal 2D videos on iphone, but when you connect a Vision Pro to your iCloud account, it can view your spatial videos in 3D.
- jader201 3y agoMaybe I misunderstood, or maybe you did. But I thought the parent question was about watching the spacial videos — as 3D — on non-Apple VR devices (e.g. Quest).
- krackers 3y agoCorrect. Although it wouldn't surprise me if this _was_ locked into the apple ecosystem, it would be nice if it took advantage of existing open standards (whatever those may be).
- dagmx 3y agoThey’re just MV-HEVC http://hevc.info/mvhevc http://hevc.info/mvhevc with a specific profile which is published as well https://developer.apple.com/av-foundation/HEVC-Stereo-Video-Profile.pdf https://developer.apple.com/av-foundation/HEVC-Stereo-Video-... Any platform can implement playback of them.
- dylan604 3y agoI wonder if they'll have a way of converting to the old skool top/bottom for displaying on things like home TVs with 3D, or if they'll add update to AppleTV 4K for existing TVs. Next is having them as sources in an NLE. Lots of ways to do that now, but that's been for separate L/R sources or top/bottom type of files. It's been so long since I've had to deal with that, I've not kept up with what the current state is for that world. Maybe they've all been updated to handle mv-hevc too??
- ozten 3y agoIf you have a VR headset, you can try these videos out. I followed [1] on a Pico 4 headset (video is about Quest 3) and it was pretty fun. The aspect ratio was a bit wonky. [1] https://www.youtube.com/watch?v=SH96y2QjCGw https://www.youtube.com/watch?v=SH96y2QjCGw
- sod 3y agoThanks for the link. I tried it in my quest 3. The picture quality is awesome. I wonder why those "VR Experiences" are filmed for 180° which is super pixellated. High resolution + hdr + 3d is way more immersive then a super stretched out pixelated mess. But the 3d effect is very subtle. I guess thats what you get from two cameras sitting 2cm next to each other (instead of the typical 6cm of two eyes). IMO the early reports from people trying the vision pro have entirely oversold it. Yes, it's better then your typical 180°, but the 3d is disappointing. The samsung galaxy and pixel pro with their 3 lenses next to each other, using the two outer ones could do much better, if they implemented something similar. So in typical apple fashion, they are hamstrung by their own design.
- ricardobeat 3y ago> the early reports from people trying the vision pro have entirely oversold it You can bet those demos were not recorded by an iPhone, but a pair of studio cameras spaced further apart. There were reports of apple working with movie studios to develop new hardware. We might still get that quality from professional productions.
- kderbe 3y ago"Spatial videos are captured in 1080p at 30 frames per second in standard dynamic range." Won't this look extremely soft and stuttery on a Vision Pro? Even on a standard screen these videos will look quite bad compared to the 4k60 recordings that every Apple phone since the iPhone 8 can record.
- sneak 3y ago30fps is not stuttery. Movies are 24fps.
- diffeomorphism 3y agoHence infamously stuttery: https://www.rtings.com/tv/tests/motion/24p https://www.rtings.com/tv/tests/motion/24p
- ricardobeat 3y agoStutter and judder aren’t the same, but the whole point of that analysis is that 24p without judder does not feel jumpy, so you want a TV that can play it as intended.
- theodric 3y agoI see motion aberration in film, so I can't agree with this
- extraduder_ire 3y agoWhat's the bitrate like? Even 4k looks noticeably soft to me, when it's filling my entire FOV.
- deleted 3y ago[deleted]
- ryandamm 3y agoCan anyone who has seen these demos confirm if they are 6 DoF (the perspective changes with viewer position) or merely stereo? The latter is boring and easy to implement, the former much more compelling and technically difficult.
- stavros 3y agoHow can they be 6 DOF if they're recorded with two cameras?
- andybak 3y agoAI magic. (They aren't 6dof but it is possible)
- starcraft2wol 3y agoAI is not magic, and still constrained by math. If you need 3 variables and only have 2, AI isn't going to do it.
- andybak 3y agoUnless we're talking about different things - it already exists. There's two parts to it - estimating a depth map ("monocular depth estimation") and inpainting the occluded parts.
- Torkel 3y agoThey have a lidar. So they have very accurate depth. This gives some limited possibility at extrapolating and showing things from different viewing angles. I think this is also how they can give a good stereo experience from cameras that are closer than the average human IPD.
- j4yav 3y agoIt would have to be AI generated, right? The cameras can't see behind what they can't see behind.
- stavros 3y agoIt seems like a glaring omission for the article to not explain what spatial videos are.
- perryizgr8 3y agoIt seems like this is just plain old stereo video. Probably why they don't explain it, keeps the "magic" alive.
- coldtea 3y agoProbably also because almost no consumer knows or uses the term "stereo video".
- perryizgr8 3y agoAs opposed to the incredibly well known and widely understood "spatial video".
- coldtea 3y agoWell, more people saw and know about "spatial video" after the keynote, than the totality of "stereo video" enthusiasts in the past 50 years.
- Luc 3y agoThere must be something else going on. Those lenses on the iPhone are perhaps 1 or 2 cm apart. That wouldn't result in a very natural 3D effect. So I guess they're doing some processing to generate 3D models, or something along those lines.
- masto 3y agoIt's just stereo. I don't know how it works, I'm not opticist, but I assume there's some kind of math involved. Or maybe it diminishes the 3D effect but it's still enough to look interesting.
- SirMaster 3y agoWhy doesn't the iPhone 14 Pro support it?
- dagmx 3y agoThe iPhone 15 Pro swapped the Ultrawide and telephoto lens positions such that the Wide and Ultrawide are now along a single axis. That’s likely necessary to shoot this format, as it uses the pair. On a 14 pro, you’d have to hold the phone diagonally and even then the sensors wouldn’t align with the axis causing cutoff.
- tsunamifury 3y agoTwo points are always on a single axis… it’s not like we lose our stereo vision when we tilt our heads.
- dagmx 3y agoI really don’t think you’ve thought through your curt response. 1. Clearly the array of cameras on these phones is along two axes, so the pedantic definition of two points on a line is irrelevant 2. You’d be shooting everything at a Dutch angle to maintain stereo if they weren’t aligned horizontally or vertically. 3. The sensors are rectangles and need to be aligned along their longest axes to maximize the most field of view.
- charcircuit 3y ago>3. The sensors are rectangles and need to be aligned along their longest axes to maximize the most field of view. The iPhone 14 could offer a lower FoV. The video format encodes the FoV of the recording already.
- dagmx 3y agoIt’s already cutting the resolution to 1080p to crop into the Ultrawide. How much lower would you go? Bear in mind this has to play back on one of the highest ppd displays on the market when the Vision Pro releases.
- dylan604 3y agoIn a former life, I spent quite a bit of time shooting 3D with various rigs, but ultimately, the resulting video was just never anything much more than gimmicky. Without 6 DoF type of movement, the 3D just had the look of flat objects moving around on flat planes. It's just 2.5D animation in realtime. So much work and effort goes into getting it shot correctly, posting it correctly, delivering it correctly, and then viewing it correctly. All for such meh result.
- amelius 3y agoNice. But where is my Apple Vision Pro?
- unsignednoop 3y ago[dead]
- wkat4242 3y agoIt would work better if the cameras were further apart, about 6cm, giving the same parallax as the average human eyes. Right now the system has to make up what is behind objects in the picture as it can't really see for rea.
- Xenoamorphous 3y agoMaybe the next step is to put one camera at each end of the phone?
- wkat4242 3y agoYeah that would work well. In fact it's a little too far off. But AI could bring the view point back together and in that case the furthest possible distance would be best indeed (to get the most parallax)
- TT-392 3y agoDid apple just rebrand 3D video as "spacial video"?
- deleted 3y ago[deleted]
- out_of_protocol 3y agoYou mean stereo rebranded as "3D"? I don't know any actual 3D video stuff with any kind of adoption.
- supermatt 3y agoWhat about volumetric video? It was used in the recent 7th guest VR game, for example. You can move around the actors and view them from any angle.
- out_of_protocol 3y agoCool, this one is real 3D, looks like. Ever heard about any content made in this format? VR games are, well, games - we know they can do actual 3D just fine
- supermatt 3y agoThere's a few use-case examples on arcturus (the company that did the capture for 7th guest) YouTube channel: https://www.youtube.com/watch?v=HXmEQKrv_fU https://www.youtube.com/watch?v=HXmEQKrv_fU
- ragazzina 3y agoYes, they did. I wonder if this title breaks the hn guidelines, after all the title is misleading.
- macguillicuddy 3y agoShameless plug - we recently released our $80 headset with support for spatial videos. It's a plastic headset, with two bluetooth controllers, that you put your phone into 'Google Cardboard style'. No need to wait for Vision Pro :-) https://www.zappar.com/zapbox/ https://www.zappar.com/zapbox/
- peblos 3y agoThis looks great! Any plans to ship to Australia?
- macguillicuddy 3y agoThanks :-D Yes - our Gen 2 model ships internationally and our site hasn't quite been updated yet. If you drop us a message at support@zappar.com we can get one to you.
- lancewiggs 3y agoGood on you for the product. Yes please update the site. I want to see if the ability to take spatial videos actually appears in local (NZ) iOS* before committing, and sending emails is a right pain versus a buy-now button "Some features may not be available in all regions...." is appearing on my update text.
- JKCalhoun 3y agoLove to get just the headset (sans Bluetooth controllers).
- macguillicuddy 3y agoYeah we've been thinking about a headset-only version too :-) The advantage of the controllers is they have 'Quest compatible' inputs - so content built for Quest can be ported with minimal fuss. That said there's many use cases that don't need the controllers, or for which hand tracking is sufficient, so we're definitely looking into it!
- replete 3y agoMy guess from this magical talk is that this allows recording of stereoscopic video using two cameras with an additional LIDAR data stream that with processing could enhance the 3D depth effect when viewed in VR. 3D Video+?
- yieldcrv 3y agoIts nice to see all of this trickle out for a whole ecosystem When I saw the Apple Vision headset, I thought it was a big miss to highlight it as a content creation device. The parent around their toddler with the unwieldy headset on needed to be ridiculued. I thought that should be a separate device, and just to let users discover on their own that the Apple Vision could technically record too. It was great to see that the iPhone 15 Pro was going to be that device, and just a software update away! Here we are now
- maaaaattttt 3y agoI called this when the vision Pro was announced [1]; so I'm going to take this opportunity to call something new (if you can tolerate my "bragginess" :)), even though both are not really crazy to predict. Apple is already producing new content for Apple TV+ in this spacial video format. It will probably be released along with the Vision Pro or before EoY 2024. [1] https://news.ycombinator.com/item?id=36225900 https://news.ycombinator.com/item?id=36225900
- gcanyon 3y agoAt the very least, Monarch Legacy of Monsters. Slow Horses -- which is an amazing series everyone should watch -- would benefit much less from this treatment.
- gandalfgreybeer 3y agoThere are some scenes in the latest episode of Foundation, particularly those in Trantor, where I found the shots/focus with "weird" focus and depths of field. I have a personal theory that those were already shot with spatial video capture, designed for Vision Pro in mind. Tangentially, if you gave up on Foundation on S1 like a couple of my friends, S2 is much much better. And if you're a book purist they sort of explain why there are differences between the show and the books in a relatively decent way.
- logtempo 3y agoAnd I bet an other Avatar movie with Apple money
- 3y ago
- masto 3y agoI played around with this when it came out in the beta: https://youtu.be/08hoyiHTxis https://youtu.be/08hoyiHTxis It took a few steps to get it out of the phone and onto YouTube in a way I could watch it in my VR headset, but the result is kind of nifty and nice to be able to produce with the phone I already have with me. I think a recent update to the Spatialify app that I was using makes it so you can do everything on the phone now (I had to run it through ffmpeg on my computer).
- petercooper 3y agoOn iPhone and other devices, spatial videos appear as regular, 2D videos and can be watched or shared just like any other video That's a shame. I would have expected the ability to have very subtle rotation/parallax effects, a la Facebook 3D photos (https://www.facebook.com/help/iphone-app/414295416095269 https://www.facebook.com/help/iphone-app/414295416095269). If you want people to actually start recording media to be viewed on the device en masse, I feel you need to throw them a bit of a carrot other than "well, you'll be able to look at them on some device you might own in years."
- samtheprogram 3y agoThe Vision Pro is right around the corner. This will be an upsell.
- afruitpie 3y agoA $3,500 first generation device is a difficult upsell.
- kevincox 3y agoI feel that it would be very hard to add value to the video with something like this. For photos I think it makes more sense because they are static and the movement adds extra depth. However for video I think it would just seem shaky. I'm not even sure I love the picture version, although it is a nice option to have.
- Syzygies 3y agoCasual book scanning could benefit from stereo cameras, if there's a way for iPhone apps to access this data. As a mathematician I've long wanted to figure out the code to match up several photographs for a 3D model of an open book, to correctly flatten out the image of each page. Bonus points to use AI to clean up the image, or convert to source that faithfully represents the page layout. There are apps that take one photo and then guess how to correct the distortion. They don't do as good a job as one could do with the actual 3D data. Many of us don't want to dedicate the space or money for a professional book scanner, and can't find an affordable flatbed scanner that scans all the way to one edge to better support books. I'll sometimes slice a second used copy of a cookbook, to scan every page using a sheetfed scanner. Grabbing one's phone should win ease-of-use over any other solution, if it could match the quality of these other approaches. 3D cameras would help that.
- hnuser123456 3y agoGiven that the distortion is always only going to be in the X dimension, unless the page is crumpled, could you just try to detect straight horizontal lines on the page, come up with the distortion to straighten them, and apply that to the whole page?
- Syzygies 3y agoI don't agree with your "X dimension" assumption. I've seen the poor performance of current phone apps. My point is that one could do better for less coding effort with access to a 3D model. Either a pair of photos, or a 3D camera.
- iamflimflam1 3y agoAgreed, most basic approaches just assume that a page is a rectangle and that you just need to apply perspective correction. Something that could map the 3D surface onto a sheet would be awesome.
- KolenCh 3y agoI’d guess, without knowing the details, this is already how it is done? Guessing can only take you so far, not to mention some pages has no straight lines (and I think you need orthogonal lines for the reconstruction to be effective?) and some has deceivingly straight lines (or even slanted lines.) What is described above is something I’ve seen in a documentary. They have a very fancy camera and the person holding the book can just slowly flipping through the pages, and it can reconstruct the flat pages. I guess the problem is not technology or mathematical model behind it, but how to make it cheap. It is analogous to Google’s self driving car, that you want to solve the same problem with much cheaper technology.
- markstos 3y agoThe great thing about spatial video will be that it takes up twice as much storage as regular video. This will give you more reason to buy a new iPhone with more storage or upgrade your iCloud account. Memories will be made and bills will be paid.
- huytersd 3y agoThank you for your blase, uninsightful comment.
- WentFullRetard 3y ago[dead]
- janandonly 3y agoIs anybody still using Live Photos? Another Apple exclusive format that never made it to the broader public as far as I can tell. This spatial/stereo/3D video format will die the same slow infamous death I guess?
- robinson7d 3y agoI’ve started again. I stopped for a while because they stopped feeling novel, but when scrolling through older photos from 5+ years ago, the photos of people that had Live enabled bring me a lot more emotions.
- hombre_fatal 3y agoI’ve noticed this too. The only reason I have live photos is because I went through a stretch of time where I accidentally took them, and now I wish more of my photos were.
- fy20 3y agoSamsung has what they call Motion Photos: https://www.samsung.com/au/support/mobile-devices/motion-photos/ https://www.samsung.com/au/support/mobile-devices/motion-pho...
- turquoisevar 3y agoI’ve enabled it by default, and I'm glad I did. It adds just a little bit more to the memory that was captured. My only gripe is of the unrealistic “want my cake and eat it too variety,” where I shoot more in RAW nowadays and use third-party camera apps, so those don’t get the Live Photos treatment. I’ve “solved” this by purposefully using the stock camera app for quick snaps of moments I want to be memorialized for sentimental reasons and reserving the RAW stuff for more artistic purposes, going even as far as taking another photo with the stock camera app if the situation calls for both. But honestly, I much more appreciate my Live Photos, if only because they’re rolled into the Moments the iPhone automatically creates, which make for a great memory.