8 ms·
Show HN: I made an iOS app recording RGBD videos and a web app playing them
- pizza 5y agogetting real braindance vibes [0] - for added fun try adding multiple sources with sequence alignment w/ e.g. dynamic warping based on the correlation between the audio tracks of 2 RGBD videos! Then look into NERFs! [1] [0] Cyberpunk 2077 - Braindance Gameplay - https://www.youtube.com/watch?v=KXXGS3MGCro https://www.youtube.com/watch?v=KXXGS3MGCro [1] https://phog.github.io/snerg/ https://phog.github.io/snerg/
- baq 5y agoMy socks are blown off. If you could add a feature to measure distance when paused, it’d make renovation planning so much less stressful - I’d just walk around the place while recording a video and could measure anything post factum. No more ‘will it fit?’ (Guess who’s doing a renovation now)
- asutekku 5y agoYou can do it already if you 3D-scan your place with this for example https://poly.cam/ https://poly.cam/
- withinboredom 5y agoI’ve used this: https://apps.apple.com/nl/app/3d-scanner-app/id1419913995?l=en https://apps.apple.com/nl/app/3d-scanner-app/id1419913995?l=... Some places like IKEA will let you get a model of the furniture that you can drop into something like Blender to see how it fits. When scanning my wife’s childhood home we found enough space for a large hidden room in the house. We couldn’t figure out how to access it without looking too suspicious (we didn’t want anyone to know we knew).
- hanseul 5y agoThat sounds like a cool idea! Maybe I can make this website to support phone AR, then you can place the video on top of some ruler or furniture for your measuring purposes? It's nice to hear about potential use cases and good luck with your renovation!
- tpoacher 5y agoWait, what? My humble phone frontcam had depth all this time??
- olso 5y agoIs this different from Snapchat/SparkAR 3D effects?
- ilolu 5y agonice work. Can you point me to resources on learning about capturing depth from iPhone cameras?
- hanseul 5y agoAs I had prior experience with RGBD cameras, for iPhones, I've learned how to use their library than the fundamentals. For example, rear depth information comes from ARKit and more specifically from ARFrame and capturedDepthData property: https://developer.apple.com/documentation/arkit/arframe/2928208-captureddepthdata https://developer.apple.com/documentation/arkit/arframe/2928....
- ChrisMarshallNY 5y agoNice work! I’m pretty sure it will only work with the last one or two gens of the phone, but it gives you an idea of what Apple is doing, under the hood. Apple’s ARKit is pretty cool. I’d like to do more with it, but I haven’t had an excuse, so far.
- hanseul 5y agoThanks, I agree that what Apple is doing is pretty cool, both hardware and software-wise. About supporting devices, while the better quality rear depth cameras are only available for 12/13 pro devices, the front depth camera is included in all iphones supporting face ID (3 years old), which is sufficient for running this app!
- tomduncalf 5y agoThis is seriously cool, great work! You can get that “Radiohead video” effect by pausing and panning around a video of your face :) I might suggest changing the name of this post as I had no idea what RGBD was initially, maybe “I made an iOS app recording videos with depth information and a web to play and explore them in 3D” or something (that’s probably too long for HN!)
- hanseul 5y agoSpending several years in a VR lab made me start thinking RGBD was some common term, but for sure I'll follow your advice and use more normal words next time. And you were right that it's probably too long since the above title was few letters short the limit already!
- polskibus 5y agoThis is great! Any plans for a point cloud/ 3d reconstruction app?
- hanseul 5y agoBetween this app with RGBD videos and 3D reconstruction, I think in the future, there can be some software turning RGBD videos to reconstructed point cloud/mesh, so that you don't need to run a dedicated 3D reconstruction app at the time recording. Maybe I'll be able to work on it like... 3 years later...?
- mgmgmgmgmg 5y agoI don't understand.
- tuyiown 5y agoHave you tried the virtual joystick (bottom left) ? You can _move_ within the video
- kdot 5y agoThis definitely gives off iGoggle vibes, would love to experience this with an A/R headset.
- hanseul 5y agoThe codebase was originally written for a HoloLens app, so you sort of correctly guessed where it is coming from! I think this app works the best with AR headsets and it also supports AR headsets through WebXR. It's just that the AR headsets are really hard to find outside of labs...
- lloydatkinson 5y agoI think I'm missing a lot of context here. Specialist cameras repurposes into blurry pixelated streaming devices?
- hanseul 5y agoI think there are two missing things here. One is that this is not coming from a specialist camera, but from iphone cameras. Many mobile phones (including some android ones) already have hardware to capture depth. And the next is about color information. This video has no information loss in terms of color compared to ordinary videos. It's just that they are mapped into 3D space. The example you are watching has 720p color resolution.
- jefe_ 5y agoThis is really awesome. I think the ability to record the camera position / perspective changes atop the video (while shooting and after shooting) for export to flat video or gif would be really cool. It would allow people to record creative clips and share them on existing platforms.
- maartn 5y agoWow, this reminds me of a scene in Minority report https://www.youtube.com/watch?v=OSDqZeI2WlA https://www.youtube.com/watch?v=OSDqZeI2WlA
- hanseul 5y agoI remember watching Minority report again last year and being really surprised watching this scene!! I was like that looks like my research project but this movie is 20 years old???
- rock_artist 5y agoWould be nice though if this would ask for confirmation before starting to download 11mb. While not a lot. Some data plans are capped
- supermatt 5y agoFYI, Almost half the current top 10 are ~10MB or more, and the rest are ~3MB. - This at 11MB - Simula One homepage weighing it at 72MB (wtf) - Aesthetics job posting is 9MB (to display a redirection link...) - Typefully (tweet 2 blog) is 10Mb I dont think its possible to browse the web these days and manage a reasonable data cap when a handful of sites can consume so much :(
- dylan604 5y ago[flagged]
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- pmarreck 5y agoI have no idea what I'm looking at and must have missed a memo oh, the D is Depth? Is the depth measured by a sensor or determined semantically via ML or something?
- hanseul 5y agoYes, D is depth! Sorry that it wasn't clear. The depth here comes from sensors.
- layer8 5y agoFYI: RGB-D = an RGB image with per-pixel depth information, as for example captured by devices like Microsoft Kinect.
- kekeblom 5y agoIn case someone is interested in recording RGB-D videos for offline processing, I wrote an app for this https://apps.apple.com/us/app/stray-scanner/id1557051662 https://apps.apple.com/us/app/stray-scanner/id1557051662. It simply writes the raw data files to disk, which you can export to the cloud/your computer.
- hanseul 5y agoHi everyone, My name is Hanseul Jun who is currently a graduate student working on augmented reality and telepresence systems. During my graduate studies, I have been working on code related to this app since four years ago, always wanting to bring telepresence technology outside of labs. Hope you enjoy this! You can also share a RGBD video via a URL like this example using my app (https://apps.apple.com/us/app/telegie/id1593918560 https://apps.apple.com/us/app/telegie/id1593918560) using a iphone that contains a face ID camera or a LIDAR camera. As I have been enjoying reading comments from here and any comments would be appreciated!
- data_scientist 5y agoAwesome! It looks like we could extract the depth from the video, but my knowledge of ffmpeg is too limited for that. Do you know a way to do it?
- hanseul 5y agoBy default, mp4 files from iphones do not include depth. The RGBD videos from telegies are custom mkv files that include depth (done by directly using libmatroska, not via ffmpeg), which the webapp can display.
- teruakohatu 5y agoI have to say this is amazing! Really good work. Might I suggest slightly more interesting examples than kicking a ball from your desk. What about a cat playing with a toy? Or a model train going round a track?
- mike256 5y ago
- dheera 5y agoThat's a good rule for products, but this is HN. As a reader, half-baked projects are super welcome.
- teruakohatu 5y agoGoogle's homepage is 2.3mb right now. This page loads 16.2mb, so about as heavy as loading Google 7x. For comparison, Youtube's homepage is 14.2mb for me. So really it is not that much.
- justwalt 5y agoThat seems a bit drastic!!
- hanseul 5y agoWhile I cannot agree with deleting this site, which would be quite a sad thing for me, I agree that the website can be improved a lot regarding this. I am planning to make the website to download more gradually than downloading the whole video first, which will let you escape from the site without downloading that much!
- dheera 5y agoMaybe we can use a neural net to fill in the shadowed regions. I have only looked at 2D inpainting algorithms and not sure what the state-of-the-art in 3D inpainting is though.
- hanseul 5y agoRight, the shadowed region does make the scene look less realistic, since there is no such thing in the real world. While the lazy part of me says it's okay to leave it this way, 3D inpainting is definitely something I should look into.
- kevinsundar 5y agoYou should add a component that tracks your head in the browser and then re-renders the perspective. Should be fairly easy and would make it a lot more immersive.
- hanseul 5y agoWhile that is a great which I will definitely consider for the future, this website at least supports WebXR. In case you have a VR headset, please consider trying to run this video using one!
- blensor 5y agoI did run it in my Quest which really makes the experience quite on another level. Did you record those videos holding the camera in your hand? Because they have a shake to it that does not correspond to my heqd movement. So if anyone wants to try it in the headset be sure that you are not nauseated by movements that don't correspond to you own head
- hanseul 5y agoThanks for trying it with the quest and for the feedback. I really haven't thought about this issue. Yes, you are right that I have held the camera with my hand. I'll think about this and probably will need to do something equivalent to video stabilization.
- blensor 5y agoThe easiest thing would probably be to just have the first video the people see be shot on a tripod, that will show everyone what it can be. I have to say that I am actually thinking about using something like that for our VR fitness game. In that case the scenes would be shot from a fixed angle anyway. I'd just have to figure out how to get that into GodotEngine and without killing the performance.
- noajshu 5y agoThis is so cool! I’ve been playing around for a while now with different objects and types of lighting. Now I’m getting this error when I try to post: {"timestamp":"2022-02-23T05:57:19.930+00:00","status":500,"error":"Internal Server Error","path":"/record"}
- hanseul 5y agoI'm really sorry about this... I don't think I can fix it at the moment but I'll keep work on making the app more stable. And thanks a lot for trying the app!!!
- deleted 5y ago[deleted]
- randomcatuser 5y agoSolid!! Have you considered working with deep learning type stuff?
- hanseul 5y agoThank you!! I have no experience in deep learning yet, but I can see it being useful for this system and would like to learn about it!
- dongjun2 5y agoAmazing!!!
- DaniDaniel5005 5y agoSo its using the power of face id/true depth? Its amazing! 5 star...
- hanseul 5y agoThank you, thank you!!