18 ms·
4K4D: Real-Time 4D View Synthesis at 4K Resolution
- r3trohack3r 3y agoI'm skeptical. The code page leads to a repository that just has a README.md saying the source code is "coming soon" If it actually works, this is huge. I'd be using it tomorrow. But that first demo gif strikes me as something being off. The algorithm isn't picking up on the legs in the background painted on the wall... In the paper, I don't understand how what they've built could differentiate between a picture of someone painted on a wall and the part of the scene that should be rendered in 3D.
- Hard_Space 3y agoI think there has been some serious misinterpretation of what 'real time' means in the context of this paper; and, possibly, that the researchers have avoided overt clickbait claims because they knew the term 'real time' would do the work for them. This is not some neural codec that can convert any novel or unseen object live, like a kind of 3D YOLO - the paper mentions that it requires up to 24 hours of training on a per case basis. Nothing can be edited, no textures or movements - all you can do is remove people or speed them up or slow them down, and that's been possible in NeRF for a few years now.
- mattsan 3y agoThe speed of development in this space is incredible
- pard68 3y agoThis seems neat but I don't understand the use of 4D. It's not four dimensional. It's 3D with the ability to have an arbitrary perspective.
- jorgemf 3y agoTime is the forth dimension. The input data is a video, so the model learns the colors and the position of the elements (basically points). You can rende the scene from any angle at any time once the models is trained
- GaggiX 3y agoIt has a time dimension.
- renewiltord 3y agoDownvoted at the time I see it, but actually correct. It's based on K-planes https://arxiv.org/pdf/2301.10241.pdf https://arxiv.org/pdf/2301.10241.pdf which effectively splits each space-time relationship off from the spatial relationship. It's just mathematics, guys. The original NeRF paper talked about a 5D coordinate. You know like a k-dimensional vector?
- bassrattle 3y agoIt's volumetric video
- easeout 3y agoSo's a video game, and we call that "real-time 3D". Time is mentioned, but it isn't counted again as a dimension, perhaps because any given momentary view is a time slice, not a time range like it is an XYZ range.
- datameta 3y agoI think the difference is that in a video game you are in one location only at any given moment and things travel only forward in time. We can view from any location at any time in volumetric video.
- ben-schaaf 3y agoSome games absolutely let you do that. It's not a limitation of the medium.
- datameta 3y agoI'm curious what are a few examples? I've missed the last decade of new games largely.
- ben-schaaf 3y agoCS:GO, TF2, GTA5 and Trackmania (and likely many more) have replay systems where you can pause, play and rewind with a freefly camera. Lots of games have a rewind mechanic: Grid, Baba is You & Viewfinder come to mind. Others have a "Photo Mode" where you can pause with a freefly camera: Starfield & Witcher 3 come to mind.
- fragmede 3y agoTalking about rewind mechanic in games, you can't possibly leave out Braid. https://store.steampowered.com/app/26800/Braid/ https://store.steampowered.com/app/26800/Braid/
- madmads 3y agoYea it's probably to have a catchy name and get some attention. Although it's technically accurate to call it 4D since it includes time, I think 3D video recording would probably get the point across to more people in a less sensationalist way.
- jayd16 3y agoIs it technically accurate? Seems like its actually 6Dof view angles + time. The paper mentions 4D view, 4D point cloud, dynamic 3D scene and 4D feature grid.
- jahewson 3y agoIf I’m understanding the paper correctly then the four dimensions are the position, density, radius, and color of the spheres in their volumetric model. So for any given viewing position and point in time, their model produces a 4D scene that is then rasterized to 2D.
- rangewookie 3y ago3D + video (motion) It's not a 3D model that is animated using a skeleton and keyframes like traditional 3D. It's many consecutive 3D models that create the illusion of continuous motion (aka video). 4D is the name that has come to describe the jump from static 3D models (photogrammetry) to 3D "video" models.
- firtoz 3y agoWhen it comes to volumetric clouds, the D stands for how many dimensions you store the points in. A 2d volume (similar to an image) has pixels stored in 2d coordinates A 3d volume has points stored in 3d coordinates. Imagine an image for every vertical slice of a brain scan. A 4d volume has points stored in 4d coordinates, where the newest dimension is time. Imagine a 3d volumetric capture for each frame in time.
- pineconewarrior 3y agoIncredible! How many cameras does this method require? As far as I can tell from the paper it still generates from multi-view source data. I can't say for sure but it seems like a large number from what I can parse as a layman.
- fooblaster 3y agoI'm no expert but the DNA rendering dataset uses 60 synchronously captured cameras.
- cchance 3y agoSo, something we could easily see done at say... an NBA event or football field... hell I imagine i can think of some ... adult use cases that would probably make a bundle off of tech like this if it can be optimized down... as my favorite youtuber would say ... WHAT A TIME TO BE ALIVE!
- accrual 3y agoThis seems unprecedented. Imagine if you have this but you can update the scene programmatically. Ask your AI to change the location or actors. Now you have a very convincing artificial scene with anything you can imagine in it.
- mistermann 3y agoWith all the various more specialized demos I've seen, that day doesn't seem too far away.
- MPSimmons 3y agoI imagine that most people will have the capacity in their home machines to render stuff like this in video games in 10ish years.
- fragmede 3y ago10? I think we'll both be surprised by where we're at in 3. That's October, 2026.
- MPSimmons 3y agoRe-reading the paper, I totally missed the hardware that they did this on, which was consumer-level GPUs, so I think you may be right - 3 years is probably a good time frame for seeing this kind of tech in commercial games. My reasoning for initially saying 10ish years: GPU architecture release cadence is frequent, but not THAT frequent. NVidia released the Ampere (RTX 30X0) in 2020, 3 years ago. Ada Lovelace (RTX 40X0) was released a year ago, in Oct 2022. It's _possible_ to use 4090s to do medium neural-network things right now, but to be on the leading edge, you need multiple datacenter-level cards, each of which is > $5k. Even though it's possible to do crazy things with the most recent generation of GPUs, there don't yet exist games that really take advantage of it. The closest that I'm aware of is the Cyberpunk-level games that make native use of raytracing capabilities. I think it'll probably be a year or two before we see games come out that really require the level of 40-series cards ( or 7900-series if you're of the AMD persuasion), which is a lag-time of ~3 years after the cards were released. I think the software development time and market saturation are the driving factors in the gap here. I was under the mistaken impression that the video output was produced on the datacenter cards. They got reasonable performance on a 30-series NVidia card. In 3 years, it's totally reasonable to expect that AAA game players will have that level of GPU performance in their gaming machines, so yeah, I think you're right.
- sosodev 3y agoDoes anybody else get the impression that holograms are inevitable? This type of tech seems like the medium now all we need is a good way of displaying them.
- judge2020 3y agoWe've been making plausible 3d models of humans via game engines for a long time now. The limiter is bringing holograms to real life.
- machdiamonds 3y agoMixed reality headsets can already do holograms, and there is a clear path to improving them.
- cooper_ganglia 3y agoThis is where my mind goes with all of these advancements. Always and immediately to how they will contribute to making a photorealistic Holodeck via mixed reality a closer reality.
- zardo 3y agoHolographic displays do exist, though I'm not aware of any being commercially available yet. https://www.nature.com/articles/s41467-020-19298-4#MOESM3 https://www.nature.com/articles/s41467-020-19298-4#MOESM3
- coffeebeqn 3y agoWow that site really killed my phone for a minute or so
- RyanMathewson 3y agoIt crashes Chrome on my iPhone.
- deleted 3y ago[deleted]
- aravindgp 3y agoIt crashes chrome on my android phone
- ShamelessC 3y agoIt crashes Chrome on my smart fridge.
- gloyoyo 3y agoCrashes Edge Canary on mine.
- sroussey 3y agoNo promo here. 15 pro
- nuancebydefault 3y agoNo probs here. Firefox on an 'old' samsung galaxy.
- skykooler 3y agoIt brought my laptop to a crawl, too.
- keras2 3y ago[dead]
- darknavi 3y agoAlways fun to see ImGui used in random projects. What a gift to software engineers everywhere!
- lowbloodsugar 3y agoWatched the video for where the idea of IMGUI came from and it was frankly terrifying. I mean, just the assumption that the frame rate is fast enough that mouse up and mouse down occur in two different passes.
- NavinF 3y agolink to said video?
- calibas 3y ago> we precompute the physical properties on the point clouds for real-time rendering. Although large in size (30 GiB for 0013_01), these precom- puted caches only reside in the main memory and are not explicitly stored on disk, Does the cache size scale linearly with the length of the video? 0013_01 is only 150 frames. And how long does the cache take to generate?
- keras2 3y agoLooks so, suspect the authors precomputed everything they can to reach the highest frame rate. Like predecoding all frames in a movie into raw pixels? I think volumetric video should be thought of as a regular video, where the decoding and playback happen at the same time. A few papers down the line this could be easily implemented.
- cchance 3y agoHoly sh*t, can you imagine a year from now if they start using something like this for concerts or basketball games? Like imagine rewatching a basketball game but being able to move the camera on the court???? Might not be possible yet but this shows the techs possible. Let alone someday being able to scale it to realtime someday maybe lol
- soylentgraham 3y agoWe did this (as a side effect) for the premier league ~2009-2012 (liquidated JUST before VR appeared, where the content worked fantastically, and then ~2014 with the moverio glasses, even better in AR) We did live player tracking (~33 cameras) on-site at every game, and for fun rendered players fifa-style free-camera. We even did some renders (capture of realtime engine) for canal+ highlights as an experiment. edit: my own gpgpu-only,(frag shaders :), sub-100ms, uncalibrated-cameras (footage directly from sky/match of the day) r&d a few years later, also works really well on a LookingGlass https://twitter.com/HoloSports/status/1327375694884646913 https://twitter.com/HoloSports/status/1327375694884646913 (I took this to sky sports but they said it was a bit too in-the-future)
- MBCook 3y agoYou want a killer app for VR/AR goggle style things? You’re right this would be amazing. Apple demoed some kind of volumetric video to the press with the Vision Pro. There was a short clip of a concert and an NBA game (Nuggets?) among other things. I heard a number of people said it was like being there. This is a step past that. Apple’s was recorded from some kind of special camera rig (I assume), but I seriously doubt it was full volumetric video from a large number of angles. It sounded more like volumetric video if you were stuck in a (very good) seat in the venue. I’d be curious to know just how much horsepower it takes to play these back.
- keras2 3y agoFrom the paper, it seems a 3060 is enough for 60FPS for the DNA-Rendering dataset. On the full-screen datasets it manages 25 fps. A 4090 might be needed to stay above 60 fps. Still pretty heavy I’d say but it certainly came a long way and shows us real volumetric video is doable.
- cchance 3y agocan't sit down to read this now anyone know if this is using standard nerfs or gaussians?
- IshKebab 3y agoNeither.
- rvz 3y agoNow that is what I call, unreal. Literally.
- rlt 3y agoAnd as usual the first application of this new technology will be porn. But seriously, this is killer technology for AR/VR.
- melchebo 3y agoI wonder what kind of rig is needed for recording that. It has to be at a least a few different viewpoints.
- addandsubtract 3y agoThis is my question as well. What's the input required to generate these 3D scenes? Is RGB video enough or does it also require spacial data? Is planning around the same scene enough or does it require multiple cameras?
- JansjoFromIkea 3y agoRelated: there was a small project that done similar stuff with Kinect v2 a ~7 years ago that was really impressive for the time. https://github.com/MarekKowalski/LiveScan3D https://github.com/MarekKowalski/LiveScan3D Now that Kinect v2 can be found for next to nothing and is very easy to mod to use without an expensive adaptor it's a bit of a shame the project was abandoned, from what I've seen the bigger limitations of the project can be overcome (only one Kinect per PC, mainly).
- keras2 3y ago[dead]
- pk-protect-ai 3y agoThere was a paper related to NeRF dynamic scenes yesterday, but FPS and quality of this one is so superior!
- sandworm101 3y agoVery cool renderings, but ironically my browser is having a heck of a time rendering their website. The short videos keep jumping around, starting and stopping randomly... which i guess is very VR.
- lima 3y agoThese are just normal HTML5 <video> elements and the jumping around is part of the video material.
- belter 3y agoOn Firefox Android is crashing.
- sandworm101 3y agoYup. I was viewing on my android tablet using chrome. It wasn't crashing but it was very jittery.
- tomalaci 3y agoAdd volumetric sound, integrate VR and you almost have recreated braindance from the Cyberpunk 2077 game. Doesn't seem that far off in the distance. The missing component from complete braindance would be integrating physical senses. AFAIK we are pretty far away from having anything revolutionary in that domain. Would love to be proven wrong, however.
- kridsdale1 3y agoYou can always hire someone to come over and poke you while you have the headset on.
- the8472 3y agoThose samples don't look like 4k to me.
- mkaic 3y agoThe render resolution is 4K, but the subjective image quality and sharpness is not quite up to what you would expect for 4K.
- fullarr 3y agoThe effect is cool but I must be the only person on this website that doesn't see a future for it Seems very niche, with massive data size restrictions, making it difficult to broadcast or stream on existing infrastructure. But even if you solved the infrastructure problem, it feels like a gimmick that would be uninteresting pretty quickly. Sporting events maybe benefit a bit by being able to find the right angle for any shot, but honestly they will probably just find the best angle and post that video as a clip
- scheeseman486 3y agoMovement is per-object, meaning camera movement can be encoded as a vector while the scene it's moving within remains largely static, that leaves a lot of redundant data. Streams could be frustum culled based on the user/camera's perspective. There's potential for high compressibility.
- LastMuel 3y agoYou may be missing the benefit of capturing historical data. Image being able to don AR glasses and to be present for the moon landing or experience the assembly of the Eiffel tower from a position at its base. Entertainment is one thing, but capturing critical events in full dimension would be world changing.
- stevofolife 3y agoEveryone wear AR glasses will be recording the event. The recordings are streamed for 4K4D processing and streamers can watch the event remotely.
- shultays 3y agoOne of my favorite things in VR is google maps, I like "walking" around in cities without leaving my house. I am longing for the day that we can also do this
- andybak 3y agoI presume you mean Google Earth VR: https://store.steampowered.com/app/348250/Google_Earth_VR/ https://store.steampowered.com/app/348250/Google_Earth_VR/ I'm keen that it gets the credit it deserves - I'm terrified it will stop working one day soon and the world will have lost a true wonder. (They did just open up the underlying APIs so it would be possible to build a replacement now - although it's free in preview and pricing hasn't been announced - so no idea if it's economically viable)
- shultays 3y agoYeah, that is the application I meant. I would miss it too!
- bloopernova 3y agoIt will be very interesting to watch how tech like this affects mainstream society. I imagine pornography will use it at some point soon. Maybe something like chaturbate where your interactions with the cam performer are more customized? Could it be used with CCTV to reconstruct crime scenes or accidents? Wedding videos might be a popular use, being able to watch from new angles could be a killer app. Or a reworking of the first Avengers movie, view all the action from multiple viewpoints. And all this will probably be built in to the pixel 18 pro or something.
- rasz 3y ago>CCTV to reconstruct crime scenes Here is Jack Black demoing similar NSA technology in 1998 https://www.youtube.com/watch?v=3EwZQddc3kY https://www.youtube.com/watch?v=3EwZQddc3kY
- actionfromafar 3y agoGood catch! “It can hypothetise” Also the Star Trek Next Generation forensics episode with the holodeck…
- qingcharles 3y agoThere's an ST:TNG episode I remember too where they have an image and they get the computer to back-trace all the reflections in the image to produce what isn't easily seen.
- VikingCoder 3y agoI went to high school with the girl who plays the sales clerk standing next to Will Smith, Ivana Miličević.
- qingcharles 3y agoIt turns out this was actually a documentary.
- thomastjeffery 3y ago
- spandextwins 3y agoSoon I'll just put on a headset, sit on my chair with my food tube and 'bate all the time!
- cooper_ganglia 3y ago"Go away, I'm 'batin!"
- icyriver2023 3y agonice
- kridsdale1 3y agoI only first heard about this Gaussian Splat field at the start of this week, and it seems it has advanced a decade’s worth by Wednesday!
- andybak 3y agoHold on to your papers...
- MPSimmons 3y agoHow would you stream the output of something like this, if you wanted to? So that people could continue to change the viewpoints. You couldn't possibly stream the full list of voxels generated by capturing the entire image with all of the cameras, right? That would probably exceed PCI bandwidth capabilities. You'd need the server-side to generate models, send those models, and then stream the vectors?
- avrionov 3y agoRed Dwarf predicted this: https://www.youtube.com/watch?v=JMIHNiR3CP8 https://www.youtube.com/watch?v=JMIHNiR3CP8
- corab 3y ago[flagged]
- qingcharles 3y agoRed Dwarf was a documentary sent back in time. What's funny here is the use of the word "uncrop"! I had never heard that word used before DALLE*2 was released. And I've been working in computer graphics for 30 years lol. Also I watched a lot of Red Dwarf. Was that word known before Red Dwarf?
- sheepscreek 3y agoI imagine this can be used as an insanely efficient compression scheme. Transitions in videos may not need as many frames using this.