6 ms·
3D reconstruction from a single image. They stress the examples are not curated, appears to... well, gosh darnit, it appears to work. If it runs fast enough I
by HopenHeyHi 4y ago
3D reconstruction from a single image. They stress the examples are not curated, appears to... well, gosh darnit, it appears to work.
If it runs fast enough I wonder whether one could just drive around with a webcam and generate these 3d models on the fly and even import them into a sort of GTA type simulation/game engine in real time. (To generate a novel view, Zero-1-to-3 takes only 2 seconds on an RTX A6000 GPU)
This research is based on work partially supported by:
- Toyota Research Institute
- DARPA MCS program under Federal Agreement No. N660011924032
- NSF NRI Award #1925157
Oh, huh. Interesting.
Future Work
From objects to scenes:
Generalization to scenes with complex backgrounds remains an important challenge for our method.
From scenes to videos:
Being able to reason about geometry of dynamic scenes from a single view would open novel research directions --
such as understanding occlusions and dynamic object manipulation.
A few approaches for diffusion-based video generation have been proposed recently and extending them to 3D would be key to opening up these opportunities.
- TylerE 4y agoSeems like there is a bit of a gap between “runs at 0.5 fps on a $7000 workstation-grade GPU with 48GB of VRAM” and consumer applications. With the fairly shallow slope of the GPU performance curve overtime, I don’t see them just Moores Lawing out of it either. this would need two, maybe three orders of magnitude more performance.
- jiggawatts 4y agoComputer power goes up exponentially thanks to Moore's law. Sprinkle some software optimisations on top, and it's conceivable for that to be running at interactive framerates on consumer GPUs within 5-10 years.
- ffitch 4y agothe processing may as well shift to the clouds. With the subscription fee, of course : )
- TylerE 4y agoUntil we break the speed of light, I’m very bearish on cloud gaming. It just feels so bad. You’ve got like 9 layers of latency between you and the screen.
- fooker 4y agoYou don’t have to break the speed of light, just have the ping below human perception. ~20ms is that threshold, but even 40ms latency is barely noticeable for single player games.
- TylerE 4y agoA wireless controller ALONE is already over 20ms, and that’s before you touch the network, actually doing with that input, wait for the display to redraw… At a 20ms total round trip, that only buys you about a 1500 mile radius, again completely ignoring all other latencies.
- enlyth 4y agoIt's quite noticeable actually, and it adds up, it's not just an extra 20ms. For casual gamers and turn based games maybe it could work, as a niche. For FPS, multiplayer, ARPG, and so on, it's a dealbreaker, anything over 100ms feels too sluggish. We should be happy we have so much autonomy with our own hardware, I don't want some big cloud company to be able to tell me what I can play and render, unless we want the "you will own nothing and be happy" meme to become reality.
- TylerE 4y agoI actually, in my testing, JRPG/other turn based games were amongst the worst because there is so much “management” (inventory, loot, gear, etc) and the extra lag really throws you off
- jimmySixDOF 4y agoOne possible definition of Edge Compute is GPU capacity at every last mile POP
- nitwit005 4y ago> Computer power goes up exponentially thanks to Moore's law If you look at a graph, that stopped being true well over a decade ago.
- jiggawatts 4y agoOnly for single-threaded programs. Multi-threaded performance continued along the curve unabated.
- nitwit005 4y agoMoore's law is specifically about the number (density) of transistors in an integrated circuit. You could always get as much parallelism as you wanted by adding more chips.
- HopenHeyHi 4y agoOf course there is a gap. This is at the exploratory proof of concept stage. The fact that it works at all is what is interesting. Furthermore once you've identified the make and model of the car, its relative position in 3d, any anomalies -- that ain't just a Ford pickup, it is loaded with cargo that overhangs in a particular way -- its velocity, 'etc -- I'm quite sure that extrapolating additional information from the subsequent frames will be significantly cheaper as you don't have to generate a 3d model from scratch each time. I think this is a viable exploratory path forward. Make it work <- you are here Make it work correctly Make it work fast Edit: Scotty does know ;)
- scotty79 4y agoI prefer: Make it work <- you are here Make it work correctly Make it work fast
- deleted 4y ago[deleted]
- frozenport 4y ago>> fairly shallow slope of the GPU performance curve overtime Not true.