15 ms·
Voyager – An interactive video generation model with realtime 3D reconstruction
- mingtianzhang 1y agoWhat's your opinion on modeling the world? Some people think the world is 3D, so we need to model the 3D world. Some people think that since human perception is 2D, we can just model the 2D view rather than the underlying 3D world, since we don't have enough 3D data to capture the world but we have many 2D views. Fixed question: Thanks a lot for the feedback that human perception is not 2D. Let me rephrase the question: since all the visual data we see on computers can be represented as 2D images (indexed by time, angle, etc.), and we have many such 2D datasets, do we still need to explicitly model the underlying 3D world?
- AIPedant 1y agoHuman perception is not 2D, touch and proprioception[1] are three-dimensional senses. And of course it really makes more sense to say human perception is 3+1-dimensional since we perceive the passage of time. [1] https://en.wikipedia.org/wiki/Proprioception https://en.wikipedia.org/wiki/Proprioception
- WithinReason 1y agothe sensors are 2D
- soulofmischief 1y agoTwo of them, giving us stereo vision. We are provided visual cues that encode depth. The ideal world model would at least have this. A world model for a video game on a monitor might be able to get away with no depth information, but a) normal engines do have this information and it would make sense to provide as much data to a general model as possible, and b) the models wouldn't work on AR/VR. Training on stereo captures seems like a win all around.
- WithinReason 1y ago> We are provided visual cues that encode depth. The ideal world model would at least have this. None of these world models have explicit concepts of depth or 3D structure, and adding it would go against the principle of the Bitter Lesson. Even with 2 stereo captures there is no explicit 3D structure.
- soulofmischief 1y agoIncreasing the fidelity and richness of training data does not go against the bitter lesson. The model can learn 3D representation on its own from stereo captures, but there is still richer, more connected data to learn from with stereo captures vs monocular captures. This is unarguable. You're needlessly making things harder by forcing the model to also learn to estimate depth from monocular images, and robbing it of a channel for error-correction in the case of faulty real-world data.
- WithinReason 1y agoStereo images have no explicit 3D information and are just 2D sensor data. But even if you wanted to use stereo data, you would restrict yourself to stereo datasets and wouldn't be able to use 99.9% of video data out there to train on which wasn't captured in stereo, that's the part that's against the Bitter Lesson.
- deleted 1y ago[deleted]
- soulofmischief 1y agoYou don't have to restrict yourself to that, you can create synthetic data or just train on both kinds of data. I still don't understand what the bitter lesson has to do with this. First of all, it's only a piece of writing, not dogma, and second of all it concerns itself with algorithms and model structure itself, increasing the amount of data available to train on does not conflict with it.
- 2OEH8eoCRo0 1y agoAnd the brain does sensor fusion to build a 3d model that we perceive. We don't perceive in 2d There are other sensors as well. Is the inner ear a 2d sensor?
- AIPedant 1y agoInner ear is a great example! I mentioned in another comment that if you want to be reductive the sensors in the inner ear - the hairs themselves - are one dimensional, but the overall sense is directly three dimensional. (In a way it's six dimensional since it includes direct information about angular momentum, but I don't think it actually has six independent degrees of freedom. E.g. it might be hard to tell the difference between spinning right-side-up and upside-down with only the inner ear, you'll need additional sense information.)
- reactordev 1y agoIncorrect. My sense of touch can be activated in 3 dimensions by placing my hand near a heat source. Which radiates in 3 dimensions.
- Nevermark 1y agoYou are still sensing heat across 2 dimensions of skin. The 3rd dimension gets inferred from that data. (Unless you have a supernatural sensory aura!)
- AIPedant 1y agoThe point is that knowing where your hand is in space relative to the rest of your body is a distinct sense which is directly three-dimensional. This information is not inferred, it is measured with receptors in your joints and ligaments.
- Nevermark 1y agoNo it is inferred. You are inferring 3D positions based on many sensory signals combined. From mechanoreceptors and proprioceptors located in our skin, joints, and muscles. We don’t have 3-element position sensors, nor do we have 3-d sensor volumes, in terms of how information is transferred to the brain. Which is primarily in 1D (audio) or 2D (sensory surface) layouts. From that we learn a sense of how our body is arranged very early in life. EDIT: I was wrong about one thing. Muscle nerve endings are distributed throughout the muscle volume. So 3D positioning is not sensed, but we do have sensor locations distributed in rough and malleable 3D topologies. Those don’t give us any direct 3D positioning. In fact, we are notoriously bad at knowing which individual muscles we are using. Much less what feeling correspond to what 3D coordinate within each specific muscle, generally. But we do learn to identify anatomical locations and then infer positioning from all that information.
- reactordev 1y agoYour analysis is incorrect again. Having sensors spread out across a volume is, by definition, measuring 3D space. It’s a volume. Not a surface. Humans are actually really good at knowing which muscles we are using. It’s called body sculpting. Lifting. Body building. And all of that. So nice try.
- AIPedant 1y agoIt is simply wrong to describe touch and proprioception receptors as 2D. a) In a technical sense the actual receptors are 1D, not 2D. Perhaps some of them are two dimensional, but generally mechanical touch is about pressure or tension in a single direction or axis. b) The rods and cones in your eyes are also 1D receptors but they combine to give a direct 2D image, and then higher-level processing infers depth. But touch and proprioception combine to give a direct 3D image. Maybe you mean that the surface of the skin is two dimensional and so is touch? But the brain does not separate touch on the hand from its knowledge of where the hand is in space. Intentionally confusing this system is the basis of the "rubber hand illusion" https://en.wikipedia.org/wiki/Body_transfer_illusion https://en.wikipedia.org/wiki/Body_transfer_illusion
- Nevermark 1y agoI think you mean 0D for individual receptors. Point (I.e. single point/element) receptors, that encode a single magnitude of perception, each. The cochlea could be thought of 1D. Magnitude (audio volume) measured across 1D = N frequencies. So a 1D vector. Vision and (locally) touch/pressure/heat maps would be 2D, together.
- AIPedant 1y agoNo, the sensors measure a continuum of force or displacement along a line or rotational axis, 1D is correct.
- Nevermark 1y agoThat would be a different use of dimension. The measurement of any one of those is a 0 dimensional tensor, a single number. But then you are right, what. is being measured by that one sensor is 1 dimensional. But all single sensors measure across a 1 dimensional variable. Whether it’s linear pressure, rotation, light intensity, audio volume at 1 frequency, etc.
- echelon 1y agoThe GPCRs [1] that do most of our sense signalling are each individually complicated machines. Many of our signals are "on" and are instead suppressed by detection. Ligand binding, suppression, the signalling cascade, all sorts of encoding, ... In any case, when all of our senses are integrated, we have rich n-dimensional input. - stereo vision for depth - monocular vision optics cues (shading, parallax, etc.) - proprioception - vestibular sensing - binaural hearing - time I would not say that we sense in three dimensions. It's much more. [1] https://en.m.wikipedia.org/wiki/G_protein-coupled_receptor https://en.m.wikipedia.org/wiki/G_protein-coupled_receptor
- imtringued 1y ago2D models don't have object persistence, because they store information in the viewport. Back when OpenAI released their Sora teasers, they had some scenes where they did a 360° rotation and it produced a completely different backdrop.
- hambes 1y agoyou're telling me my depth perception is not creating a 3D model of the world in my brain?
- kylebenzle 1y ago[dead]
- rubzah 1y agoIt's 2D if you only have one eye.
- __alexs 1y agoIt's not even 2D with one eye. We can estimate distance purely from your eyes focal point.
- yeoyeo42 1y agowith one eye you have temporal parallax, depth cues (ordering of objects in your vision), lighting cues, relative size of objects (things further away are smaller) together with your learned comparison size etc.
- supermatt 1y agoNope. There are a number of monocular depth cues: https://en.wikipedia.org/wiki/Depth_perception#Monocular_cues https://en.wikipedia.org/wiki/Depth_perception#Monocular_cue...
- glitchc 1y agoIt's simple: Those who think that human perception is 2D are wrong.
- KaiserPro 1y agoSo a lot of text to "world" engines have been basically 2d, in that they create a static background and add sprites in to create the illusion of 3D. I'm not entirely convinced that this isn't one of those, or if its not it sure as shit was trained on one.
- reactordev 1y agoYou have two eyes for a reason. The world is not 2D.
- b3lvedere 1y agoThe world is also not just human perception. https://theoatmeal.com/comics/mantis_shrimp https://theoatmeal.com/comics/mantis_shrimp
- reactordev 1y agoThe fucking Mike Tyson’s of the seas.
- SirHackalot 1y ago> Minimum: The minimum GPU memory required is 60GB for 540p. Cool, I guess… If you have tens of thousands of $ to drop on a GPU for output that’s definitely not usable in any 3D project out-of-the-box.
- y-curious 1y agoI mean, still awesome that it's OSS. Can probably just rent GPU time online for this
- HPsquared 1y agoI assume it can be split between multiple GPUs, like LLMs can. Or hire an H100 for like $3/hr.
- kittoes 1y agohttps://www.amd.com/en/products/accelerators/radeon-pro/amd-radeon-pro-v620.html https://www.amd.com/en/products/accelerators/radeon-pro/amd-... Is more approachable than one might think, as you can currently find two of these for less than 1,000 USD.
- esafak 1y agoHow much performance penalty is there for doubling up? What about 4x?
- kittoes 1y agoI just found out about these last week and haven't received the hardware yet, so I can't give you real numbers. That said, one can probably expect at least a 10-30% penalty when the cards need to communicate with one another. Other workloads that don't require constant communication between cards can actually expect a performance boost. Your mileage will vary.
- iamsaitam 1y agoInteresting that they chose the color red in the comparison table to determine the best score of that entry.
- FartyMcFarter 1y agoJust like the stock market in China. Red means the price is going up, green means it's going down.
- jsheard 1y agoThat's also why the stonks-going-up emoji traditionally has a red line, Japan shares that convention. https://blog.emojipedia.org/why-does-the-chart-increasing-emoji-show-in-red/ https://blog.emojipedia.org/why-does-the-chart-increasing-em...
- dlisboa 1y agoBy the way, people might think this has to do with communism but it’s cultural and way before the 20th century. Red is associated with happiness and celebration.
- MengerSponge 1y agoAlmost like the communists chose what iconography to use!
- mananaysiempre 1y agoThe (blood-)red flag as an anti-monarchist symbol originates in the French Revolution, was adopted by the Bolshevik faction (“the Reds”) in the Russian Civil War, and spread from there.
- kridsdale1 1y agoAnd ironically the news networks in 2000 chose red to show Bush’s electoral votes vs Gore, and thus we retain the notion of Red States and Blue States, even though it’s backwards.
- ambitiousslab 1y agoThis is not open source. It is weights-available. Also, there is no training data, which would be the "preferred form" of modification. From their license: [1] If, on the Tencent HunyuanWorld-Voyager version release date, the monthly active users of all products or services made available by or for Licensee is greater than 1 million monthly active users in the preceding calendar month, You must request a license from Tencent, which Tencent may grant to You in its sole discretion, and You are not authorized to exercise any of the rights under this Agreement unless or until Tencent otherwise expressly grants You such rights. You must not use the Tencent HunyuanWorld-Voyager Works or any Output or results of the Tencent HunyuanWorld-Voyager Works to improve any other AI model (other than Tencent HunyuanWorld-Voyager or Model Derivatives thereof). As well as an acceptable use policy: Tencent endeavors to promote safe and fair use of its tools and features, including Tencent HunyuanWorld-Voyager. You agree not to use Tencent HunyuanWorld-Voyager or Model Derivatives: 1. Outside the Territory; 2. In any way that violates any applicable national, federal, state, local, international or any other law or regulation; 3. To harm Yourself or others; 4. To repurpose or distribute output from Tencent HunyuanWorld-Voyager or any Model Derivatives to harm Yourself or others; 5. To override or circumvent the safety guardrails and safeguards We have put in place; 6. For the purpose of exploiting, harming or attempting to exploit or harm minors in any way; 7. To generate or disseminate verifiably false information and/or content with the purpose of harming others or influencing elections; 8. To generate or facilitate false online engagement, including fake reviews and other means of fake online engagement; 9. To intentionally defame, disparage or otherwise harass others; 10. To generate and/or disseminate malware (including ransomware) or any other content to be used for the purpose of harming electronic systems; 11. To generate or disseminate personal identifiable information with the purpose of harming others; 12. To generate or disseminate information (including images, code, posts, articles), and place the information in any public context (including –through the use of bot generated tweets), without expressly and conspicuously identifying that the information and/or content is machine generated; 13. To impersonate another individual without consent, authorization, or legal right; 14. To make high-stakes automated decisions in domains that affect an individual’s safety, rights or wellbeing (e.g., law enforcement, migration, medicine/health, management of critical infrastructure, safety components of products, essential services, credit, employment, housing, education, social scoring, or insurance); 15. In a manner that violates or disrespects the social ethics and moral standards of other countries or regions; 16. To perform, facilitate, threaten, incite, plan, promote or encourage violent extremism or terrorism; 17. For any use intended to discriminate against or harm individuals or groups based on protected characteristics or categories, online or offline social behavior or known or predicted personal or personality characteristics; 18. To intentionally exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm; 19. For military purposes; 20. To engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or other professional practices. [1] https://github.com/Tencent-Hunyuan/HunyuanWorld-Voyager/blob/main/LICENSE https://github.com/Tencent-Hunyuan/HunyuanWorld-Voyager/blob...
- Ragnarork 1y agoThe license used for this is quite a read. Available to the world except the European Union, the UK, and South Korea Not sure what led to that choice. I'd have expected either the U.S. & Canada to be in there, or not these. 3. DISTRIBUTION. [...] c. You are encouraged to: (i) publish at least one technology introduction blogpost or one public statement expressing Your experience of using the Tencent HunyuanWorld-Voyager Works; and (ii) mark the products or services developed by using the Tencent HunyuanWorld-Voyager Works to indicate that the product/service is “Powered by Tencent Hunyuan”; [...] What's that doing in the license? What's the implications of a license-listed "encouragement"?
- wkat4242 1y agoI wonder if you can still download and use it here in the EU.. I don't care about licensing legalese, but I guess you have to sign up somewhere to get the goods?
- notpushkin 1y agoIt’s on HF: https://huggingface.co/tencent/HunyuanWorld-Voyager https://huggingface.co/tencent/HunyuanWorld-Voyager
- wkat4242 1y agoThanks!! I saw there wasn't all that much on github so I missed that part.
- mushufasa 1y agoThe EU and others listed are actively trying to regulate AI. Permissive OSS libraries' "one job" is to disclaim liability. This is interesting that they are just prohibiting usage altogether in jurisdictions where the definition of liability is uncertain & worrying to the authors.
- amelius 1y agoThat would be an extremely lazy way of writing a license.
- bilsbie 1y agoI’m waiting like crazy for one of these to show up on vr.
- jsheard 1y agoPlease don't hold your breath, they're still pretty far from high-res 120fps with consistent stereo and milliseconds of latency.
- geokon 1y agoIsn't it picture to 3D model? You'd generate the environment/model ahead of time and then "dive in" to the photo
- jsheard 1y agoI suppose that's an option yeah, but when people envision turning this kind of thing into a VR holodeck I think they're expecting unbounded exploration and interactivity, which precludes pre-baking everything. Flattening the scene into a diorama kind of defeats the point.
- andoando 1y agoYou just need to prerender the 3d world. If its truly exportable as a 3D model, rerendering it real time based on input is trivial
- throwmeaway222 1y agoI actually would rather it be a 3d model so that I don't need to believe they're microwaving a goddamn full size whale for 45 minutes (worth of electricity)
- jimmySixDOF 1y agoWhile discussing Google Genie v3 and AndroidXR, Bilawal Sidhu said : "to create an even faster, lower latency pipeline to go from like 24 fps to like 100 fps. I could see that being more of an engineering problem than a research one at this point." https://youtu.be/VslvofY16I0&t=886 https://youtu.be/VslvofY16I0&t=886
- user_7832 1y agoI see a lot of skeptical folks here... isn't this the first such model? I remember seeing a lot of image to 3d models before, but they'd all produce absurd results in a few moments. This seems to produce really good output in comparison.
- explorigin 1y agoIf you click on the link, they show a comparison chart with other similar models.
- neuronic 1y ago> isn't this the first such model? The linked Github page has a comparison with other world models...
- geokon 1y agoSeems the kind of thing StreetView data would have been perfect to train on. I wonder if you could loop back the last frame of each video to extend the generated world further. Creating a kind of AI fever dream
- kridsdale1 1y agoWhy the past tense? Google is holding on to all of that, going back years.
- Cthulhu_ 1y agoYeah, they have all the raw data (Google is a self-confessed data hoarder, after all), I'm sure they have research projects where they use AI and similar to stitch street view images together. I also wouldn't be surprised if their Street View cars / people record video instead of stills these days. Assuming they started capturing stuff in 2007 (and it was probably a lot earlier), storage technology has improved at least tenfold in terms of storage (probably more), video processing too.
- NullCascade 1y agoWhat is currently the best model (or multi-model process) to go from text-to-3D-asset? Ideally based on FOSS models.
- neutronicus 1y agoPiggybacking ... what about text-to-sprite-sheet? Or even text-and-single-source-image-to-sprite-sheet?
- nzach 1y agoI've never done this task specifically, but I imagine the new google model (Gemini 2.5 Flash Image) is what you want. It has really good character consistency, so you should be able to paste a single sprite and ask it to generate the rest.
- SXX 1y agoThis is possible, but mostly for generating assets in the somewhat same style that you already have. Problem is that AI models are not good at tracking state of multiple entities on one image including 2.5 Flash Image. If you actually want something consistent you should really generate images one by one and provide extensive description of what you expect to see on each frame And if you want to make something like animation it's only really possible if you basically generate thousand of "garbage" images and then edit together what fits.
- stargrazer 1y agoIt explicitly says using a single picture. Wouldn't the world become even more expressive if multiple pictures could be added, such as in a photogrammetry scenario?
- btbuildem 1y agoI had the same question! I will have to try this, I have a super edge use case: incomplete bathymetric depth map (lidar boat could not access some areas), coincidentally the most interesting areas are not in the data. My second piece of data is from flyover video (areas of interest where water is also clear enough to see the bottom). With enough video I can mostly remove the water-borne artifacts (ripples, reflections etc) and enhance the river bottom imagery enough to attempt photogrammetric reconstruction. The bottleneck here is that it takes multiple angles to do that, and the visibility through water is highly dependent on the angle of sunlight vs angle of camera. Instead of doing multiple flyovers at different times of day to try and get enough angles for a mesh reconstruction, maybe this can do it relatively well from one angle!
- Miraste 1y agoThat sounds quite interesting. Why are you trying to reconstruct a river bottom?
- loudmax 1y agoThis does sound interesting, but is generative AI the right tool for this use case? A generative AI model sounds great for making a video game or even exploring historical photos, where introducing invented artifacts is a feature not a bug. In your case, wouldn't hallucinations be a problem?
- amelius 1y agoCan I use this to replace a LiDAR?
- garbthetill 1y agoif it does, then elon really won the bet of no lidar
- odie5533 1y agoAll he had to do was remove the LIDAR and wait 15-20 years for the tech to catch up. I'm sure Tesla owners don't mind waiting. They're used to it by now.
- HeWhoLurksLate 1y agothere's a huge difference between "feature that mostly works and is kinda neat" and "5000 pound robot relies on this to work all the time or people will probably get hurt at minimum" in how much you should trust a feature. Doesn't really matter if an imgTo3d script gets a face's depth map inverted, kinda problematic if your car doesn't think there's something where there is.
- Cthulhu_ 1y agoI wasn't aware there was a competition or a bet.
- incone123 1y agoIt's generating a 3d world from a photo or other image, rather than giving you a 3d model of the real world.
- krystofee 1y agoI think its a matter of time when we will have photorealistic playable computer games generated by these engines.
- gadders 1y agoAnd hopefully AI-powered NPCs to fight against/interact with.
- Cthulhu_ 1y agoI believe there's games that have that already. My concern is that it's all going to be sameish slop. Read ten AI generated stories and you've read them all. It could work, but they would have to both write unique prompts for each NPC (instead of "generate me 100 NPC personality prompts") and limit the possible interactions and behaviours. But, emergent / generative behaviour would be interesting to a point. There's plenty of roguelikes / roguelites where this could work in, given their generative behaviours.
- gadders 1y agoI guess for combat, you would want ones that could sensibly work together and adapt, possibly different levels of aggression, stealth etc Even as good as FEAR would be something.
- Keyframe 1y agoThere's a reason Tencent is doing this https://en.wikipedia.org/wiki/Tencent#Foreign_studio_assets https://en.wikipedia.org/wiki/Tencent#Foreign_studio_assets
- netsharc 1y agoYeah, MS Flight Simulator with a world that's "inspired by" ours... The original 2020 version had issues with things like the Sydney Harbour Bridge (did it have the Opera House?), using AI to generate 3D models of these things based on pictures would be crazy (of course they'd generate once, on 1st request). So if you're the first to approach the Opera House, it would ask the engine for 3D models of the area, and it would query its image database, see the fancy opera house, and generate its own interpretation.. if there's no data (e.g. a landscape in the middle of Africa), it'd use the satellite image plus typical fauna of the region..
- forrestthewoods 1y agoSpin the camera 1080 degrees in place you cowards!! These clips are very short and don’t rotate the camera more than like 45 degrees. Genie3 also cheats and only rotate the camera 90 degrees. It’s always important to pay attention to what models don’t do. And in this case it’s turn the bloody camera around. I refuse to accept any model to be a “world model” if it can’t pass a simple “spin in place” test. Bah hum bug.
- indiantinker 1y agoMatrix
- londons_explore 1y ago> The minimum GPU memory required is 60GB for 540p. We're about to see next gen games requiring these as minimum system requirements...
- pbd 1y agoThis is genuinely exciting.
- bglazer 1y agoPlease don’t post chatgpt output
- chamomeal 1y agoSo could it actually turn around, like a full 360, and the image would stay the same? It looks super cool but the videos I saw just pan a little one way or the other
- tzumaoli 1y agoIt could in theory. The model generates a depth image per frame, so each pixel becomes a small 3D point. It also assumes that the 3D scene is static. From this, you can then simply register all the frames into a huge 3D point cloud by unprojecting the pixels to 3D and render it anyway you like (using a classical 3D renderer) and it will be consistent. Though, a problem is that if the generated video itself has inconsistent information, e.g., the object changes color between frames, then your point cloud would just be "consistently wrong". In practice this will lead to some blurry artifacts because you blend different inconsistent colors together. So when you turn around you will still see the same thing, but that thing is uglier and blurrier because it blends between inconsistent coloring. It will also be difficult to put a virtual object into the generated scene, because you don't have the lighting information and the virtual object can't blend its color with the environment well. Overall cool idea but obviously more interesting problems to be solved!
- forinti 1y agoIn 1995 I went to a talk on Image Processing by an Indian professor. I asked him if there were any methods for improving low resolution images, just to make them look better (I think this was in the context of TV transmissions). He said you couldn't make up information. Well, 30 years later, you can generate a video from a photograph.
- bufferoverflow 1y ago[dead]
- Terr_ 1y agoWhile there's been a lot of technological progress, I think that story confuses different meanings of "could" and "information". From a photo of someone's face and shoulders, a child can add "information" by extending it to a stick-figure body with crayons. However it's not information from the original event that was recorded. Then there's the difference between strictly capable versus permissible or wise. A researcher "can't" make up data, a journalist "can't" invent quotes, a US President "can't" declare himself dictator, etc.
- deleted 1y ago[deleted]
- IanCal 1y agoAlso you can get a lot more information from images than you think, and even more from video. Superresolution was the term iirc. You can’t make up information but you can use knowledge of the subject to accurately fill things in and other assumptions to plausibly fill things in.
- maelito 1y agoWhat I'm interested with is taking Panoramax pictures (a free StreetView alternative) and recreate 3D navigable scenes from them.