9 ms·
World Labs: Generate 3D worlds from a single image
- byteknight 2y agoI feel like a lot of the good that projects like this do get muddied by the overly ambitious claims.
- echelon 2y agoThe overly ambitious claims are what led them to raise $230M+ without a product. Fei-Fei Li is a luminary in the field, and she's assembled a stellar team of some of the best researchers in the space. Their gamble is that they'll be able to move faster than the open research and other companies looking to productionize that research. Time will tell if this can become an ElevenLabs or if it'll fizzle out like Character.ai. My worry is that without a product, they'll malinvestment their research into cool problems that don't satisfy market demand. There's nothing like the north star of customers. They'll also have a tough time with hiring going forward with that valuation. The market of open research and models is producing a lot of neat stuff in 3D. But there's no open pool of data yet, despite HuggingFace and others trying. We'll see what happens.
- mclau156 2y agoI believe Fei-Fei is focused on physical world interaction, https://behavior.stanford.edu/ https://behavior.stanford.edu/ is a project she works on for more physical interaction with AI
- bko 2y agoI thought you were kidding but World Labs really did raise $230M after being founded less than 1 year ago. Andreessen Horowitz too. What would have to be true for this to be worth $100 billion after 5-10 years? https://www.crunchbase.com/organization/world-labs https://www.crunchbase.com/organization/world-labs
- whiplash451 2y agoDoesn't have to be 100 billions. They probably raised at a few billion valuation -- which I do agree is still a lot
- xnx 2y agoI don't know enough about venture funding. Did $230M really get transferred into World Labs bank account, or is this a "commitment" of $230M which is trickled out a few million at a time?
- whiplash451 2y agoPart of it is often dedicated to compute through credit commitments at Azure/Google/AWS. Probably not all is cash and available at t0. That money will go fast though, given GPU costs and salary ranges in the bay
- corysama 2y agoThe general theme is... Before: You and your cofounders share 100% of the stock in your company that's valued at $X. After: Your company now has everything it had before plus $Y worth of "something" from the VCs. Your company is now valued at $X plus $Y. The VCs now hold stock in your company worth $Y. You and your cofounders still hold stock your company worth $X. "Something" might be anything. Cash, stocks, commitments to resources, whatever.
- Minor49er 2y agoApparently a "world" is about the size of your average walk-in closet these days
- dmarcos 2y agoDon’t know the exact details but I imagine the further from the original input image the more the system needs to make up stuff. Same why generative video models are limited to a few seconds. It will improve
- anticorporate 2y agoCan you point at some data that would indicate it will improve? There are lots of statements today about GenAI akin to "that will get fixed later" but we don't actually seem to know what will actually improve and what will just get incrementally prettier without fixing the underlying issue.
- dmarcos 2y agoImage generation has improved a lot in just 2 years: no more 7 fingers hands, text rendering, general image quality… We’re just getting started with 3D and incentives for it to improve are strong
- oneshtein 2y agoAI can outpaint more images in similar style, then map them to 3D. IMHO, AI should generate a story from the image, then use image + story + location & direction to generate consistent 3D world.
- deleted 2y ago[deleted]
- marcsj 2y agoThe boundaries make it pretty obvious how flat this world still is and even on blurring the edges it's obvious there really isn't anything to the models. This is cool for sure, and I can see it being useful for better photogrammetry and assisting in building out worlds, but it isn't going to suddenly be used to make entire game worlds on its own.
- mattlondon 2y agoYeah I thought the same and was immediately disappointed that you could only step a tiny bit forwards. BUT, you can turn around and see something that I presume was entirely generated. So I don't think it is just doing some clever tricks to make the photo look 3D, but also "infilling" what is behind the camera too. That is kinda cool. I'd love to see this improved so I can walk around some more though, to see what is down those alleys etc.
- dmarcos 2y agoI’m sure It’ll improve. I imagine, the further from the input image the more the model has to make up stuff. Gen-AI video models are limited to a few seconds. In 3D you’re constrained to a volume
- dyauspitr 2y agoIs this just a perspective trick or are 3D models generated?
- dmarcos 2y agoIn some angles you can appreciate artifacts that resemble those of gaussian splats. I’d bet is a 3D representation but not your traditional mesh. Very cool
- echoangle 2y agoIs that important? If the perspective trick works, you can probably do some variation of photogrammetry and get a 3D model
- chrsw 2y agoI don't see any mention of being able to export 3d models from this.
- dag11 2y agoIt's generating gaussian splats, so not quite 3D worlds. In short, they're pseudo 3D in that there is in fact a 3D point cloud, but the points are elliptical angular-dependent colored splats that get projected into 2D space[1]. They're better for reconstructing source-realistic renderings from a constrained view box, but break down outside of that. They're a really cool way to capture spatial memories though! Friends and I occasionally use Polycam or Luma Labs apps. But there's not _too much_ you can do with them due to the above limitations. From a brief look at the OP link, World Labs seems to be generating a 360º gaussian splat (for a limited view box) from a still photo, which is cool as hell! But we still have the same problem of "what do we do with gaussian splats". [1] This description is hand-wavey as I'm a relative layman when it comes to how these work. I'm sure someone can reply with a more precise answer if this one is bad.
- kfarr 2y agoNot my project, but another approach recently published used Depth Anywhere to create a virtual depthmap for a given 360º equirectangular image and then apply to point cloud and render using three.js / A-Frame. Appears to be similar capability as OP for creating scene depth from 2D, but using point cloud instead of gaussian splatting for rendering so looks more pixelated: https://github.com/akbartus/360-Depth-in-WebXR https://github.com/akbartus/360-Depth-in-WebXR Also unlike the World Lab example you have the ability to go further outside the bounds of the point cloud to inspect the deficiencies of the approach. It's getting there but still needs work.
- dmarcos 2y agoYeah A-Frame! It makes me happy to see my many years of maintaining it paying off
- kfarr 2y agoYes and this is a great example of how open A-Frame is compared to OP example. You can inspect every part of the experience from the code to the actual runtime inspector to see how Akbartus achieved the effect -- and then help to make it even better! :) I do think there is the possibility to use something like this eventually to do all the processing in the browser for Depth Anywhere + Splat reconstruction to fill in the holes of the current point cloud approach: https://github.com/ArthurBrussee/brush https://github.com/ArthurBrussee/brush
- raywu 2y agoAs a consumer of A-Frame, I thank you!
- dmarcos 2y agoThank you!
- lastdong 2y agoFirst reaction after trying it was a bit of a surprise when I got an “Out of bounds” message – not what I expected for 3D worlds. Scrolling down to the "Looking Ahead" section, they are working on improving both size and fidelity.
- latexr 2y agoOnce you try the demos, the animated image at the top feels misleading. Each segment cuts at just the right point to make you think you’d be able continue exploring these vast worlds, but in practice you can only walk a couple os steps before hitting an invisible wall, which becomes more frustrating than not being able to move at all. It feels like being trapped in a box. My reaction went from impressed to disappointed fast. I get these are early steps, but they oversold it.
- 6451937099 2y ago[dead]
- dmarcos 2y agoIn the looking ahead section of the post it says: “We are hard at work improving the size and fidelity of our generated worlds” I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation.
- latexr 2y ago> I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation. Which is the same thing as saying this may turn out to be a dud, like so many other things in tech and the current crop of what we’re calling AI. Like I said, I get this is an early demo, but don’t oversell it. They could’ve started by being honest and clarifying they’re generating scenes (or whatever you want to call them, but they’re definitely not “worlds”), letting you play a bit, then explain the potential and progress. As it is, it just sounds like they want to immediately wow people with a fantasy and it detracts from what they do have.
- dmarcos 2y agoFair criticism. I’m also not a fan of hyperbole. Still find World Labs stuff super intriguing and I’m optimist about them to be able to fulfill the vision.
- billconan 2y agowhat would be the business model?
- tantalor 2y ago[flagged]
- deleted 2y ago[deleted]
- swframe2 2y agoTo build an LLM that can reason about the 3d world. I suspect they will add the ability to reason about the physics of the world next. It's just another attempt to get closer to AGI. They most likely will have to pivot a few times but once they show their LLM solving problems that others can't, the others will quickly add these features too. Right now, it is cheaper to wait for World Labs to go first. The others are not that far behind: https://cat-4d.github.io/ https://cat-4d.github.io/
- Uehreka 2y agoI’ve been trying to get into this sort of 3D Gaussian Splatting stuff, particularly with this focus on environments as opposed to just individual objects or characters. Does anyone know of a model that’s good at doing that and is openly distributed/locally runnable?
- evan_ 2y agoWhen watching 3D movies with a VR headset you have to keep your head perfectly still or the lack of parallax destroys the 3D illusion. Compare to a 3D game where moving your head actually lets you move through space and actually look around objects. Something like this applied to every frame of the movie would allow you to move around a little and preserve the perspective shifts. The limitation that you can only move about 4 feet in any direction would not matter for this use case. Of course this comes at the expense of the director and cinematographer's intention, which is no small thing.
- dmarcos 2y agoDefinitely if there’s a future for 3D and immersive video it depends on adding more cues other than just stereo. Lack of parallax one of main reasons causing discomfort for many.
- ChicagoBoy11 2y agoHave you ever seen the Google Lightfields demo? They have a rig they concocted to essentially capture a "volume" of video to allow for the stereoscopic effect in VR AND which then cleverly presents a different combination of the footage it captured based on your precise head position, so it makes up for these distortions. I found it absolutely breathtaking... first time seeing VR for a space that actually made me feel like I was in it. This was A LONG time ago and I suspected I'd be seeing a lot more of that content, but I was... very wrong, it seems. Your point is completely correct. Even Apple's awesome new stereoscopic 3D short film for the AVP immediately loses what it could be its total awesomeness from this basic fact. The perspective being perfectly fixed will never quite be there to fool our brains so used to dealing with these micro-movements.
- dmarcos 2y agoYeah parallax, reflexions, shadows are as important as stereo. We’ve been always sold that stereo = 3D but it’s just one among many cues that the brain relies on.
- Stevvo 2y agoEach frame was between 200 and 300mb, at a much lower resolution than AVP. The storage and bandwidth required is a bit wild.
- julianeon 2y agoI was interested to see that a co-founder is Stanford CS prof Fei-Fei Li. I'm reading her nonfiction book now, "The Worlds I See," about her experience with AI; she testified before Congress about it.
- doctorpangloss 2y agoTheir bet is that XYZ can generalize from Unreal and NVIDIA Isaac recordings. Is XYZ diffusion-transformers? Or is XYZ Chameleon? Or some novel architecture? It takes the absolute fastest teams, it seems, 7 months to develop a first version of a model. And it also seems that models are like babies, 9 moms do not produce a model in 1 month. The tough thing is that it may be possible to develop a great video model with DiTs for $220m; or it may be possible to develop a great video model with Chameleon for $1b; but if it's 3D + time, will it be too expensive for them to do? The craziest thing to me is that these guys are super talented, but they might not have enough money!
- byearthithatius 2y agoThen they need to sell _billions_ of dollars worth of these worlds to ... game studios? In order for this valuation to make sense they need to convince the majority of major game studios to spend all their world creation budget solely on this company. Seems unrealistic but I guess only time can tell.
- dmarcos 2y agoif they fulfill the mission it will apply to domains other than games like movies, robotics, architecture...
- byearthithatius 2y agoGood point, that's fair there are more use cases. IDK about architecture, typically you want more structure/determinism instead of probabilistic generation. Overall this is very cool. I like the consistency it has and it does generally amaze me nonetheless. But you gotta admit selling a billion dollars of anything is really hard. That is three times the budget of the highest budget movie ever created! (avengers endgame at 356 million). It is almost the ENTIRE budget of the biggest game ever, grand theft auto six.
- idunnoman1222 2y agoRobotics?
- recursive 2y agoI couldn't get the "tap to interact" panels to work. No mouse events had any effect. I had to take it very literally, but first, I had to drag my browser to my laptop screen, which did enable me to literally tap the screen.
- jcjohns 2y agoThat's weird, what device are you using? (I'm part of World Labs)
- recursive 2y agoFirefox on Windows 11 on a Lenovo Thinkpad with a touch screen.
- FergusArgyll 2y agoNot working for me either, Chrome win 11
- xnx 2y agoCool, but not as impressive as https://cat-4d.github.io/ https://cat-4d.github.io/ to me.
- wordpad25 2y agoDoes this also work for macro shots, like a landscape? all the examples are focused on specific objects
- robblbobbl 2y agoThe idea is good but the result must be better
- dmarcos 2y agoWhat’s your use case?
- tnolet 2y agoThis is more like the moving "still" pictures in a Harry Potter movie. Not a 3D world.
- dmarcos 2y agoIn some angles you can see there’s some gaussian splat / point cloud representation underneath. There’s definitely a 3D representation. But yeah navigable volume is limited at the moment. It will improve
- vinkelhake 2y agoThis is neat I guess. Maybe I'm just blase with seeing yet another AI demo where I'm supposed to fill in the blanks in coming up with ways to make the tech actually useful. The "Step into Paintings" section cracked me up. As soon as you pan away from the source material, the craziness of the model is on full display. So sure, I can experience iconic pieces of art in a new way, it's just not a good experience.
- jsheard 2y agoWho knew that Hopper's Nighthawks had a biblically-accurate table and chairs just out of frame?
- lacoolj 2y agoso there's a bunch of potential here, but how long did each of these take to generate from the model and what hardware was used for it?
- thrance 2y agoYet another of those AI image-to-grotesque-interactive-video model marketed as a "3D world from scratch!". Can you use this "3D world" with blender, unity or whatever else? Can you even do anything remotely useful with it?
- dmarcos 2y agoYou can definitely mix gaussian splats and “traditional” meshes Splats very new and still many things to figure out: relighting, animation, interactivity.
- ValentinA23 2y agowasd isn't accessible for those of us who have the unfortunate disability of not using a qwerty keyboard. If your project isn't a competitve FPS, arrows are fine.
- aleph_minus_one 2y agoYou can install multiple keyboard layouts in your OS. Many users do this.
- jcjohns 2y agoArrow keys also work now, thanks for the feedback!
- PeterCorless 2y agoUgh. The AI-generated rear views being nowhere near at the level of detail of even the uncanny valley foreground images. Is not really generating a "3D 'world'" so much as extrapolating a 360º view from a single scene. There's no sense to the architecture or flora. Staircases that lead nowhere. It's more hypecycle nonsense. But they'll poor billions into this rather than pay human artists what they're worth.
- Falimonda 2y agoToo many demos loading on the site at the same time makes it unusable
- deleted 2y ago[deleted]
- wkat4242 2y agoThis is amazing. 3D content generation is so time consuming..
- Vanit 2y agoI'm keen to drop in a few PSX-era Final Fantasy backgrounds to see what it does!
- deleted 2y ago[deleted]
- bastloing 2y agoWhat a great start, it's only going to get better from here!
- iamleppert 2y agoIs a 2D image really the best input primitive for 3D world construction? As a user, I'd prefer to have 3D primitives (plane, sphere, mesh) as tools when building my worlds.
- cchance 2y agoPeople complaining that it's a small area, lol my man, this is fucking insane, i know AI is starting to get normalized, but they converted an image into a 3d world! even if its 1ft/1ft its still amazing.
- Double_a_92 2y agoThe issue is that they are overselling it. It's not a world, it's a 3D image at best.
- blensor 2y agoWe recently added an environment creator for our VR game which does kinda a similar thing but for even less freedom of movement, so I think I have a little bit of insight into it. After seeing what they are showing on their page I am majorly impressed and don't feel they are misleading anyone at all
- dmarcos 2y agoNice! Is the environment creator available as standalone? What format do you output?
- blensor 2y agoThe prompting is done from the browser and it's then automatically added in the game so the next time you do your workout it is already there to select. https://www.youtube.com/watch?v=7SVD_tLGAJk https://www.youtube.com/watch?v=7SVD_tLGAJk The environment is 2.5D so mostly a displaced depth map but with some additional logic to keep the ground and horizon sane. We aren't planning to do it for external use but it is creating a glb that you can download in theory if you know where it points to on our server
- dmarcos 2y agoVery nice, thanks for sharing. Are you using any open source model? I maintain and open source framework called A-Frame for Web based 3D, VR, AR and the community would love something like that Keep up the good work
- albtaiuti 2y agoit looks like they're basing the infilling on 360 photos / videos. that's why you can't walk around freely: the inpainting must be done from the center of the sphere
- amelius 2y agoDid anyone try it on famous paintings?
- Thaxll 2y agoIt looks like the Facebook feature that transforms your image in "3d".
- fullstackwife 2y agobaseline images seem to be rendered, because there is shading, lightining, shadows etc. when I tried other tool (image -> 3d model) it tended to work only for their example images, and when I used anything else it produced some black and flat shape. so the headline should be: Generate 3D worlds from a single image rendered by us that we used to train our model.
- KeplerBoy 2y agoIt clearly says the reference images are ai generated.
- frisco 2y agoSomeone please make a player for these things for Vision Pro and/or Quest!
- dmarcos 2y agoWe have a gaussian splat component (same format that it looks World lab uses) for A-Frame that should work in VR on Quest and Vision Pro in their respective Web browsers. Quest FPS might not be ideal yet https://x.com/dmarcos/status/1714364349928837147 https://x.com/dmarcos/status/1714364349928837147
- jcims 2y agoThe width of modern mobile phones isn't far off from the average pupillary distance of adults. It feels like there is an opportunity to create a glut of useful 3D data by simply placing one camera on each side of the phone rather than trying to infer it from a single view of the scene. Maybe when there is better technology for viewing 3D content.
- RecycledEle 2y agoI'm tempted to use this with some sci-fi and fantasy book covers.
- krenzo 2y agoFor me, the camera is just looking up and spinning clockwise. I tried in both Brave and Chrome.
- throwaway314155 2y agoBrave and Chrome are both Chrome. Have you tried Firefox, Safari?
- falcor84 2y agoThis is some cool progress, but from what I gather it's not actually generating a "world". What I'd be really interested in is the capability of fully generating the geometrical world description in something like USD (Universal Scene Description).
- edwardmacnamra 2y ago[dead]
- ellis0n 2y agoAmazing! It looks like this is one step closer to singularity and this startup showcases what future startups should aspire to be. Although the technology for world generation is just in its infancy, but it’s impressive and atmosphere is great with a stunning impact. Everything you need to wow an investor and secure funding while the technology itself can be hotfixed for years to come. I think at this stage the showcased technology seems more aligned with cinema than the metaverse. Great work! Looking forward to the updates.
- TinkersW 2y agoI think it does look much nicer than the other examples of this nature that I've seen. I would guess they are pre-generating the world from the image, not generating it on the go as it runs reasonably well, but doesn't this really limit world size? I noticed some solid geometry that is accidentally transparent. The stuff behind the camera looks pretty good, which is presumably fully generated, so if they can make it so you can actually move around more, and with similar quality it could be interesting. I do wish the examples had a full screen button, the view is tiny..
- api 2y agoIn 20 years a fiction writer will be able to upload their work to something and generate a movie. It won't be Hollywood quality. It'll either look like animation or early mixed CGI/live action stuff with "wooden" performances, etc., but it will let them see their work acted out which will be super cool. Obviously the pro version of this with detailed editing and incorporating real human actors for the starring roles will be what's used to make a lot of real film and TV serious content.