4 ms·

WorldGen: Open-source 3D scene generator for Game/VR/XR

138 points by ziyangxie 1y ago

pdntspa 1y ago
The splat outputs remind me of Cyberpunk 2077's braindances
atrus 1y ago
We really need models that produce intermediate files. A .blend, a whole godot project, an actual krita/psd file. Things like this product such cool things....all the way up until you need to move a specific tree half a meter to the left, and then you're either stuck in some prompt/seed hell or have to open an abomination of a file in an editor.
- ziyangxie 1y ago
  WorldGen supports exporting the generated scene as a vertices colored mesh in .ply format, which you can load into any 3D editor (like Blender). The output is also in metric scale, so you can easily align it with other assets
ivape 1y ago
Looks like only nvidia cards. Can anyone guess how much vram you need for this?
- ziyangxie 1y ago
  It is currently 26GB, but I will reduce it to under 24GB so that you can at least run it on a 4090.
  - wordpad 1y ago
    Is it available via API somewhere?
  - Maxious 1y ago
    Have you seen https://github.com/mit-han-lab/nunchaku https://github.com/mit-han-lab/nunchaku for running flux on lower VRAM and thus faster performance? I have put up a PR https://github.com/ZiYang-xie/WorldGen/pull/7 https://github.com/ZiYang-xie/WorldGen/pull/7
ziyangxie 1y ago
Optimized the webpage layout for mobile users.
Ono-Sendai 1y ago
Not really proper 3D, more like a kind of 2.5D.
jimmySixDOF 1y ago
An opensource Blockade Labs Skybox.ai sounds good and thats a lot of tooling in your pipeline so congratulations putting it all together I would 100% clone this on a HuggingFace space !!
socalgal2 1y ago
Neat! But not quite what I was expecting. It doesn't really make a 3D "scene". It makes a 3D panorama (building a mesh based off the depth of the panorama). Move off the center and all the things that were occluded are missing. I guess, because we're in the time of generative content, that I was expecting it would recognize what's in the scene and fill in the missing parts. Maybe in version 2
- albumen 1y ago
  From GitHub page: "We also support background inpainting for better scene generation, but it's currently an experimental feature, which may not work for all scenes. It can be enabled by setting WorldGen(inpaint_bg=True)"
  - bengarney 1y ago
    Is there an example of this? A skybox with depth is only marginally better than a skybox for any sort of 3d experience. Using the depth for occlusion would be kind of cool.
- tamat 1y ago
  which for me doesnt make any sense as why to generate a 3D mesh that can only be seen from one point of view? why not map the panorama to an sphere?
doshaa 1y ago
I really hope there is like an "AI" flair on projects like these, i love what they aim to do but man the saturation levels mixed with "enough to make it work in theory" effort makes me want to filter out these posts. I understand its all vibes and fun, but like at least some warnings that "this thing might lag your computer" and buttons that work.
- ziyangxie 1y ago
  Thanks for your interest! the AI space is moving fast, and there is always something new grabbing attention. I believe it is better to release an MVP early and keep improving it, rather than holding everything back and risking being outdated or scooped. Just like in startups, I believe fast iteration is key for AI projects too.
pedalpete 1y ago
Really nicely done, and I'm excited to see where this goes. I'm the founder of 3d real-world scene generator https://ayvri.com https://ayvri.com - the tech was acquired a while back, but I still have people reaching out to me to ask me to bring it back, and I've told people that the way to do it today would be to generate the scenes with splats and AI.
shihabkhanbd 1y ago
[dead]
firtoz 1y ago
I like that there are so many different approaches to generate worlds. One, few, or eventually all of them will stick. We have been playing around with something similar at Greybox too, where we ask Claude 3.7 to create Lua scripts to define the scenes with primitives (cubes, spheres, etc) that you can then move around. It's not perfect but did better than we expected! https://greybox.app/blog/articles/introducing-greybox-ai-creative-assistant-3d-prototyping https://greybox.app/blog/articles/introducing-greybox-ai-cre... has a video at the top that shows what it looks like at the moment. We saw that Meta, Krea are working on "compose the scene from 3d model generations or imports as individual components", we'll give that a try too, soon.
whywhywhywhy 1y ago
for what it does, its cool but claiming it's useful for games/vr/xr is a bit of a stretch, in it's current state it's only good for fixed position vr (which is a pretty rough experience) as the illusion falls apart when you move your head slightly. For games you could use it for environment maps or maybe a non-interactive background in menus or visual novel style. But kinda think claiming a panoramic generator as a "World Generator" is intentionally hoping people expect it to be more than it delivers.
barberhenna 1y ago
[dead]

138 points by ziyangxie 1y ago

The splat outputs remind me of Cyberpunk 2077's braindances

We really need models that produce intermediate files. A .blend, a whole godot project, an actual krita/psd file. Things like this product such cool things....all the way up until you need to move a specific tree half a meter to the left, and then you're either stuck in some prompt/seed hell or have to open an abomination of a file in an editor.

ziyangxie 1y ago
WorldGen supports exporting the generated scene as a vertices colored mesh in .ply format, which you can load into any 3D editor (like Blender). The output is also in metric scale, so you can easily align it with other assets

ivape 1y ago

Looks like only nvidia cards. Can anyone guess how much vram you need for this?

ziyangxie 1y ago
It is currently 26GB, but I will reduce it to under 24GB so that you can at least run it on a 4090.
- wordpad 1y ago
  Is it available via API somewhere?
- Maxious 1y ago
  Have you seen https://github.com/mit-han-lab/nunchaku https://github.com/mit-han-lab/nunchaku for running flux on lower VRAM and thus faster performance? I have put up a PR https://github.com/ZiYang-xie/WorldGen/pull/7 https://github.com/ZiYang-xie/WorldGen/pull/7

ziyangxie 1y ago

Optimized the webpage layout for mobile users.

Ono-Sendai 1y ago

Not really proper 3D, more like a kind of 2.5D.

jimmySixDOF 1y ago

An opensource Blockade Labs Skybox.ai sounds good and thats a lot of tooling in your pipeline so congratulations putting it all together I would 100% clone this on a HuggingFace space !!

socalgal2 1y ago

Neat! But not quite what I was expecting. It doesn't really make a 3D "scene". It makes a 3D panorama (building a mesh based off the depth of the panorama). Move off the center and all the things that were occluded are missing. I guess, because we're in the time of generative content, that I was expecting it would recognize what's in the scene and fill in the missing parts. Maybe in version 2

albumen 1y ago
From GitHub page: "We also support background inpainting for better scene generation, but it's currently an experimental feature, which may not work for all scenes. It can be enabled by setting WorldGen(inpaint_bg=True)"
- bengarney 1y ago
  Is there an example of this? A skybox with depth is only marginally better than a skybox for any sort of 3d experience. Using the depth for occlusion would be kind of cool.
tamat 1y ago
which for me doesnt make any sense as why to generate a 3D mesh that can only be seen from one point of view? why not map the panorama to an sphere?

doshaa 1y ago

I really hope there is like an "AI" flair on projects like these, i love what they aim to do but man the saturation levels mixed with "enough to make it work in theory" effort makes me want to filter out these posts. I understand its all vibes and fun, but like at least some warnings that "this thing might lag your computer" and buttons that work.

ziyangxie 1y ago
Thanks for your interest! the AI space is moving fast, and there is always something new grabbing attention. I believe it is better to release an MVP early and keep improving it, rather than holding everything back and risking being outdated or scooped. Just like in startups, I believe fast iteration is key for AI projects too.

pedalpete 1y ago

Really nicely done, and I'm excited to see where this goes. I'm the founder of 3d real-world scene generator https://ayvri.com https://ayvri.com - the tech was acquired a while back, but I still have people reaching out to me to ask me to bring it back, and I've told people that the way to do it today would be to generate the scenes with splats and AI.

shihabkhanbd 1y ago

[dead]

firtoz 1y ago

I like that there are so many different approaches to generate worlds. One, few, or eventually all of them will stick. We have been playing around with something similar at Greybox too, where we ask Claude 3.7 to create Lua scripts to define the scenes with primitives (cubes, spheres, etc) that you can then move around. It's not perfect but did better than we expected! https://greybox.app/blog/articles/introducing-greybox-ai-creative-assistant-3d-prototyping https://greybox.app/blog/articles/introducing-greybox-ai-cre... has a video at the top that shows what it looks like at the moment. We saw that Meta, Krea are working on "compose the scene from 3d model generations or imports as individual components", we'll give that a try too, soon.

whywhywhywhy 1y ago

for what it does, its cool but claiming it's useful for games/vr/xr is a bit of a stretch, in it's current state it's only good for fixed position vr (which is a pretty rough experience) as the illusion falls apart when you move your head slightly. For games you could use it for environment maps or maybe a non-interactive background in menus or visual novel style. But kinda think claiming a panoramic generator as a "World Generator" is intentionally hoping people expect it to be more than it delivers.

barberhenna 1y ago

[dead]