5 ms·

Once you try the demos, the animated image at the top feels misleading. Each segment cuts at just the right point to make you think you’d be able continue explo

by latexr 2y ago

Once you try the demos, the animated image at the top feels misleading. Each segment cuts at just the right point to make you think you’d be able continue exploring these vast worlds, but in practice you can only walk a couple os steps before hitting an invisible wall, which becomes more frustrating than not being able to move at all. It feels like being trapped in a box. My reaction went from impressed to disappointed fast. I get these are early steps, but they oversold it.

6451937099 2y ago
[dead]
dmarcos 2y ago
In the looking ahead section of the post it says: “We are hard at work improving the size and fidelity of our generated worlds” I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation.
- latexr 2y ago
  > I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation. Which is the same thing as saying this may turn out to be a dud, like so many other things in tech and the current crop of what we’re calling AI. Like I said, I get this is an early demo, but don’t oversell it. They could’ve started by being honest and clarifying they’re generating scenes (or whatever you want to call them, but they’re definitely not “worlds”), letting you play a bit, then explain the potential and progress. As it is, it just sounds like they want to immediately wow people with a fantasy and it detracts from what they do have.
  - dmarcos 2y ago
    Fair criticism. I’m also not a fan of hyperbole. Still find World Labs stuff super intriguing and I’m optimist about them to be able to fulfill the vision.
  - add-sub-mul-div 2y ago
    Maybe they think it's a good deal, producing some oversold tech demos in exchange for a decade's worth of funding and not having to produce anything more than an "Our Incredible Journey" letter at the end. The prospect of replacing all human labor has made it easier than ever to run the grift on investors in this time of peak FOMO.
- deleted 2y ago
  [deleted]
- modeless 2y ago
  Models are really great at making stuff up though. And video models already have very good consistency over thousands of frames. It seems like larger worlds shouldn't be a huge hurdle. I wonder why they launched without that, as this doesn't seem much better than previous work.
  - dmarcos 2y ago
    To be fair they haven’t launched they are showing progress and laying out the vision. What previous work are you referring to?
- TeMPOraL 2y ago
  In general, it depends on how much the model ends up "understanding" the input. (I use "understand" here in the sense some would claim SOTA LLMs do.) You can imagine this as a spectrum. On the one end you have models that, at each output pixel, try to predict pixels that are locally similar to ones in previous frame; on the other end, you could imagine models that "parse" the initial input image to understand the scene - objects (buildings, doors, people, etc.) and their relationships, and separately, the style with which they're painted, and use that to extrapolate further frames[0]. The latter would obviously fare better, remaining stylistically consistent for longer. (This model claims to be of the second kind.) The way I see it: a human could do it[1], so there's no reason an ML model wouldn't be able to. -- [0] - Brute-force approach: 1) "style-untransfer" the input, i.e. style-transfer to some common style, e.g. photorealistic or sketch, 2) extrapolate the style-untransfered image, and 3) style-transfer result back using original input as style reference. Feels like it should work somewhat okay-ish; wonder if anyone tried that. [1] - And the hard part wouldn't be extrapolating the scene, but rather keeping the style.
  - dmarcos 2y ago
    This indeed looks more like photogrammetry than a diffusion model predicting the next frame. There's 3D information extracted from the input image and likely additional generated poses that allow reconstructing the scene with gaussian splats. Not sure how much segmentation (understanding of each part of the scene) is going on. Probably not much if I have to guess.
Hakkin 2y ago
You can bypass the "Out of bound" message by setting a Javascript breakpoint after `let t = JSON.parse(d[e].config_str)` and then run `Object.values(t.camera.presets).map(o=>o.max_distance=50&&o)` in the console. It breaks down pretty quickly once you get outside the default bounds, as expected, though.
- blendaddict 2y ago
  recorded what it looks like here https://imgur.com/a/OPPHnU6 https://imgur.com/a/OPPHnU6
- dmarcos 2y ago
  Good hack!
- jfactorial 2y ago
  I wonder how much of the remaining work boils down to generating a new scene based on the camera's POV when the player hits one of the bounds, and keeping these generated scenes in a tree structure, joining scenes at boundaries.
  - lukev 2y ago
    Yes, and you wouldn't even need to do it in realtime as a user walks around. Generate incrementally using a pathfinding system for a bot to move around and "create the world" as it goes, as if a Google street view car followed the philosophy of George Berkeley.
  - tayistay 2y ago
    I suspect the problem there is that the multiple paths to a new location will not yield consistent results.
    - kbutler 2y ago
      Yes, infinite exploration, but inconsistent
deleted 2y ago
[deleted]
SubiculumCode 2y ago
It seems that you could take the image of the location near the boundary, then create a new 3d world from that, continually.
- deleted 2y ago
  [deleted]
- jsheard 2y ago
  You could try, but it would quickly devolve into non-euclidean nonsense without global knowledge of the areas it's already generated.
deleted 2y ago
[deleted]
boringg 2y ago
I mean its a marketing hype for their product. Its a pretty good starting step though - assuming they can build on it and expand that world space as opposed to just converting an image to 3D. Certainly has some value to it.. marketing, hiring, fundraising (Assuming its a private company) My take is that its a good start and 3-4 years from now it will have a lot of potential value in world creation if they can make the next steps.
- dmarcos 2y ago
  It's definitely a balancing act. World labs was stealth for a bit. Without a brand, stated mission, examples / demos of what you are capable of... is harder to hire, fund raise or get the attention and mind-share you need once you are ready to ship product. The risk is setting expectations that can't be fulfilled. I'm in the 3D space and I'm optimistic about World Labs.
qwertox 2y ago
I first got irritated a bit by this as well, but then the game Myst came to mind. So I'm willing to accept the limitation, and at this point we know that this can only get better. Next I thought about the likelihood of Nvidia releasing an AI game engine, or more of a renderer, fully AI based. It should be happening within the next 10 years. Imagine creating a game by describing scenes, like the ones in the article, with a good morphing technology between scenes, so that the transitions between them are like auto-generated scenes which are just as playable. The effects shown in the article were very interesting, like the ripple, sonar or wave. The wave made me think about how trippy games could get in the future, more extreme versions of the Subnautica video [0] which was released last month. We could generate video games which would periodically slip into hallucinations, a thing that is barely doable today, akin to shader effects in Far Cry or other games when the player gets poisoned. Fiebertraum engine. [0] https://www.youtube.com/watch?v=AJaV92DXN0s&t=218s https://www.youtube.com/watch?v=AJaV92DXN0s&t=218s
- dmarcos 2y ago
  Yeah. That's the attitude! It's all about playing around the constraints. Tech has limitations? Yes, but also opens tons of new possibilities.
- latexr 2y ago
  You’re describing a pie in the sky. A vision. Not reality. We have been burned many times already, nothing in this field is a given. > at this point we know that this can only get better. We don’t know that. It will probably get better, but will it be better enough? No one knows. > It should be happening within the next 10 years. Every revolution in tech is always ten years away. By now that’s a meme. Saying something is ten years away is about as valuable as saying one has no idea how doable it is. > Imagine Yes, I understand the goal. Everyone does, it’s not complicated. We can all imagine Star Trek technology, we all know where the compass is pointed, that doesn’t make it a given. In fact, the one thing we can say for sure about imagining how everything will be great in ten years is that we routinely fail to predict the bad parts. We don’t live in fantasy land, advancements in tech are routinely used for detrimental reasons.
idunnoman1222 2y ago
Obviously, the generation has to stop at some point and obviously from any key image you could continue generating if you had unlimited GPU, which I’m sorry they didn’t provide for you.
- ajmurmann 2y ago
  I am not sure it's obvious that you could continue generating from any key image and it wouldn't deteriorate into mush. If you take that museum scene and look at the vase-like display piece while walking around it as much as you can it already becomes fuzzy and has the beginnings of weird artifacts growing out of it.
Buttons840 2y ago
I was also disappointed by a still image showing a vast sky, but in motion you see it's just a painting on a short ceiling. The model interpreted the vast sky as a painting on a short ceiling.
fxtentacle 2y ago
As someone completely not involved in this project, I would predict that increasing the scene size while remaining halfway consistent isn't that difficult. Let me elaborate by using cat-4d.github.io, one of their competitors in this field of research: If you look at the "How it works" section you can see that the first step is to take an input video and then create artificial viewpoints of the same action being observed by other cameras. And then in the 2nd step, those viewpoints are merged into one 4D gaussian splatting scene. That 2nd step is pretty similar to 4D NeRF training, BTW, just with a different data format. Now if you need a small scene, you generate a few camera locations that are nearby. But there's nothing stopping you from generating other camera locations or even from using previously generated camera locations and moving the camera again, thereby propagating details that the AI invented outwards. So you could imagine this as you start with something "real" at the center of the map and then you create AI fakes with different camera positions in a circle around the real stuff, and then the next circle around the 1st-gen fakes, and the next circle, and so on. This process is mostly limited by 2 things: The ability of your AI model to capture a global theme. World Labs has demonstrated that they can invent details consistent with a theme in this demos, so I would assume they solved this already. And the other limit is computing time. A world box 2x in each direction is 8x the voxel data and I wouldn't be surprised if you need something like 16x to 32x the number of input images to fit the GSplats/NeRF. So most likely, the box limit is purely because the AI model is slow and execution is expensive and they didn't want to spend 10,000x the resources for making the box 10x larger.

by latexr 2y ago

6451937099 2y ago

[dead]

dmarcos 2y ago

In the looking ahead section of the post it says: “We are hard at work improving the size and fidelity of our generated worlds” I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation.

latexr 2y ago
> I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation. Which is the same thing as saying this may turn out to be a dud, like so many other things in tech and the current crop of what we’re calling AI. Like I said, I get this is an early demo, but don’t oversell it. They could’ve started by being honest and clarifying they’re generating scenes (or whatever you want to call them, but they’re definitely not “worlds”), letting you play a bit, then explain the potential and progress. As it is, it just sounds like they want to immediately wow people with a fantasy and it detracts from what they do have.
- dmarcos 2y ago
  Fair criticism. I’m also not a fan of hyperbole. Still find World Labs stuff super intriguing and I’m optimist about them to be able to fulfill the vision.
- add-sub-mul-div 2y ago
  Maybe they think it's a good deal, producing some oversold tech demos in exchange for a decade's worth of funding and not having to produce anything more than an "Our Incredible Journey" letter at the end. The prospect of replacing all human labor has made it easier than ever to run the grift on investors in this time of peak FOMO.
deleted 2y ago
[deleted]
modeless 2y ago
Models are really great at making stuff up though. And video models already have very good consistency over thousands of frames. It seems like larger worlds shouldn't be a huge hurdle. I wonder why they launched without that, as this doesn't seem much better than previous work.
- dmarcos 2y ago
  To be fair they haven’t launched they are showing progress and laying out the vision. What previous work are you referring to?
TeMPOraL 2y ago
In general, it depends on how much the model ends up "understanding" the input. (I use "understand" here in the sense some would claim SOTA LLMs do.) You can imagine this as a spectrum. On the one end you have models that, at each output pixel, try to predict pixels that are locally similar to ones in previous frame; on the other end, you could imagine models that "parse" the initial input image to understand the scene - objects (buildings, doors, people, etc.) and their relationships, and separately, the style with which they're painted, and use that to extrapolate further frames[0]. The latter would obviously fare better, remaining stylistically consistent for longer. (This model claims to be of the second kind.) The way I see it: a human could do it[1], so there's no reason an ML model wouldn't be able to. -- [0] - Brute-force approach: 1) "style-untransfer" the input, i.e. style-transfer to some common style, e.g. photorealistic or sketch, 2) extrapolate the style-untransfered image, and 3) style-transfer result back using original input as style reference. Feels like it should work somewhat okay-ish; wonder if anyone tried that. [1] - And the hard part wouldn't be extrapolating the scene, but rather keeping the style.
- dmarcos 2y ago
  This indeed looks more like photogrammetry than a diffusion model predicting the next frame. There's 3D information extracted from the input image and likely additional generated poses that allow reconstructing the scene with gaussian splats. Not sure how much segmentation (understanding of each part of the scene) is going on. Probably not much if I have to guess.

Hakkin 2y ago

You can bypass the "Out of bound" message by setting a Javascript breakpoint after `let t = JSON.parse(d[e].config_str)` and then run `Object.values(t.camera.presets).map(o=>o.max_distance=50&&o)` in the console. It breaks down pretty quickly once you get outside the default bounds, as expected, though.

blendaddict 2y ago
recorded what it looks like here https://imgur.com/a/OPPHnU6 https://imgur.com/a/OPPHnU6
dmarcos 2y ago
Good hack!
jfactorial 2y ago
I wonder how much of the remaining work boils down to generating a new scene based on the camera's POV when the player hits one of the bounds, and keeping these generated scenes in a tree structure, joining scenes at boundaries.
- lukev 2y ago
  Yes, and you wouldn't even need to do it in realtime as a user walks around. Generate incrementally using a pathfinding system for a bot to move around and "create the world" as it goes, as if a Google street view car followed the philosophy of George Berkeley.
- tayistay 2y ago
  I suspect the problem there is that the multiple paths to a new location will not yield consistent results.
  - kbutler 2y ago
    Yes, infinite exploration, but inconsistent

deleted 2y ago

[deleted]

SubiculumCode 2y ago

It seems that you could take the image of the location near the boundary, then create a new 3d world from that, continually.

deleted 2y ago
[deleted]
jsheard 2y ago
You could try, but it would quickly devolve into non-euclidean nonsense without global knowledge of the areas it's already generated.

deleted 2y ago

[deleted]

boringg 2y ago

I mean its a marketing hype for their product. Its a pretty good starting step though - assuming they can build on it and expand that world space as opposed to just converting an image to 3D. Certainly has some value to it.. marketing, hiring, fundraising (Assuming its a private company) My take is that its a good start and 3-4 years from now it will have a lot of potential value in world creation if they can make the next steps.

dmarcos 2y ago
It's definitely a balancing act. World labs was stealth for a bit. Without a brand, stated mission, examples / demos of what you are capable of... is harder to hire, fund raise or get the attention and mind-share you need once you are ready to ship product. The risk is setting expectations that can't be fulfilled. I'm in the 3D space and I'm optimistic about World Labs.

qwertox 2y ago

I first got irritated a bit by this as well, but then the game Myst came to mind. So I'm willing to accept the limitation, and at this point we know that this can only get better. Next I thought about the likelihood of Nvidia releasing an AI game engine, or more of a renderer, fully AI based. It should be happening within the next 10 years. Imagine creating a game by describing scenes, like the ones in the article, with a good morphing technology between scenes, so that the transitions between them are like auto-generated scenes which are just as playable. The effects shown in the article were very interesting, like the ripple, sonar or wave. The wave made me think about how trippy games could get in the future, more extreme versions of the Subnautica video [0] which was released last month. We could generate video games which would periodically slip into hallucinations, a thing that is barely doable today, akin to shader effects in Far Cry or other games when the player gets poisoned. Fiebertraum engine. [0] https://www.youtube.com/watch?v=AJaV92DXN0s&t=218s https://www.youtube.com/watch?v=AJaV92DXN0s&t=218s

dmarcos 2y ago
Yeah. That's the attitude! It's all about playing around the constraints. Tech has limitations? Yes, but also opens tons of new possibilities.
latexr 2y ago
You’re describing a pie in the sky. A vision. Not reality. We have been burned many times already, nothing in this field is a given. > at this point we know that this can only get better. We don’t know that. It will probably get better, but will it be better enough? No one knows. > It should be happening within the next 10 years. Every revolution in tech is always ten years away. By now that’s a meme. Saying something is ten years away is about as valuable as saying one has no idea how doable it is. > Imagine Yes, I understand the goal. Everyone does, it’s not complicated. We can all imagine Star Trek technology, we all know where the compass is pointed, that doesn’t make it a given. In fact, the one thing we can say for sure about imagining how everything will be great in ten years is that we routinely fail to predict the bad parts. We don’t live in fantasy land, advancements in tech are routinely used for detrimental reasons.

idunnoman1222 2y ago

Obviously, the generation has to stop at some point and obviously from any key image you could continue generating if you had unlimited GPU, which I’m sorry they didn’t provide for you.

ajmurmann 2y ago
I am not sure it's obvious that you could continue generating from any key image and it wouldn't deteriorate into mush. If you take that museum scene and look at the vase-like display piece while walking around it as much as you can it already becomes fuzzy and has the beginnings of weird artifacts growing out of it.

Buttons840 2y ago

I was also disappointed by a still image showing a vast sky, but in motion you see it's just a painting on a short ceiling. The model interpreted the vast sky as a painting on a short ceiling.

fxtentacle 2y ago

As someone completely not involved in this project, I would predict that increasing the scene size while remaining halfway consistent isn't that difficult. Let me elaborate by using cat-4d.github.io, one of their competitors in this field of research: If you look at the "How it works" section you can see that the first step is to take an input video and then create artificial viewpoints of the same action being observed by other cameras. And then in the 2nd step, those viewpoints are merged into one 4D gaussian splatting scene. That 2nd step is pretty similar to 4D NeRF training, BTW, just with a different data format. Now if you need a small scene, you generate a few camera locations that are nearby. But there's nothing stopping you from generating other camera locations or even from using previously generated camera locations and moving the camera again, thereby propagating details that the AI invented outwards. So you could imagine this as you start with something "real" at the center of the map and then you create AI fakes with different camera positions in a circle around the real stuff, and then the next circle around the 1st-gen fakes, and the next circle, and so on. This process is mostly limited by 2 things: The ability of your AI model to capture a global theme. World Labs has demonstrated that they can invent details consistent with a theme in this demos, so I would assume they solved this already. And the other limit is computing time. A world box 2x in each direction is 8x the voxel data and I wouldn't be surprised if you need something like 16x to 32x the number of input images to fit the GSplats/NeRF. So most likely, the box limit is purely because the AI model is slow and execution is expensive and they didn't want to spend 10,000x the resources for making the box 10x larger.