4 ms·

In the looking ahead section of the post it says: “We are hard at work improving the size and fidelity of our generated worlds” I imagine the further you move

In the looking ahead section of the post it says: “We are hard at work improving the size and fidelity of our generated worlds” I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation.

latexr 2y ago
> I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation. Which is the same thing as saying this may turn out to be a dud, like so many other things in tech and the current crop of what we’re calling AI. Like I said, I get this is an early demo, but don’t oversell it. They could’ve started by being honest and clarifying they’re generating scenes (or whatever you want to call them, but they’re definitely not “worlds”), letting you play a bit, then explain the potential and progress. As it is, it just sounds like they want to immediately wow people with a fantasy and it detracts from what they do have.
- dmarcos 2y ago
  Fair criticism. I’m also not a fan of hyperbole. Still find World Labs stuff super intriguing and I’m optimist about them to be able to fulfill the vision.
- add-sub-mul-div 2y ago
  Maybe they think it's a good deal, producing some oversold tech demos in exchange for a decade's worth of funding and not having to produce anything more than an "Our Incredible Journey" letter at the end. The prospect of replacing all human labor has made it easier than ever to run the grift on investors in this time of peak FOMO.
deleted 2y ago
[deleted]
modeless 2y ago
Models are really great at making stuff up though. And video models already have very good consistency over thousands of frames. It seems like larger worlds shouldn't be a huge hurdle. I wonder why they launched without that, as this doesn't seem much better than previous work.
- dmarcos 2y ago
  To be fair they haven’t launched they are showing progress and laying out the vision. What previous work are you referring to?
TeMPOraL 2y ago
In general, it depends on how much the model ends up "understanding" the input. (I use "understand" here in the sense some would claim SOTA LLMs do.) You can imagine this as a spectrum. On the one end you have models that, at each output pixel, try to predict pixels that are locally similar to ones in previous frame; on the other end, you could imagine models that "parse" the initial input image to understand the scene - objects (buildings, doors, people, etc.) and their relationships, and separately, the style with which they're painted, and use that to extrapolate further frames[0]. The latter would obviously fare better, remaining stylistically consistent for longer. (This model claims to be of the second kind.) The way I see it: a human could do it[1], so there's no reason an ML model wouldn't be able to. -- [0] - Brute-force approach: 1) "style-untransfer" the input, i.e. style-transfer to some common style, e.g. photorealistic or sketch, 2) extrapolate the style-untransfered image, and 3) style-transfer result back using original input as style reference. Feels like it should work somewhat okay-ish; wonder if anyone tried that. [1] - And the hard part wouldn't be extrapolating the scene, but rather keeping the style.
- dmarcos 2y ago
  This indeed looks more like photogrammetry than a diffusion model predicting the next frame. There's 3D information extracted from the input image and likely additional generated poses that allow reconstructing the scene with gaussian splats. Not sure how much segmentation (understanding of each part of the scene) is going on. Probably not much if I have to guess.

latexr 2y ago

> I imagine the further you move from the input image, the more the model has to make up information and the harder to keep it consistent. Similar problem with video generation. Which is the same thing as saying this may turn out to be a dud, like so many other things in tech and the current crop of what we’re calling AI. Like I said, I get this is an early demo, but don’t oversell it. They could’ve started by being honest and clarifying they’re generating scenes (or whatever you want to call them, but they’re definitely not “worlds”), letting you play a bit, then explain the potential and progress. As it is, it just sounds like they want to immediately wow people with a fantasy and it detracts from what they do have.

dmarcos 2y ago
Fair criticism. I’m also not a fan of hyperbole. Still find World Labs stuff super intriguing and I’m optimist about them to be able to fulfill the vision.
add-sub-mul-div 2y ago
Maybe they think it's a good deal, producing some oversold tech demos in exchange for a decade's worth of funding and not having to produce anything more than an "Our Incredible Journey" letter at the end. The prospect of replacing all human labor has made it easier than ever to run the grift on investors in this time of peak FOMO.

deleted 2y ago

[deleted]

modeless 2y ago

Models are really great at making stuff up though. And video models already have very good consistency over thousands of frames. It seems like larger worlds shouldn't be a huge hurdle. I wonder why they launched without that, as this doesn't seem much better than previous work.

dmarcos 2y ago
To be fair they haven’t launched they are showing progress and laying out the vision. What previous work are you referring to?

TeMPOraL 2y ago

In general, it depends on how much the model ends up "understanding" the input. (I use "understand" here in the sense some would claim SOTA LLMs do.) You can imagine this as a spectrum. On the one end you have models that, at each output pixel, try to predict pixels that are locally similar to ones in previous frame; on the other end, you could imagine models that "parse" the initial input image to understand the scene - objects (buildings, doors, people, etc.) and their relationships, and separately, the style with which they're painted, and use that to extrapolate further frames[0]. The latter would obviously fare better, remaining stylistically consistent for longer. (This model claims to be of the second kind.) The way I see it: a human could do it[1], so there's no reason an ML model wouldn't be able to. -- [0] - Brute-force approach: 1) "style-untransfer" the input, i.e. style-transfer to some common style, e.g. photorealistic or sketch, 2) extrapolate the style-untransfered image, and 3) style-transfer result back using original input as style reference. Feels like it should work somewhat okay-ish; wonder if anyone tried that. [1] - And the hard part wouldn't be extrapolating the scene, but rather keeping the style.

dmarcos 2y ago
This indeed looks more like photogrammetry than a diffusion model predicting the next frame. There's 3D information extracted from the input image and likely additional generated poses that allow reconstructing the scene with gaussian splats. Not sure how much segmentation (understanding of each part of the scene) is going on. Probably not much if I have to guess.