6 ms·

The adage "a picture is worth a thousand words" has the nice corollary "A thousand words isn't enough to be precise about an image". Now expand that to movies

The adage "a picture is worth a thousand words" has the nice corollary "A thousand words isn't enough to be precise about an image". Now expand that to movies and games and you can get why this whole generative-AI bubble is going to pop.

szundi 2y ago
Comment was probably rather about the 360 degree turning heads etc.
GistNoesis 2y ago
(2020) https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 : an image is worth 16x16 words transformers for image recognition at scale (2021) https://arxiv.org/abs/2103.13915 https://arxiv.org/abs/2103.13915 : An Image is Worth 16x16 Words, What is a Video Worth? (2024) https://arxiv.org/abs/2406.07550 https://arxiv.org/abs/2406.07550 : An Image is Worth 32 Tokens for Reconstruction and Generation
- dartos 2y ago
  Those are indeed 3 papers.
  - GistNoesis 2y ago
    Yes in a nutshell they explain that you can express a picture or a video with relatively few discrete information. First paper is the most famous and prompted a lot of research to using text generation tools in the image generation domain : 256 "words" for an image, Second paper is 24 reference image per minutes of video, Third paper is a refinement of the first saying you only need 32 "tokens". I'll let you multiply the numbers. In kind of the same way as a who's who game, where you can identify any human on earth with ~32bits of information. The corollary being that contrary to what parent is telling there is no theoretical obstacle to obtaining a video from a textual description.
    - dartos 2y ago
      I think something is getting lost in translation. These papers, from my quick skim (tho I did read the first one fully years ago,) seem to show that some images and to an extent video can be generated from discrete tokens, but does not show that exact images nor that any image can be. For instance, what combination of tokens must I put in to get _exactly_ Mona Lisa or starry night? (Tho these might be very well represented in the data set. Maybe a lesser known image would be a better example) As I understand, OC was saying that they can’t produce what they want with any degree of precision since there’s no way to encode that information in discrete tokens.
      - GistNoesis 2y ago
        If you want to know what tokens you want to obtain _exactly_ Mona Lisa, or any other image, you take the image and put it through your image tokenizer aka encode it, and if you have the sequence of token you can decode it to an image. VQ-VAE (Vector Quantised-Variational AutoEncoder), (2017) https://arxiv.org/abs/1711.00937 https://arxiv.org/abs/1711.00937 The whole encoding-decoding process is reversible, and you only lose some imperceptible "details", the process can be either trained with a L2Loss, or a perceptual loss depending what you value. The point being that images which occurs naturally are not really information rich and can be compressed a lot by neural networks of a few GB that have seen billions of pictures. With that strong prior, aka common knowledge, we can indeed paint with words.
        dartos 2y ago
        Maybe I’m not able to articulate my thought well enough. Taking an existing image and reversing the process to get the tokens that led to it then redoing that doesn’t seem the same as inserting token to get a precise novel image. Especially since, as you said, we’d lose some details, it suggests that not all images can be perfectly described and recreated. I suppose I’ll need to play around with some of those techniques.
        GistNoesis 2y ago
TeMPOraL 2y ago
> Now expand that to movies and games and you can get why this whole generative-AI bubble is going to pop. What will save it is that, no matter how picky you are as a creator, your audience will never know what exactly was that you dreamed up, so any half-decent approximation will work. In other words, a corollary to your corollary is, "Fortunately, you don't need them to be, because no one cares about low-order bits". Or, as we say in Poland, "What the eye doesn't see, the heart doesn't mourn."
- jsheard 2y ago
  > What will save it is that, no matter how picky you are as a creator, your audience will never know what exactly was that you dreamed up, so any half-decent approximation will work. Part of the problem is the "half decent approximations" tend towards a clichéd average, the audience won't know that the cool cyberpunk cityscape you generated isn't exactly what you had in mind, but they will know that it looks like every other AI generated cyberpunk cityscape and mentally file your creation in the slop folder. I think the pursuit of fidelity has made the models less creative over time, they make fewer glaring mistakes like giving people six fingers but their output is ever more homogenized and interchangable.
  - randomcatuser 2y ago
    a somewhat counterintuitive argument is this: AI models will make the overall creative landscape more diverse and interesting, ie, less "average"! Imagine the space of ideas as a circle, with stuff in the middle being more easy to reach (the "cliched average"). Previously, traversing the circle was incredibly hard - we had to use tools like DeviantArt, Instragram, etc to agglomerate the diverse tastes of artists, hoping to find or create the style we're looking for. Creating the same art style is hiring the artist. As a result, on average, what you see is the result of huge amounts of human curation, effort, and branding teams. Now reduce the effort 1000x, and all of a sudden, it's incredibly easy to reach the edge of the circle (or closer to it). Sure, we might still miss some things at the very outer edge, but it's equivalent to building roads. Motorists appear, people with no time to sit down and spend 10000 hours to learn and master a particular style can simply remix art and create things wildly beyond their manual capabilities. As a result, the amount of content in the infosphere skyrockets, the tastemaking velocity accelerates, and you end up with a more interesting infosphere than you're used to.
throwup238 2y ago
“A frame is worth a billion rays” The last production I worked on averaged 16 hours per frame for the final rendering. The amount of information encoded in lighting, models, texture, maps, etc is insane.
- bongodongobob 2y ago
  What were you working on? It took a month to render 2 seconds of video?
  - elmigranto 2y ago
    I would guess there is more than one computer :) Pixar's stuff famously takes days per frame.
    - Arelius 2y ago
      > Pixar's stuff famously takes days per frame. Do you have a citation for this? My guess would be much closer to a couple of hours per frame.
      - elmigranto 2y ago
        https://sciencebehindpixar.org/pipeline/rendering https://sciencebehindpixar.org/pipeline/rendering
  - Arelius 2y ago
    Most VFX productions take over 2 CPU hours a frame for final video, and have for a very long time. It takes shorter then a month since this gets parallelized on large render farms.
  - throwup238 2y ago
    VFX heavy feature for a Disney subsidiary. Each frame is rendered independently of each other - it’s not like video encoding where each frame depends on the previous one, they all have their own scene assembly that can be sent to a server to parallelize rendering. With enough compute, the entire film can be rendered in a few days. (It’s a little more complicated than that but works to a first order approximation) I don’t remember how long the final rendering took but it was nearly two months and the final compute budget was 7 or 8 figures. I think we had close to 100k cores running at peak from three different render farms during crunch time, but don’t take my word for it I wasn’t producing the picture.
Al-Khwarizmi 2y ago
If you can build a system that can generate engaging games and movies, from an economic (bubble popping or not popping) point of view it's largely irrelevant whether they conform to fine-grained specifications by a human or not.
- dartos 2y ago
  In other words: If you find a silver bullet then everything else is largely irrelevant. Idk if you noticed but that “if” is carrying an insane amount of weight.
- jsheard 2y ago
  Text generation is the most mature form of genAI and even that isn't even remotely close to producing infinite engaging stories. Adding the visual aspect to make that story into a movie or the interactive element to turn it into a game is only uphill from there.
  - deleted 2y ago
    [deleted]
beambot 2y ago
Corollary: I couldn't create an original visual piece of art to save my life, so prompting is infinitely better than what I could do myself (or am willing to invest time in building skills). The gen-AI bubble isn't going to burst. Pareto always wins.
mrandish 2y ago
I agree that people who want any meaningful precision in their visual results will inevitably be disappointed.
fooker 2y ago
Sure it's going to pop. But when is the important question. Being too early about this and being wrong are the same.
stale2002 2y ago
You are half right. Its funny because I use the same same. Mine is "A picture is worth a thousand words. thats why it takes 1000 words to describe the exact image that you want! Much better to just use Image to Image instead". Thats my full quote on this topic. And I think it stands. Sure, people won't describe a picture. instead, they will take an existing picture or video, and do modifications of it, using AI. That is much much simpler and more useful, if you can file a scene, and then animate it later with AI.
meta_x_ai 2y ago
A picture is worth a thousand words. A word is worth a thousand pictures. (E.g Love) It is abstraction all the way
- gloosx 2y ago
  it is all Information to be precise.
raincole 2y ago
The point is not to be precise. It's to be "good enough". Trust me, even if you work with human artists, you'll keep saying "it's not quite I initially invisioned, but we don't have budget/time for another revision, so it's good enough for now." all the time.
soheil 2y ago
Maybe your AI bubble! If you define AI to be something like just another programming language yes you will be sadly disappointed. You see it as an employee with its own intuitions and ways of doing things that you're trying to micromanage. I have a bad feeling that you'd be a horrible manager if you ever were one.
ben_w 2y ago
> Now expand that to movies and games and you can get why this whole generative-AI bubble is going to pop. The prior sentence does not imply the conclusion.
8n4vidtmkvmk 2y ago
Actually, I've gotten some great results with image2text2image with less than a thousand words. Maybe not enough for a video, but for some not too crazy images, it is enough!

szundi 2y ago

Comment was probably rather about the 360 degree turning heads etc.

GistNoesis 2y ago

(2020) https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 : an image is worth 16x16 words transformers for image recognition at scale (2021) https://arxiv.org/abs/2103.13915 https://arxiv.org/abs/2103.13915 : An Image is Worth 16x16 Words, What is a Video Worth? (2024) https://arxiv.org/abs/2406.07550 https://arxiv.org/abs/2406.07550 : An Image is Worth 32 Tokens for Reconstruction and Generation

dartos 2y ago
Those are indeed 3 papers.
- GistNoesis 2y ago
  Yes in a nutshell they explain that you can express a picture or a video with relatively few discrete information. First paper is the most famous and prompted a lot of research to using text generation tools in the image generation domain : 256 "words" for an image, Second paper is 24 reference image per minutes of video, Third paper is a refinement of the first saying you only need 32 "tokens". I'll let you multiply the numbers. In kind of the same way as a who's who game, where you can identify any human on earth with ~32bits of information. The corollary being that contrary to what parent is telling there is no theoretical obstacle to obtaining a video from a textual description.
  - dartos 2y ago
    I think something is getting lost in translation. These papers, from my quick skim (tho I did read the first one fully years ago,) seem to show that some images and to an extent video can be generated from discrete tokens, but does not show that exact images nor that any image can be. For instance, what combination of tokens must I put in to get _exactly_ Mona Lisa or starry night? (Tho these might be very well represented in the data set. Maybe a lesser known image would be a better example) As I understand, OC was saying that they can’t produce what they want with any degree of precision since there’s no way to encode that information in discrete tokens.
    - GistNoesis 2y ago
      If you want to know what tokens you want to obtain _exactly_ Mona Lisa, or any other image, you take the image and put it through your image tokenizer aka encode it, and if you have the sequence of token you can decode it to an image. VQ-VAE (Vector Quantised-Variational AutoEncoder), (2017) https://arxiv.org/abs/1711.00937 https://arxiv.org/abs/1711.00937 The whole encoding-decoding process is reversible, and you only lose some imperceptible "details", the process can be either trained with a L2Loss, or a perceptual loss depending what you value. The point being that images which occurs naturally are not really information rich and can be compressed a lot by neural networks of a few GB that have seen billions of pictures. With that strong prior, aka common knowledge, we can indeed paint with words.
      - dartos 2y ago
        Maybe I’m not able to articulate my thought well enough. Taking an existing image and reversing the process to get the tokens that led to it then redoing that doesn’t seem the same as inserting token to get a precise novel image. Especially since, as you said, we’d lose some details, it suggests that not all images can be perfectly described and recreated. I suppose I’ll need to play around with some of those techniques.
        GistNoesis 2y ago

TeMPOraL 2y ago

> Now expand that to movies and games and you can get why this whole generative-AI bubble is going to pop. What will save it is that, no matter how picky you are as a creator, your audience will never know what exactly was that you dreamed up, so any half-decent approximation will work. In other words, a corollary to your corollary is, "Fortunately, you don't need them to be, because no one cares about low-order bits". Or, as we say in Poland, "What the eye doesn't see, the heart doesn't mourn."

jsheard 2y ago
> What will save it is that, no matter how picky you are as a creator, your audience will never know what exactly was that you dreamed up, so any half-decent approximation will work. Part of the problem is the "half decent approximations" tend towards a clichéd average, the audience won't know that the cool cyberpunk cityscape you generated isn't exactly what you had in mind, but they will know that it looks like every other AI generated cyberpunk cityscape and mentally file your creation in the slop folder. I think the pursuit of fidelity has made the models less creative over time, they make fewer glaring mistakes like giving people six fingers but their output is ever more homogenized and interchangable.
- randomcatuser 2y ago
  a somewhat counterintuitive argument is this: AI models will make the overall creative landscape more diverse and interesting, ie, less "average"! Imagine the space of ideas as a circle, with stuff in the middle being more easy to reach (the "cliched average"). Previously, traversing the circle was incredibly hard - we had to use tools like DeviantArt, Instragram, etc to agglomerate the diverse tastes of artists, hoping to find or create the style we're looking for. Creating the same art style is hiring the artist. As a result, on average, what you see is the result of huge amounts of human curation, effort, and branding teams. Now reduce the effort 1000x, and all of a sudden, it's incredibly easy to reach the edge of the circle (or closer to it). Sure, we might still miss some things at the very outer edge, but it's equivalent to building roads. Motorists appear, people with no time to sit down and spend 10000 hours to learn and master a particular style can simply remix art and create things wildly beyond their manual capabilities. As a result, the amount of content in the infosphere skyrockets, the tastemaking velocity accelerates, and you end up with a more interesting infosphere than you're used to.

throwup238 2y ago

“A frame is worth a billion rays” The last production I worked on averaged 16 hours per frame for the final rendering. The amount of information encoded in lighting, models, texture, maps, etc is insane.

bongodongobob 2y ago
What were you working on? It took a month to render 2 seconds of video?
- elmigranto 2y ago
  I would guess there is more than one computer :) Pixar's stuff famously takes days per frame.
  - Arelius 2y ago
    > Pixar's stuff famously takes days per frame. Do you have a citation for this? My guess would be much closer to a couple of hours per frame.
    - elmigranto 2y ago
      https://sciencebehindpixar.org/pipeline/rendering https://sciencebehindpixar.org/pipeline/rendering
- Arelius 2y ago
  Most VFX productions take over 2 CPU hours a frame for final video, and have for a very long time. It takes shorter then a month since this gets parallelized on large render farms.
- throwup238 2y ago
  VFX heavy feature for a Disney subsidiary. Each frame is rendered independently of each other - it’s not like video encoding where each frame depends on the previous one, they all have their own scene assembly that can be sent to a server to parallelize rendering. With enough compute, the entire film can be rendered in a few days. (It’s a little more complicated than that but works to a first order approximation) I don’t remember how long the final rendering took but it was nearly two months and the final compute budget was 7 or 8 figures. I think we had close to 100k cores running at peak from three different render farms during crunch time, but don’t take my word for it I wasn’t producing the picture.

Al-Khwarizmi 2y ago

If you can build a system that can generate engaging games and movies, from an economic (bubble popping or not popping) point of view it's largely irrelevant whether they conform to fine-grained specifications by a human or not.

dartos 2y ago
In other words: If you find a silver bullet then everything else is largely irrelevant. Idk if you noticed but that “if” is carrying an insane amount of weight.
jsheard 2y ago
Text generation is the most mature form of genAI and even that isn't even remotely close to producing infinite engaging stories. Adding the visual aspect to make that story into a movie or the interactive element to turn it into a game is only uphill from there.
- deleted 2y ago
  [deleted]

beambot 2y ago

Corollary: I couldn't create an original visual piece of art to save my life, so prompting is infinitely better than what I could do myself (or am willing to invest time in building skills). The gen-AI bubble isn't going to burst. Pareto always wins.

mrandish 2y ago

I agree that people who want any meaningful precision in their visual results will inevitably be disappointed.

fooker 2y ago

Sure it's going to pop. But when is the important question. Being too early about this and being wrong are the same.

stale2002 2y ago

You are half right. Its funny because I use the same same. Mine is "A picture is worth a thousand words. thats why it takes 1000 words to describe the exact image that you want! Much better to just use Image to Image instead". Thats my full quote on this topic. And I think it stands. Sure, people won't describe a picture. instead, they will take an existing picture or video, and do modifications of it, using AI. That is much much simpler and more useful, if you can file a scene, and then animate it later with AI.

meta_x_ai 2y ago

A picture is worth a thousand words. A word is worth a thousand pictures. (E.g Love) It is abstraction all the way

gloosx 2y ago
it is all Information to be precise.

raincole 2y ago

The point is not to be precise. It's to be "good enough". Trust me, even if you work with human artists, you'll keep saying "it's not quite I initially invisioned, but we don't have budget/time for another revision, so it's good enough for now." all the time.

soheil 2y ago

Maybe your AI bubble! If you define AI to be something like just another programming language yes you will be sadly disappointed. You see it as an employee with its own intuitions and ways of doing things that you're trying to micromanage. I have a bad feeling that you'd be a horrible manager if you ever were one.

ben_w 2y ago

> Now expand that to movies and games and you can get why this whole generative-AI bubble is going to pop. The prior sentence does not imply the conclusion.

8n4vidtmkvmk 2y ago

Actually, I've gotten some great results with image2text2image with less than a thousand words. Maybe not enough for a video, but for some not too crazy images, it is enough!