5 ms·

It just plain isn't possible if you mean a prompt the size of what most people have been using lately, in the couple hundred character range. By sheer informati

by jerf 2y ago

It just plain isn't possible if you mean a prompt the size of what most people have been using lately, in the couple hundred character range. By sheer information theory, the number of possible interpretations of "a zoom in on a happy dog catching a frisbee" means that you can not match a particular clip out of the set with just that much text. You will need vastly more content; information about the breed, information about the frisbee, information about the background, information about timing, information about framing, information about lighting, and so on and so forth. Right now the AIs can't do that, which is to say, even if you sit there and type a prompt containing all that information, it is going to be forced to ignore most of the result. Under the hood, with the way the text is turned into vector embeddings, it's fairly questionable whether you'd agree that it can even represent such a thing. This isn't a matter of human-level AI or superhuman-level AI; it's just straight up impossible. If you want the information to match, it has to be provided. If it isn't there, an AI can fill in the gaps with "something" that will make the scene work, but expecting it to fill in the gaps the way you "want" even though you gave it no indication of what that is is expecting literal magic. Long term, you'll never have a coherent movie produced by stringing together a series of textual snippets because, again, that's just impossible. Some sort of long-form "write me a horror movie staring a precocious 22-year old elf in a far-future Ganymede colony with a message about the importance of friendship" AI that generates a coherent movie of many scenes will have to be doing a lot of some sort of internal communication in an internal language to hold the result together between scenes, because what it takes to hold stuff coherent between scenes is an amount of English text not entirely dissimilar in size from the underlying representation itself. You might as well skip the English middleman and go straight to an embedding not constrained by a human language mapping.

yeknoda 2y ago
something like a white paper with a mood board, color scheme, and concept art as the input might work. This could be sent into an LLM "expander" that increases the words and speficity. Then multiple reviews to tap things in the right direction.
- 3form 2y ago
  And I think this realistically is going to be the shape of the tools to come in the foreseeable future.
  - echelon 2y ago
    You should see what people are building with Open Source video models like HunYuan [1] and ComfyUI + Control Nets. It blows Sora out of the water. Check out the Banodoco Discord community [2]. These are the people pioneering steerable AI video, and it's all being built on top of open source. [1] https://github.com/Tencent/HunyuanVideo https://github.com/Tencent/HunyuanVideo [2] https://banodoco.ai/ https://banodoco.ai/
- mikepurvis 2y ago
  I expect this kind of thing is actually how it's going to work longer term, where AI is a copilot to a human artist. The human artist does storyboarding, sketching in backdrops and character poses in keyframes, and then the AI steps in and "paints" the details over top of it, perhaps based on some pre-training about what the characters and settings are so that there's consistency throughout a given work. The real trick is that the AI needs to be able to participate in iteration cycles, where the human can say "okay this is all mostly good, but I've circled some areas that don't look quite right and described what needs to be different about them." As far as I've played with it, current AIs aren't very good at revisiting their own work— you're basically just tweaking the original inputs and otherwise starting over from scratch each time.
  - programd 2y ago
    We will shortly have much better tweaking tools which work not only on images and video but concepts like what aspects a character should exhibit. See for example the presentation from Shapeshift Labs. https://www.shapeshift.ink/ https://www.shapeshift.ink/
echelon 2y ago
For those not in this space, Sora is essentially dead on arrival. Sora performs worse than closed source Kling and Hailuo, but more importantly, it's already trumped by open source too. Tencent is releasing a fully open source Hunyuan model [1] that is better than all of the SOTA closed source models. Lightricks has their open source LTX model and Genmo is pushing Mochi as open source. Black Forest Labs is working on video too. Sora will fall into the same pit that Dall-E did. SaaS doesn't work for artists, and open source always trumps closed source models. Artists want to fine tune their models, add them to ComfyUI workflows, and use ControlNets to precision control the outputs. Images are now almost 100% Flux and Stable Diffusion, and video will soon be 100% Hunyuan and LTX. Sora doesn't have much market apart from name recognition at this point. It's just another inflexible closed source model like Runway or Pika. Open source has caught up with state of the art and is pushing past it. [1] https://github.com/Tencent/HunyuanVideo https://github.com/Tencent/HunyuanVideo
- baserev 2y ago
  [flagged]
- circlefavshape 2y ago
  Their online version is all in Chinese (or at least some Chinese-looking script I don't understand) ... and they recommend an 80GB GPU to run the thing, which costs ~€15-18k. Yikes, guess I won't be doing this at home anytime soon
minimaxir 2y ago
> Under the hood, with the way the text is turned into vector embeddings, it's fairly questionable whether you'd agree that it can even represent such a thing. The text encoder may not be able to know complex relationships, but the generative image/video models that are conditioned on said text embeddings absolutely can. Flux, for example, uses the very old T5 model for text encoding, but image generations from it can (loosely) adhere to all rules and nuances in a multi-paragraph prompt: https://x.com/minimaxir/status/1820512770351411268 https://x.com/minimaxir/status/1820512770351411268
- dragonwriter 2y ago
  > but image generations from it can (loosely) adhere to all rules and nuances in a multi-paragraph prompt Flux certainly does not consistently do so across an arbitrary collection of multi-paragraph prompts, as anyone whose run more than a few long prompts past it would recongize; also, the tweet is wrong in the other direction, as well, longer language-model-preprocessed prompts for models that use CLIP (like various SD1.5 and SDXL derivatives) are, in fact, a common and useful technique. (You’d kind of think that the fact that generated prompt here is significantly longer than the 256 token window of T5 would be a clue that the 77 token limit of CLIP might not be as big of a constraint as the tweet was selling it as, too.)
robotresearcher 2y ago
> Long term, you'll never have a coherent movie produced by stringing together a series of textual snippets because, again, that's just impossible. Why snippets? Submit a whole script the way a writer delivers a movie to a director. The (automated) director/DP/editor could maintain internal visual coherence, while the script drives the story coherence.
- coffeebeqn 2y ago
  This almost certainly won’t work. Feel free to feed any of the hundreds of existing film scripts and test how coherent the models can be. My guess is not at all
  - sleepybrett 2y ago
    This will almost certainly be in theaters within 5 years, probably first as a small experimental project (think blair witch).
    - runarberg 2y ago
      The Blair Witch Project was a (surprise) creative masterpiece. It worked with very limited technology to create a very clever plot which was paired with an amazing marketing. The combination of which the world hadn’t seen before. It took some creative geniuses to peace the Blair Witch Project together. Generative AI will never produce an experience like that. I know never is a long time, but I’m still gonna call it. You simply can’t produce such a fresh idea by gathering a bunch of data and interpolating. Maybe someday enough AI will be good enough to create shorter or longer videos with some dialog and even a coherent story (though I doubt it), but it won‘t be fresh or creative. And we humans will at best enjoy it for its stupidity or sloppiness. Not for its cleverness or artistry.
      - dumbfounder 2y ago
        Why does the idea need to be generated by AI? Let people generate the ideas, the AI will help execute. I think soon (3-5 years) a determined person with no video skills will be able to put together a compelling movie (maybe a short). And that is massive. AI doesn’t have to do everything. Like all tech, it’s a productivity tool.
LASR 2y ago
What you are saying is totally correct. And this applies to language / code outputs as well. The number of times I’ve had engineers at my company type out 5 sentences and then expect a complete react webapp. But what I’ve found in practice is using LLMs to generate the prompt with low-effort human input (eg: thumbs up/down, multiple-choice etc) is quite useful. It generates walls of text, but with metaprompting, that’s kind of the point. With this, I’ve definitely been able to get high ROI out of LLMs. I suspect the same would work for vision output.
- kurthr 2y ago
  I'm not sure, but I think you're saying what I'm thinking. Stick the video you want to replicate into -o1 and ask for a descriptive prompt to generate a video with the same style and content. Take that prompt and put it into Sora. Iterate with human and o1 generated critical responses. I suspect you can get close pretty quickly, but I don't know the cost. I'm also suspicious that they might have put in "safeguards" to prevent some high profile/embarrassing rip-offs.
amelius 2y ago
Can't you just give it a photo of a dog, and then say "use this dog in this or that scene"?
- alpha_squared 2y ago
  How would that even work? A dog has physical features (legs, nose, eyes, ears, etc.) that they use to interact with the world around them (ground, tree, grass, sounds, etc.). And each one of those things has physical structures that compose senses (nervous system, optic nerves, etc.). There are layers upon layers of intricate complexity that took eons to develop and a single photo cannot encapsulate that level of complexity and density of information. Even a 3D scan can't capture that level of information. There is an implicit understanding of the physical world that helps us make sense of images. For example, a dog with all four paws standing on grass is within the bounds of possibility; a dog with six paws, two of which are on it's head, are outside the bounds of possibility. An image generator doesn't understand that obvious delineation and just approximates likelihood.
  - int_19h 2y ago
    A single photo doesn't have to capture all that complexity. It's carried by all those countless dog photos and videos in the training set of the model.
    - krainboltgreene 2y ago
      Actually, it does have to capture all of that complexity because it's a photon-based analysis of reality. You cannot take a photo without doing that.
- artemisart 2y ago
  Yes, the idea works and was explored with dreambooth/textual inversion for image diffusion models. https://dreambooth.github.io/ https://dreambooth.github.io/ https://textual-inversion.github.io/ https://textual-inversion.github.io/
lmm 2y ago
> You might as well skip the English middleman and go straight to an embedding not constrained by a human language mapping. How would you ever tweak or debug it in that case? It doesn't strictly have to be English, but some kind of human-readable representation of the intermediate stages will be vital.
szundi 2y ago
Sounds like we achieved 50% of AI then. The artifical is there, now we need the intelligence part.
baq 2y ago
Sora should be evaluated on xkcd strips as inputs.
prmoustache 2y ago
The whole point of AI stuff is not to produce exactly what you have in mind, but what you are describing. Same with text, code, images, video...
fennecbutt 2y ago
This is correct and even image generation models aren't really trained for comprehension of image composition yet. Even the models based off danbooru and E621 still aren't the best at that. And us furries like to tag art in detail. The best we can really do at the moment is regional prompting, perhaps they need something similar for video.

by jerf 2y ago

yeknoda 2y ago

something like a white paper with a mood board, color scheme, and concept art as the input might work. This could be sent into an LLM "expander" that increases the words and speficity. Then multiple reviews to tap things in the right direction.

3form 2y ago
And I think this realistically is going to be the shape of the tools to come in the foreseeable future.
- echelon 2y ago
  You should see what people are building with Open Source video models like HunYuan [1] and ComfyUI + Control Nets. It blows Sora out of the water. Check out the Banodoco Discord community [2]. These are the people pioneering steerable AI video, and it's all being built on top of open source. [1] https://github.com/Tencent/HunyuanVideo https://github.com/Tencent/HunyuanVideo [2] https://banodoco.ai/ https://banodoco.ai/
mikepurvis 2y ago
I expect this kind of thing is actually how it's going to work longer term, where AI is a copilot to a human artist. The human artist does storyboarding, sketching in backdrops and character poses in keyframes, and then the AI steps in and "paints" the details over top of it, perhaps based on some pre-training about what the characters and settings are so that there's consistency throughout a given work. The real trick is that the AI needs to be able to participate in iteration cycles, where the human can say "okay this is all mostly good, but I've circled some areas that don't look quite right and described what needs to be different about them." As far as I've played with it, current AIs aren't very good at revisiting their own work— you're basically just tweaking the original inputs and otherwise starting over from scratch each time.
- programd 2y ago
  We will shortly have much better tweaking tools which work not only on images and video but concepts like what aspects a character should exhibit. See for example the presentation from Shapeshift Labs. https://www.shapeshift.ink/ https://www.shapeshift.ink/

echelon 2y ago

For those not in this space, Sora is essentially dead on arrival. Sora performs worse than closed source Kling and Hailuo, but more importantly, it's already trumped by open source too. Tencent is releasing a fully open source Hunyuan model [1] that is better than all of the SOTA closed source models. Lightricks has their open source LTX model and Genmo is pushing Mochi as open source. Black Forest Labs is working on video too. Sora will fall into the same pit that Dall-E did. SaaS doesn't work for artists, and open source always trumps closed source models. Artists want to fine tune their models, add them to ComfyUI workflows, and use ControlNets to precision control the outputs. Images are now almost 100% Flux and Stable Diffusion, and video will soon be 100% Hunyuan and LTX. Sora doesn't have much market apart from name recognition at this point. It's just another inflexible closed source model like Runway or Pika. Open source has caught up with state of the art and is pushing past it. [1] https://github.com/Tencent/HunyuanVideo https://github.com/Tencent/HunyuanVideo

baserev 2y ago
[flagged]
circlefavshape 2y ago
Their online version is all in Chinese (or at least some Chinese-looking script I don't understand) ... and they recommend an 80GB GPU to run the thing, which costs ~€15-18k. Yikes, guess I won't be doing this at home anytime soon

minimaxir 2y ago

> Under the hood, with the way the text is turned into vector embeddings, it's fairly questionable whether you'd agree that it can even represent such a thing. The text encoder may not be able to know complex relationships, but the generative image/video models that are conditioned on said text embeddings absolutely can. Flux, for example, uses the very old T5 model for text encoding, but image generations from it can (loosely) adhere to all rules and nuances in a multi-paragraph prompt: https://x.com/minimaxir/status/1820512770351411268 https://x.com/minimaxir/status/1820512770351411268

dragonwriter 2y ago
> but image generations from it can (loosely) adhere to all rules and nuances in a multi-paragraph prompt Flux certainly does not consistently do so across an arbitrary collection of multi-paragraph prompts, as anyone whose run more than a few long prompts past it would recongize; also, the tweet is wrong in the other direction, as well, longer language-model-preprocessed prompts for models that use CLIP (like various SD1.5 and SDXL derivatives) are, in fact, a common and useful technique. (You’d kind of think that the fact that generated prompt here is significantly longer than the 256 token window of T5 would be a clue that the 77 token limit of CLIP might not be as big of a constraint as the tweet was selling it as, too.)

robotresearcher 2y ago

> Long term, you'll never have a coherent movie produced by stringing together a series of textual snippets because, again, that's just impossible. Why snippets? Submit a whole script the way a writer delivers a movie to a director. The (automated) director/DP/editor could maintain internal visual coherence, while the script drives the story coherence.

coffeebeqn 2y ago
This almost certainly won’t work. Feel free to feed any of the hundreds of existing film scripts and test how coherent the models can be. My guess is not at all
- sleepybrett 2y ago
  This will almost certainly be in theaters within 5 years, probably first as a small experimental project (think blair witch).
  - runarberg 2y ago
    The Blair Witch Project was a (surprise) creative masterpiece. It worked with very limited technology to create a very clever plot which was paired with an amazing marketing. The combination of which the world hadn’t seen before. It took some creative geniuses to peace the Blair Witch Project together. Generative AI will never produce an experience like that. I know never is a long time, but I’m still gonna call it. You simply can’t produce such a fresh idea by gathering a bunch of data and interpolating. Maybe someday enough AI will be good enough to create shorter or longer videos with some dialog and even a coherent story (though I doubt it), but it won‘t be fresh or creative. And we humans will at best enjoy it for its stupidity or sloppiness. Not for its cleverness or artistry.
    - dumbfounder 2y ago
      Why does the idea need to be generated by AI? Let people generate the ideas, the AI will help execute. I think soon (3-5 years) a determined person with no video skills will be able to put together a compelling movie (maybe a short). And that is massive. AI doesn’t have to do everything. Like all tech, it’s a productivity tool.

LASR 2y ago

What you are saying is totally correct. And this applies to language / code outputs as well. The number of times I’ve had engineers at my company type out 5 sentences and then expect a complete react webapp. But what I’ve found in practice is using LLMs to generate the prompt with low-effort human input (eg: thumbs up/down, multiple-choice etc) is quite useful. It generates walls of text, but with metaprompting, that’s kind of the point. With this, I’ve definitely been able to get high ROI out of LLMs. I suspect the same would work for vision output.

kurthr 2y ago
I'm not sure, but I think you're saying what I'm thinking. Stick the video you want to replicate into -o1 and ask for a descriptive prompt to generate a video with the same style and content. Take that prompt and put it into Sora. Iterate with human and o1 generated critical responses. I suspect you can get close pretty quickly, but I don't know the cost. I'm also suspicious that they might have put in "safeguards" to prevent some high profile/embarrassing rip-offs.

amelius 2y ago

Can't you just give it a photo of a dog, and then say "use this dog in this or that scene"?

alpha_squared 2y ago
How would that even work? A dog has physical features (legs, nose, eyes, ears, etc.) that they use to interact with the world around them (ground, tree, grass, sounds, etc.). And each one of those things has physical structures that compose senses (nervous system, optic nerves, etc.). There are layers upon layers of intricate complexity that took eons to develop and a single photo cannot encapsulate that level of complexity and density of information. Even a 3D scan can't capture that level of information. There is an implicit understanding of the physical world that helps us make sense of images. For example, a dog with all four paws standing on grass is within the bounds of possibility; a dog with six paws, two of which are on it's head, are outside the bounds of possibility. An image generator doesn't understand that obvious delineation and just approximates likelihood.
- int_19h 2y ago
  A single photo doesn't have to capture all that complexity. It's carried by all those countless dog photos and videos in the training set of the model.
  - krainboltgreene 2y ago
    Actually, it does have to capture all of that complexity because it's a photon-based analysis of reality. You cannot take a photo without doing that.
artemisart 2y ago
Yes, the idea works and was explored with dreambooth/textual inversion for image diffusion models. https://dreambooth.github.io/ https://dreambooth.github.io/ https://textual-inversion.github.io/ https://textual-inversion.github.io/

lmm 2y ago

> You might as well skip the English middleman and go straight to an embedding not constrained by a human language mapping. How would you ever tweak or debug it in that case? It doesn't strictly have to be English, but some kind of human-readable representation of the intermediate stages will be vital.

szundi 2y ago

Sounds like we achieved 50% of AI then. The artifical is there, now we need the intelligence part.

baq 2y ago

Sora should be evaluated on xkcd strips as inputs.

prmoustache 2y ago

The whole point of AI stuff is not to produce exactly what you have in mind, but what you are describing. Same with text, code, images, video...

fennecbutt 2y ago

This is correct and even image generation models aren't really trained for comprehension of image composition yet. Even the models based off danbooru and E621 still aren't the best at that. And us furries like to tag art in detail. The best we can really do at the moment is regional prompting, perhaps they need something similar for video.