6 ms·

Obviously incredibly cool, but it seems that people are incredibly overstating the applications of this. Realistically, how do you fit this into a movie, a TV

Obviously incredibly cool, but it seems that people are incredibly overstating the applications of this. Realistically, how do you fit this into a movie, a TV show, or a game? You write a text prompt, get a scene, and then everything is gone—the characters, props, rooms, buildings, environments, etc. won’t carry over to the next prompt.

fassssst 3y ago
It generates up to 1 minute videos which is like what all the kids are watching on TikTok and YouTube Shorts, right? And most ads are shorter than 1 minute.
- Janicc 3y ago
  A few months ago ai generated videos of people getting arrested for wearing big boots went viral on TikTok. I think this sort of silly "interdimensional cable" stuff will be really big on these short form video type sites once this level of quality becomes available to everyone.
  - JamesSwift 3y ago
    Robot chicken, but full motion video
dragonwriter 3y ago
> You write a text prompt, get a scene, and then everything is gone—the characters, props, rooms, buildings, environments, etc. won’t carry over to the next prompt. Sure, you can't use the text-to-video frontend for that purpose. But if you've got a t2v model as good as Sora clearly is, you've got the infrastructure for a lot more, as the ecosystem around the open-source models in the space has shown. The same techniques that allow character, object, etc., consistency in text-to-image models can be applied to text-to-video models.
matsemann 3y ago
It also seems hard to control exactly what you get. Like you'd want a specific pan, focus etc. to realize your vision. The examples here look good, but they aren't very specific. But it was the same with Dall-E and others in the beginning, and there's now lots of ways to control image generators. Same will probably happen here. This was a huge leap just in how coherent the frames are.
wilg 3y ago
You wait a year and they'll figure it out.
superconduct123 3y ago
It doesn't need to replace the whole movie You could use it for stuff like wide shots, close ups, random CG shots, rapid cut shots, stuff where you just cut to it once and don't need multiple angles To me it seem most useful for advertising where a lot of times they only show something once, like a montage
- Boss0565 3y ago
  I also see advertising (especially lower-budget productions, such as dropshipping or local TV commercials) being early adopters of this technology once businesses have access to this at an affordable price.
- planckscnst 3y ago
  And it would be magic for storyboarding. This would be such a useful tool for a director to iterate on a shot and then communicate that to the team
  - _sys49152 3y ago
    i could arrange in frameforge 3d shot by shot, even adjusting for motion in between, then export to an AI solution. that to me would be everything. of course then comes issues of consistency, adjustments & tweaks, etc
JKCalhoun 3y ago
You could use it to storyboard right now. Continuity of characters/wardrobe, etc. is not that important in storyboarding.
seydor 3y ago
tiktok
nprateem 3y ago
It's pretty obvious they just need to add the ability to prompt it with an image saying "continue in this style and make the character..."
deleted 3y ago
[deleted]
barbarr 3y ago
Explicit video clips? 4chan is gonna have a field day with this.
- matheusmoreira 3y ago
  Not necessarily explicit. Maybe just deliberately offensive. Maybe just weirdly specific. It's gonna be great.
sethammons 3y ago
What came to mind is what is right around the corner: you create segments and stitch them together. "ok, continue from the context on the last scene. Great. Ok, move the bookshelf. I want that cat to be more furry. Cool. Save this as scene 34." As clip sizes grow and context can be inferred from a previous scene, and a library of scenes can be made, boom, you can now create full feature length films, easy enough that elementary school kids will be able to craft up their imaginations.
al_borland 3y ago
Family Guy is built on out of context clips. It could also fill it for background videos in scenes, instead of getting real content they’d have to pay for, or making their own. The gangster movie Kevin was playing in Home Alone was specifically shot for that movie, from what I remember.
dogcomplex 3y ago
What? You're serious? Script => Video baseline. Take a frame of any character/prop/room/etc you want to remain consistent, and one shitty photoshop and it's part of the new scene. Incredibly overstating. That is an incredible lack of imagination buddy. Or even just basic craftsmanship.
padolsey 3y ago
Nah just fine-tune the model to a specific set of characters or aesthetic. It's not hard, already done with SDXL LoRAs. You can definitely generate a whole movie from just a storyboard.. if not now, then in maybe five yrs.
TrackerFF 3y ago
lots and lots and lots of b-roll and stock footage is about to get cheaper. Also, using this kind of footage is the bread and butter for a lot of marketers for their content. Imagine never having to pay stock footage companies

fassssst 3y ago

It generates up to 1 minute videos which is like what all the kids are watching on TikTok and YouTube Shorts, right? And most ads are shorter than 1 minute.

Janicc 3y ago
A few months ago ai generated videos of people getting arrested for wearing big boots went viral on TikTok. I think this sort of silly "interdimensional cable" stuff will be really big on these short form video type sites once this level of quality becomes available to everyone.
- JamesSwift 3y ago
  Robot chicken, but full motion video

dragonwriter 3y ago

> You write a text prompt, get a scene, and then everything is gone—the characters, props, rooms, buildings, environments, etc. won’t carry over to the next prompt. Sure, you can't use the text-to-video frontend for that purpose. But if you've got a t2v model as good as Sora clearly is, you've got the infrastructure for a lot more, as the ecosystem around the open-source models in the space has shown. The same techniques that allow character, object, etc., consistency in text-to-image models can be applied to text-to-video models.

matsemann 3y ago

It also seems hard to control exactly what you get. Like you'd want a specific pan, focus etc. to realize your vision. The examples here look good, but they aren't very specific. But it was the same with Dall-E and others in the beginning, and there's now lots of ways to control image generators. Same will probably happen here. This was a huge leap just in how coherent the frames are.

wilg 3y ago

You wait a year and they'll figure it out.

superconduct123 3y ago

It doesn't need to replace the whole movie You could use it for stuff like wide shots, close ups, random CG shots, rapid cut shots, stuff where you just cut to it once and don't need multiple angles To me it seem most useful for advertising where a lot of times they only show something once, like a montage

Boss0565 3y ago
I also see advertising (especially lower-budget productions, such as dropshipping or local TV commercials) being early adopters of this technology once businesses have access to this at an affordable price.
planckscnst 3y ago
And it would be magic for storyboarding. This would be such a useful tool for a director to iterate on a shot and then communicate that to the team
- _sys49152 3y ago
  i could arrange in frameforge 3d shot by shot, even adjusting for motion in between, then export to an AI solution. that to me would be everything. of course then comes issues of consistency, adjustments & tweaks, etc

JKCalhoun 3y ago

You could use it to storyboard right now. Continuity of characters/wardrobe, etc. is not that important in storyboarding.

seydor 3y ago

tiktok

nprateem 3y ago

It's pretty obvious they just need to add the ability to prompt it with an image saying "continue in this style and make the character..."

deleted 3y ago

[deleted]

barbarr 3y ago

Explicit video clips? 4chan is gonna have a field day with this.

matheusmoreira 3y ago
Not necessarily explicit. Maybe just deliberately offensive. Maybe just weirdly specific. It's gonna be great.

sethammons 3y ago

What came to mind is what is right around the corner: you create segments and stitch them together. "ok, continue from the context on the last scene. Great. Ok, move the bookshelf. I want that cat to be more furry. Cool. Save this as scene 34." As clip sizes grow and context can be inferred from a previous scene, and a library of scenes can be made, boom, you can now create full feature length films, easy enough that elementary school kids will be able to craft up their imaginations.

al_borland 3y ago

Family Guy is built on out of context clips. It could also fill it for background videos in scenes, instead of getting real content they’d have to pay for, or making their own. The gangster movie Kevin was playing in Home Alone was specifically shot for that movie, from what I remember.

dogcomplex 3y ago

What? You're serious? Script => Video baseline. Take a frame of any character/prop/room/etc you want to remain consistent, and one shitty photoshop and it's part of the new scene. Incredibly overstating. That is an incredible lack of imagination buddy. Or even just basic craftsmanship.

padolsey 3y ago

Nah just fine-tune the model to a specific set of characters or aesthetic. It's not hard, already done with SDXL LoRAs. You can definitely generate a whole movie from just a storyboard.. if not now, then in maybe five yrs.

TrackerFF 3y ago

lots and lots and lots of b-roll and stock footage is about to get cheaper. Also, using this kind of footage is the bread and butter for a lot of marketers for their content. Imagine never having to pay stock footage companies