39 ms·

Lumiere: A space-time diffusion model for realistic video generation

• whywhywhywhy 3y ago
  [flagged]
• codetrotter 3y ago
  Their GitHub doesn’t have anything other than the linked page currently https://github.com/lumiere-video https://github.com/lumiere-video Nor did they claim it would. But I had to check anyway, and there wasn’t any link I could see to the GitHub profile. So here’s a link for anyone else that wants to check and don’t want to type the url of their profile manually from looking at the hosted website url.
  • gardnr 3y ago
    A popular move with ai / ml folks: Use GitHub to publish information about a thing that is not open. Then "It's on GitHub".
    • sho_hn 3y ago
      I see this a lot as well, and I think we really ought to call it out more often. It should be clear whether a GitHub publication enables downstream use or contribution.
    • whywhywhywhy 3y ago
      So tired of the academia brained ML researchers. Can’t wait for the next generation of teenagers to completely change this space and bypass this silliness completely.
      • jampekka 3y ago
        Isn't this more for-profit-corporation brained ML researchers? ML from academia tends to be released open source nowadays.
      • Der_Einzige 3y ago
        There are few better ways to get a cushy 300K a year plus jobs, and publishing Ml research is one of those ways. The new generation will simply do more publishing.
  • 3abiton 3y ago
    LLMs have set a new trend unfortunately.
• smusamashah 3y ago
  The examples are lot more consistent and longer than other techniques we have seen before. Legs are not sliding on the floor as much as they do with other models. On the other hand, human faces didn't look good. e.g. the mona lisa smiling. To me this looks like the first good video generation model. EDIT: Just noticed its by Google, NVM, will never be released publicly.
  • ithkuil 3y ago
    No but researchers will build on this research, as researchers do and eventually some company will run a successful product based on the result of a lot of research that includes this and we'll be bitching about google falling behind. Google is sponsoring a lot of cutting edge research and sharing it openly. How cool is that? How long will it last?
  • i-use-nixos-btw 3y ago
    If it were to be released publicly, I'd give it a week before NSFW models based on it were uploaded to Civitai.
    • Frost1x 3y ago
      A lot of current AI techniques are making people reevaluate their perspectives on free speech. We seem to value freedom of speech (and expression) only to a tipping point that it begins to invade other aspects of life. So far the noise and rate has been low enough people at large support free speech but newer information techniques are making it possible to generate a lot more realistic noise (faux signal, if you will) at higher rates (it’s becoming cheaper and easier to do and scale). So while you certainly have a point I mostly agree with, we’re letting private entities policies dictate the limitations of expression, at least for the time being (until someone comes along and makes these widely available for free or cheap without such ethical policies). It does go to show just how much sway industries have on markets through their policies with no public oversight, which to me is concerning.
      • kjqgqkejbfefn 3y ago
• feverzsj 3y ago
  As realistic as my blurry dream.
  • yard2010 3y ago
    Give it a break, until 2 years ago you wouldn't even dream about a fraction of what's out already Everything is relative!
• wiz21c 3y ago
  newb question: Do these models actually learn a 3D representation or do they just learn "something" that is good enough to produce an very convincing impression of 3D ? Subquestion: if they don't learn 3D, can we say that models learning a 3D representation first will lead to even better productions ?
  • astrange 3y ago
    > Do these models actually learn a 3D representation or do they just learn "something" that is good enough to produce a very convincing impression of 3D ? The second, but at the limit it's the same thing of course. > Subquestion: if they don't learn 3D, can we say that models learning a 3D representation first will lead to even better productions ? Generally speaking manual feature engineering almost always turns out to be a waste of time if you can just make the model bigger; this is called "the bitter lesson".
  • l33tman 3y ago
    It has been shown that at least still-image generators learn a 3D representation internally and uses it to bootstrap their generation. If you think about it this is the only way they can be so good at shadows and reflections, perspective and lighting etc.
    • wiz21c 3y ago
      Could you point at some articles which explores that aspect ? It sounds very interesting. Thanks !
      • l33tman 3y ago
        Please see https://arxiv.org/abs/2306.05720 https://arxiv.org/abs/2306.05720 Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model "... In this work, we investigate a basic interpretability question: does an LDM create and use an internal representation of simple scene geometry? Using linear probes, we find evidence that the internal activations of the LDM encode linear representations of both 3D depth data and a salient-object / background distinction. These representations appear surprisingly early in the denoising process−well before a human can easily make sense of the noisy images. ..."
• sho_hn 3y ago
  With the weird creepy dream-like nature of these little AI video gen samples, I'm perpetually disappointed that none of these papers ever include a "dreaming of electric sheep" prompt as an easter egg.
• 55555 3y ago
  The negative comments here shock me. This is the most amazing text-to-video we've ever seen, by a longshot. It's good enough for many uses. Absolutely mindblowing. Great job to the people who worked on this.
  • boesboes 3y ago
    What negative comments?
    • saurik 3y ago
      Yeah... I have only found a single negative comment about the quality--"As realistic as my blurry dream."--and it comes across as more of a cynical joke than a true negative review.
  • eurekin 3y ago
    Just chiming in to second your opinion. Years ago, I wouldn't even dare to dream it would be possible. It's nowhere near, what people are used to watch normally, but the fact it's even trying to compete is insane.
  • danielbln 3y ago
    I'm excited about these text 2 video models, what I'm not excited about is that it's Google publishing this. That means, no code, nothing deployed to try and most likely we will never ever hear about this ever again, or maybe it quietly hits Vertex AI in 2 years (like Imagen) and no-one will care. Also, ever since the Gemini marketing video shenanigans, I don't really feel like trusting whatever Google's research says they have, if I can't test it myself.
    • sho_hn 3y ago
      > Also, ever since the Gemini marketing video shenanigans, I don't really feel like trusting whatever Google's research says they have, if I can't test it myself. The video was released by Google product marketing for a launch to customers, not research. I'm still somewhat confused by this one. I understand the community has decided to be harsh on Google for that video to draw a line - fair, truth in advertising, etc. -, but at the same time, we all had an understanding of where that tech is at currently and the pace it progresses at. Did anyone watching it really assume it was realtime? Can we not differentiate between technical publications and marketing anymore? Do we have to vilify everyone in an R&D department for the sins of the product marketing wing?
• throwuwu 3y ago
  We will see the first feature length AI generated movie this year. If you think I’m crazy then consider that even way back at the dawn of cinema the average shot length was 12 seconds and today it is only 2.5 seconds. There are a few important techniques to be refined such as keeping consistent subjects between generations but I could see many inconsistencies being made up for by applying existing methods such as separating the layers based on depth allowing more static images to be used or creating simple 3D models with textures where more depth is needed. With enough effort and skill someone could probably do it with existing technologies.
  • __loam 3y ago
    It will probably be utter dogshit like every other piece of media people are pumping out with this crap.
    • throwuwu 3y ago
      90 percent of everything is crap but I’ve seen plenty of creative people make compelling films with digital tools. This technology puts that capability within each of people who aren’t also 3D modellers or graphic artists so we’re bound to get more output, good and bad. Same deal with when film cameras became cheap and widely available or digital cameras or iPhones.
  • seydor 3y ago
    why would we make a "movie" instead of one storyline where viewers can customize the costumes at will?
  • felipeerias 3y ago
    It’s easy to imagine a film maker creating multiple draft versions of a movie to polish the script and the cinematography, similar to how now they use storyboards.
• rysertio 3y ago
  The amount of computing resources it's going to take to retrain the model is enormous. So the most of us will have to wait for a big company to publish or leak their weights until we get to use anything written in the paper.
• qwertox 3y ago
  If this isn't the bell ringing, announcing that an entire industry will soon collapse, then I don't know what could announce it more clearly. I give it 5 years until it has been normalized to see AI generated TV/YT ads, 10 to 15 when traditionally made ones will be in the minority. In the beginning just a bunch of geeks in front of computers crafting the prompts, later everyone will be able to make it. It will probably be access to computing resources which will be the limiting factor.
  • tetris11 3y ago
    Yep, and my cynical side is just hoping that the GPU vendors aren't going to deliberately limit the number of user-accessible resources there are to force people to depend on their cloud platforms.
    • __loam 3y ago
      There's probably going to be more and more specialized hardware for this stuff. Things like H100s are already pretty inaccessible to consumers.
  • dkjaudyeqooe 3y ago
    That sounds like a great increase in productivity. But also you're making the mistake of extrapolating against the realities of the techniques. Things may improve over time but prompts and random seeds aren't great for detailed work, so there are limitations which seriously limit the usefulness. "Everyone will be able to make it" is likely true, but the specialist stuff will likely remain and those users will likely be made more productive. It's those in the middle that will lose out. That an industry is destroyed is neither here nor there. Sucks to have your business/job taken away but that's how the system works. That which created your business also will destroy it.
    • wegfawefgawefg 3y ago
      Have you played with control net over comfyui? Try it. You can pose arbitrary figures. Theres gonna be full kits that provide control over every aspect of generation.
• richrichardsson 3y ago
  The video inpainting is interesting. My kids were watching old Spongebob episodes recently and the 4:3 aspect ratio was jarring to me. I thought it would be an interesting use case to in-paint the side borders to bring it back into 16:9 aspect, but I suppose it would need some careful fine-tuning with some kind of look-ahead for objects that enter frame from the sides.
  • AwesomeLemon 3y ago
    Can't you just process the video backwards?
  • araes 3y ago
    That actually sounds like a product somebody in the television and movie industry might buy. Dynamic adjustment of fixed aspect ratio film imagery to non-native sizes without stretch or obvious distortion. Guess all the added edges accurately enough that audiences won't notice. 4:3 <-> 16:9 <-> 143:100 (IMAX) <-> 11:8 (Academy) <-> 3:2 (35mm) <-> 16:10 (tablets/desktops) Make a new movie look like a classic b/w silent, then give it the correct frame. Adapt any movie to smoothly work on IMAX displays.
    • berniedurfee 3y ago
      Or make those darn new fangled vertical TikTok style videos watchable!
      • theWreckluse 3y ago
        I recently saw someone do the opposite - Uncrop Bollywood movies into vertical format.
• mdrzn 3y ago
  DAMN! Take this announcement back just 2-3 years and it would have been MIND BLOWING. I know we're all used to new releases like this coming very soon and very fast, but I'm amazed. I can't wait to have a software with this abilities. edit: nvm, it's by Google. I'll wait for an open source to be released.
• pmontra 3y ago
  > Hover over the video to see the input prompt That doesn't work on a phone. I hoped they added an event handler for touching the animations. Instead they forgot they have a mobile OS and that they sell phones.
  • johnnymellor 3y ago
    At least on Chrome for Android, you can long-press to trigger the hover effect. Works on many websites. (There are inconvenient side-effects like selecting text, but it's better than nothing.)
  • itishappy 3y ago
    Did they? Works fine on my Pixel.
  • 7734128 3y ago
    Worked in Kiwi, which is a Chrome derivate
  • pmontra 3y ago
    OP here: my bad. I didn't enable enough JS sites with NoScript. It works now but touching the images. Thanks to everybody that replied to me.
• jampekka 3y ago
  [flagged]
• 88j88 3y ago
  Is this all real, or faked a-la gemini?
• ilaksh 3y ago
  Congratulations to the researchers. It would be nice if it wasn't Google though. Because we probably will have to wait 3-6 months for it show up in their Vertex API. For special customers only.
  • deleted 3y ago
    [deleted]
• sorenjan 3y ago
  This is very impressive, but their approach to generate the whole temporal duration at once limits it to short clips. I guess one of the next steps is to make overlapping "clips" that then becomes longer videos.
• pylua 3y ago
  This is absolutely unbelievable. Truly impressive. I felt like this was maybe 5-10 years away.
• araes 3y ago
  Pixel themed post for pixel themed paper. It's rather impressive and quite quickly will likely result in a huge hoard of "make a movie with a paragraph" programs. It's Google - It will probably go in a box and be a Rick and Morty gadget we never see. It has a cool author format list I like. The 1,2,3,4,*,+ thing is nice for lead authors, institute attribution, and core contributors. I read so many astronomy and physics papers that are 10+ authors long, and I have no idea who did anything. The arXiv link for example shows no similar formatting. It will probably be immediately used for abusive porn. Walking Woman Example: (5th variation) "Wearing no clothing"
  • whamlastxmas 3y ago
    This didn't occur to me but yeah, abusive porn is about to be rampant with this sort of tech. Every single person in the world is soon to have graphic realistic looking pornography with their face on it
• deleted 3y ago
  [deleted]
• StarterPro 3y ago
  What is the point of this? I feel like it only serves to hinder real artists who could use the money that people are paying for these services and models. Maybe I'm too poor or short-sighted to see it. I would rather an actual animator create something beautiful for me rather than an AI spit out something that needs to be worked on by an actual animator ANYWAY.
  • chankstein38 3y ago
    You're clearly not the target audience for this then. That's usually my assumption when I can't figure out a use case for some research a bunch of people are excited about.
    • sofixa 3y ago
      Crypto generally and NFTs in particular are good indicators that things can get people excited and have no substance. Even scams and Ponzi schemes have "target audiences" but that doesn't make them any useful or good.
      • chankstein38 3y ago
        Right but this is generating decent-quality video in segments longer than the time of your average movie shot. I'm sure it'd take some fiddling but I'm excited for a model this good to come out so I can try some fancy multi-shot videos. I saw someone else say "I'm sure it'll be crap like all of the other AI stuff I've seen" but that's a naive view. Things that have been 100% created by AI, sure they're kind of boring a lot of the time. But this kind of tech gives people with a creative mind, but no money or time or resources to create a storytelling movie/video, the resources to do it. Obv ignoring the fact that Goog will never release this, if something like this did come out, it'd be game changing for a lot of people. Think about something like RPG Maker. Yeah we've had a ton of random garbage come out of that platform but there were also incredible. AI isn't just some garbage maker. It is a paint brush that enables people who are alone in their room to make something bigger than them.
• seydor 3y ago
  I wonder who s going to make a model that creates and textures a 3D world with AI. It's going to be a necessity for VR goggles to find some nongimmicky use cases
• RcouF1uZ4gsC 3y ago
  Sorry, I discount all AI text/image/video generation that actually doesn’t have a demo site where I can put in prompts and see what is being generated. It is so easy to game and tweak examples, especially since there is a random component to them. For example, you could do a prompt 1 million times and only show the best response. Or you could use prompts that it’s optimized for. The reason ChatGPT and Dall-e captured the public’s imagination is that the public could actually put in their prompts and see the results.
• alkonaut 3y ago
  If "translator" was the victim of LLMs and "stock photographer" of diffusion models, which job is the first to be threatened by diffusion models for moving pictures? OnlyFans streamers?
  • rwmj 3y ago
    The people involved in producing TV adverts.
  • caballeto 3y ago
    Video editors, special effects artists, influencers or content creators who heavily rely on video content. @ChatGPT
• deleted 3y ago
  [deleted]
• thih9 3y ago
  Looks like they're frequently mixing old images with a modern dataset; if I took a portrait of George Washington and prompt for "a man smiling", would I see dentures[1] or pearly whites? [1] https://en.wikipedia.org/wiki/George_Washington%27s_teeth https://en.wikipedia.org/wiki/George_Washington%27s_teeth
  • mattnewton 3y ago
    I think you'd have to provide that out-of-distribution data in the prompt of course - it's not clear these models have built large world models of facts like some of the larger LLMS need to, they are figuring out how things move. Most of the time people have pearly whites to show in the dataset, and there are no videos of Washington's mouth, so I would expect that to be the default unless prompted with a detailed description of the dentures you are looking for.
• blurbleblurble 3y ago
  This is remarkable, it would have been unthinkable 5 years ago.
• abkolan 3y ago
  How soon can Google _Productize_ it?
• ativzzz 3y ago
  Me, watching the video and looking at samples, excitement level high Me, scanning for a download link or a prompt to run the model and not finding any, excitement level medium Me, realizing it's by google, excitement level zero
  • zitterbewegung 3y ago
    Don’t worry OpenAI will copy it and put it in chatgpt
  • baldgeek 3y ago
    Here is the dataset they used: https://paperswithcode.com/dataset/ucf101 https://paperswithcode.com/dataset/ucf101
    • gs17 3y ago
      The paper seems to say UCF101 was used for evaluation, not training.
  • deleted 3y ago
    [deleted]
• Aerbil313 3y ago
  Eh. I knew this day would come. Video is no evidence of nothing now.
• harha_ 3y ago
  This pace of progress almost scares me.
• max_ 3y ago
  Why won't Google publish a product that show does this ?
• wantsanagent 3y ago
  I find it deeply offensive that this work is presented under the auspices of scientific research. The only way to describe this is bragging, advertising, or marketing. There are no reproducible processes described. While the diagram of their architecture may inspire others it does not allow for the most crucial aspect of the scientific endeavor, falsification. There is no way we can know if Google is lying because there's no way to check. It should be assumed that every example has been cherry-picked and post processed. It should be assumed that the data used to train the model (if one was trained at all) was illicitly acquired. We have to start from a mindset of extreme skepticism because Google now routinely makes claims that cannot be demonstrated. When the performance of Gemini in bard is compared to GPT-4 for example, it falls far short. When they release a video claiming to be an interaction with a model it turns out it wasn't anything of the kind. Ideally no organization would operate like this but Google has become a particularly egregious repeat offender.
  • bugglebeetle 3y ago
    > There is no way we can know if Google is lying because there's no way to check it. We can gather that they are likely to be lying or cherry-picking examples to make themselves look better, since they were already caught faking an AI demo. In the world of actual research, if you got caught doing this, all your subsequent and prior work would be under severe scrutiny.
  • GaggiX 3y ago
    >When the performance of Gemini in bard is compared to GPT-4 for example, it falls far short. How did people get access to Gemini Ultra? Or are you talking about Gemini Pro, the one that compares to GPT-3.5?
  • Workaccount2 3y ago
    Just an FYI, it's not illegal to use data to train a model. It's illegal to have a model output that (identical) data for commercial gain. This difference is purposely muddied, but important to understand.
    • leereeves 3y ago
• vessenes 3y ago
  Some comments: Google, so we'll probably never get to use this directly. That said, the idea is very interesting -- train the model to generate a small full-time representation of the video, then upscale on both time and pixels. Essentially, we have seen models adding depth maps. This one adds a 'time map' as another dimension. Coherence is pretty good, to my eye. The jankiness seems to be more about the model deciding what something should 'do' over time, where a lot of models struggle on keeping coherence frame by frame. The big insight from the Googlers is that you could condition / train / generate on coherence as its own thing, then fill in the frames. I think this is likely copyable by any number of the model providers out there; nothing jumps out as not implementable by Stability, for instance.
• Planktonne 3y ago
  "We made Girl with a Pearl Earring smile and wink" demonstrates the fundamental failure of this (and similar) technology: it's the promise of generating art, made by people who really don't understand what art is.
  • AwesomeLemon 3y ago
    Seems like a fun video to generate, not everything has to be snobby "true" art
  • adrenvi 3y ago
    The same was probably said about photography, moving film, film with sound, computer graphics, etc.
    • Planktonne 3y ago
      No it wasn't; absolutely no one ever thought the issue with films with sound is that their creators fundamentally misunderstood Girl with a Pearl Earring. Some people thought that [new medium] wasn't art, they didn't think it was driven by and for people who didn't understand any art. I do enjoy the irony though of you copy-and-pasting a generic pro-AI rebuttal to a comment you didn't understand.
• interestica 3y ago
  Video Inpainting: 4:3 --> 16:9 conversions
• sleight42 3y ago
  How long before generating deep fakes is trivial? This seems many steps closer.
• mangoo84 3y ago
  [dead]