7 ms·
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
- Fauntleroy 4y agoA bit ridiculous seeing the much-maligned "trending on artstation" keywords on nVidia's own tech demo page.
- joegibbs 4y agoWhat's the problem with "trending on artstation"? Does it not do anything to the output?
- wokwokwok 4y agoCome on, it’s like slapping a doge sticker on your research paper. I think prompting is a bit past “trending on Artstation in the style of Greg Rutkowski” at this point..?
- nonbirithm 4y agoIt implies that the researchers are okay with generating outputs based on data scraped from ArtStation portfolios without consent. There was also the time months ago when the art of a deceased artist (Qinni) was featured prominently on the front cover of an img2img/style transfer paper until the artist's sister requested it was taken down. https://twitter.com/ZirocketArt/status/1629557296563888128 https://twitter.com/ZirocketArt/status/1629557296563888128
- deleted 4y ago[deleted]
- robarr 4y agoI understand the technical marvel, but there are obvious inconsistencies in all the images and I would call them more animations than video sequences.
- chinaman425 4y ago[dead]
- echelon 4y agoThe pace of these is improving at a remarkable clip. I remember when image generation was similarly derided. I wouldn't count anything out at this point. (Also, this was an intern project. There are a couple of senior staff researchers on the paper, but it's mostly interns.)
- vrglvrglvrgl 4y ago[dead]
- marapuru 4y agoThe generated 'driving scene' videos eerily resemble a lot of crash dashcam video's. I even have the idea that the bottom right video is composited of video's right before a crash. The white SUV seems to make a hard steer right, but then just continues straight on. It even looks as if some wreckage is visible flying over the road.
- jonititan 4y agoThat's probably what they were trained on. Gotta be a lot of them that have been posted online.
- 152334H 4y agoNo model, no code. About as exciting as Imagen-video, which was released eons ago.
- ipsum2 4y agoThe approach is different, it can use pretrained models, i.e. stable diffusion, which is a pretty exciting research development. This means that it only requires 'fine-tuning' existing models to get this result.
- 152334H 4y agoI agree with that, but it's hard for me to get excited with the knowledge that it'll almost certainly be discarded and forgotten. I've seen too many papers that looked interesting from a theoretical perspective, but were simply never brought to the public because of the barrier of dev+training. In this case, you need someone that can implement the method as described (hard!), and then you need someone with a setup better than a rented 8xA100 (expensive and not available on many cloud providers) to actually reproduce the model.
- ipsum2 4y agoTo put it in context, in almost all areas of research (physics, biology, chemistry, electronics, etc), running experiments is expensive. ML is in the category that there can still be advances done by amateurs at home. I don't think it's worth writing off everything that requires more resources than a hobbyist.
- alwayslikethis 4y agoYeah. You can rent 8xA100s, it's a steep price but you can. It's much harder for a hobbyist to rent electron microscopes or NMR machines.
- p-e-w 4y agoI doubt you'll ever again see a model being publicly released for something as capable as this. I can't say I fully understand the mechanisms by which they achieve that, but it's clear that the powers that be have decided that the public cannot be trusted with powerful AI models. Stable Diffusion and LLaMA were mistakes that won't be repeated anytime soon.
- j45 4y agoIt's interesting the generative video attempts aren't starting with something simpler and and growing from there with the audience. No model, no code makes it a challenge to explore.
- ShamelessC 4y ago> † Andreas, Robin and Tim did the work during internships at NVIDIA. Some pretty experienced/expensive interns you got there, NVIDIA.
- atleastoptimal 4y agoWhenever I see these text to video diffusion models, I always think that a much better result would come out of some sort of multi-modal vision-aware AI agent were able to use 3d modeling tools like blender, continuously iterate it, work with physics and particle simulation tools then refine the less realistic details with a diffusion model based filter. It would also allow all the assets created along the way to be reused and refined. I imagine it will be the de-facto way AI generates video in the next few years, once huge models are smart enough to use tools and work in high breadth and depth jobs like humans do.
- red75prime 4y agoI concur. Throwing in more and more compute to make digital dreams more and more realistic doesn't seem to be an efficient strategy for video generation where you need to keep lots of out-of-the-picture context in memory.
- jiggawatts 4y agoThis actually exists, already... sort-of. A major topic of research is to make differentiable renderers, which can then have their parameters tuned via gradient descent, much like how AIs are trained. The problem is that most renderers are very discrete, and hence aren't differentiable. For the same reason that reverse rendering is difficult on such systems, it's hard to train an AI to use them also. Having said that, I can imagine it happening. Look at Google's Alpha Star and related AIs, which can play games with discrete events at grandmaster level. You could train an AI to operate a 3D design program similarly. The input training could be recordings of actions taken by human designers, or simply train the AI by making it match target images. Start out simple, and then crank up the complexity over time.
- Maken 4y agoDifferentiable renderers have started to volumetric representations and distance fields to represent geometry, which kinda solves the discretization of the scene.
- Closi 4y ago
- hasa 4y agoThat guitar teddybear sample clearly played Stevie Ray Vaughan's "Scuttle Buttin'". Good taste for music!
- Lerc 4y agoIt's a bit light on details beyond trained-on-video, and interpolating between keyframes in latent space. The stable diffusion community is getting impressive results generating keyframes in a batch and using existing interpolation mechanisms on the decoded frames. https://www.reddit.com/r/StableDiffusion/comments/12o8qm3/finally_installed_the_newer_controlnet_models_a/ https://www.reddit.com/r/StableDiffusion/comments/12o8qm3/fi... That example is using control-nets to induce the animation they want but it would be quite easy to train a model on a sequence of video frames in the same layout.
- atorodius 4y agoHm? the paper is actually fairly detailed for an industry paper. They are able to freeze much of the model and only train temporal layers inbetween.
- blauditore 4y agoThe results are impressive, but also serious nightmare fuel. I suppose that's a good example of the "uncanny valley": https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley
- melbourne_mat 4y agoI've seen the phrase "uncanny valley" thrown around a bunch of times. Who are these people who have emotional reactions to CGI which is not quite real looking?
- capableweb 4y agoEmotional reaction doesnt just mean crying or laughing, the feeling of "that looks unreal" is a emotional reaction too and that's what's being referred to here.
- Nevermark 4y agoFor many millennia CGI has had difficulty creating anything that looked real. But each advance has been an opportunity to unnerve people as they react to how much more real something looks, but still isn’t real. I think the uncanny valley is now getting trained out of us. We are all getting used to a continuum of real to stylistically unreal, with no more unexplored valley. Except for the weird mistakes. Those will likely remain weird to us since most of us don’t want to immerse ourselves in worlds of uncountable fingers and third arms for long enough for that to start feeling normal.
- taneq 4y agoAlmost-real-looking CGI is one thing. I find some of these AI images to be far creepier because they look so plausible, but then the longer you look the worse it gets. Pretty girl on a beach at sunset! But there's a hand resting on her waist, and she's alone. And she has six fingers on one hand. And one foot is the wrong way around. And then you realise that she has too many teeth and they go past the end of her lips... and one is poking out from one of her eyes. And then it gets weirder.
- numpad0 4y ago
- deleted 4y ago[deleted]
- cornedor 4y agoA lot of those examples have a Shutterstock watermark. I doubt that Shutterstock allows the use of unlicensed videos for this use case.
- goodells 4y agoI’m also now mildly disturbed/worried by the idea of going about my business in AI-generated VR a few years from now (maybe with some 2D-to-3D-ifying compatibility layer) and being haunted by ghosts of stock image watermarks.
- donkeyd 4y agoI noticed the same thing. I'm not surprised about the current discussion over copyright because this training data pretty clearly has a basis in unlicensed Shutterstock images.
- riotnrrd 4y agoThey are licensed: https://nvidianews.nvidia.com/news/shutterstock-teams-with-nvidia-to-build-ai-foundation-models-for-generative-3d-artist-tools https://nvidianews.nvidia.com/news/shutterstock-teams-with-n...
- in3d 4y agoIf they posted these videos on publicly accessible websites, it could be fair use. We need some more court cases to really know.
- dragonwriter 4y ago> A lot of those examples have a Shutterstock watermark. I doubt that Shutterstock allows the use of unlicensed videos for this use case. That almost everyone (other than, maybe, people specifically and very loudly selling the fact that they aren’t, e.g., Adobe) training base AI models is relying on the idea that doing so is fair use and doesn’t require a license is hardly news.
- knocknock 4y agohttps://nvidianews.nvidia.com/news/shutterstock-teams-with-nvidia-to-build-ai-foundation-models-for-generative-3d-artist-tools https://nvidianews.nvidia.com/news/shutterstock-teams-with-n...
- Aissen 4y agoI was curious about the use of "sks" in the examples. It is explained in the paper: > Using DreamBooth [66], we fine-tune our Stable Diffusion spatial backbone on small sets of images of certain objects, tying their identity to a rare text token (“sks”). I wonder how long that token will stay "rare".
- astrange 4y agoDreamBooth replaces the old meaning of the token in a model, so the only importance of "rareness" is you weren't already using it.
- Rakib_7 4y agoI think their main focus is generate more real lift simulation of driving and creating many scenarios to train on. Its good for the autonomous car mostly targeting Cruise and Tesla.
- usrusr 4y agoAn interesting take on the "diffusion can't do hands" trope: take a teddy bear (the most finger-less humanoid shape known to man, and now also to machine I guess) and put it in that one scene where all attention is on fingers (guitar solo)
- Yenrabbit 4y agoSome things to note: - This was finished last year, impressive now but even more so back then - No code or models released BUT several authors have moved to StabilityAI and are working on their own improved open video models, which is hopeful as the field continues to move forward - The paper uses existing image models as a base, and so a better base model (the new XL stable diffusion variant, or Midjourney's underlying model) will give even better results.
- bighoki2885000 4y ago[dead]
- bighoki2885000 4y ago[dead]
- GaggiX 4y agoI give a summary about the training of the text-to-video model: they used 48 GPUs for the keyframes model 128k steps, 24 GPUs for the interpolation model 14k steps, 32 GPUs for the upscaler model 10k steps, 40 GPUs for the autoencoder 35k steps. They used A100-80G in all cases except the autoencoder (A100-40G). This is not particularly expensive considering the numbers these companies usually use, thanks to the fact that they only trained the temporal layers in the LDM and the autoencoder.
- m0llusk 4y agoThis seemed like it might be an interesting article to read, but it turns out that clicking on this link initiated a huge many megabyte download of some kind of media experience that totally locked up my machine. I had a bunch of apps going and was hoping for a smooth start to the day. Had this been an actual article to read then it could have loaded and been put in the background. If you are going to link to some hypermedia experience that requires a high end machine, good net connection, and well configured ad blockers then there should be a warning. This is not some taste issue about me not liking the custom font or whatever. Much of the population does their work on lower end machines with modest net connections and this site is not compatible with that. Do you really want HN to be a site only for the ascended? If you can't communicate without an avalanche of crud coming along then maybe you need to consider what you are saying and maybe consult an editor or something.
- atorodius 4y agoWat. It‘s a demo for a text to video model, of course the authors would show videos, expecting anything else is wild. It should be on your machine to handle this gracefully…
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- lelag 4y agoThe fox dressed in suit dancing in park is obviously Roald Dahl's Fantastic Mr Fox and his tail was just shot by the farmers it seems.