9 ms·
Text2Video-Zero Code and Weights Released by Picsart AI Research (12G VRAM)
- ftxbro 4y agoThe associated paper is "Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators" at https://arxiv.org/abs/2303.13439 https://arxiv.org/abs/2303.13439
- meghan_rain 4y agoWow, scroll to the part where stick figures are provided as guidance https://github.com/Picsart-AI-Research/Text2Video-Zero#text-to-video-with-pose-guidance-1 https://github.com/Picsart-AI-Research/Text2Video-Zero#text-... and tell me this won't upend the VFX industry!
- whitemary 4y agoIt won't upend the VFX industry. If a workflow cannot support iterating on every detail in isolation with psychotic levels of control, it won't even be adopted by the VFX industry.
- nine_k 4y agoIt will help add video illustrations to ordinary presentations or talking-head videos. A market enough for some attention from, say, Adobe, and for a bunch of less expensive offerings.
- Zetobal 4y agoSo, it doesn't upend a multibillion-dollar market because someone who has no idea how the market works made an overly enthusiastic comment on HN? Who would have thought.
- tanseydavid 4y agoGive it a couple of generations...
- deleted 4y ago[deleted]
- echelon 4y agoOnce this tech matures, they won't have any work left to do. Kids at home will be making Star Wars, not Industrial Lights and Magic.
- deleted 4y ago[deleted]
- DrewADesign 4y agoThat's like saying kids at home will be making enterprise software once code generation tools become smooth enough. ChatGPT won't replace novelists, either, no matter how solid its grammar gets. People who don't mind having output that's an aesthetic amalgam of whatever it has already ingested won't mind using these sorts of tools to generate work. For a movie studio that lives and dies on presenting imagery so precisely and thoughtfully crafted that it leaves a lasting impression for decades, I doubt it will be anything more than a tool in the toolkit to smooth things along for the people who've made careers figuring out how to do that. I think there's two reasons this sort of thinking is so prevelant. A) Since developers are only really exposed to the tools-end of other people's professions, they tend to forget that the most important component is the brain that's dreaming up the imagery and ideas to begin with. Art school or learning to be an artist is a lot more about thinking, ideas, analyzing, interpreting, and honing your eye than about using tools... people without those skills who can use amazing generative tools will make smooth, believable looking garbage in no time flat from the comfort of their own living rooms. Great. B) Most people, especially those from primarily STEM backgrounds, don't really understand what art communicates beyond what it physically represents and maybe some intangible vibe or mood. Someone who really knows what they're talking about would probably take longer to accurately describe an existing artistic image than would be reasonable to feed to a prompt. Once again, that will be fine for a lot of people-- even small game studios, for example, that need to generate assets for their 3rd mobile release this month, but it's got miiiiles to go before it's making a movie scene or key game moment that people will remember for decades.
- ttjjtt 4y agoThere are so many well described points in this post. Thank you.
- detrites 4y agoThis is exactly what it supports, and limitlessly. An AI model doesn't get tired of receiving direction and doing retakes. It's a matter of time.
- rvz 4y agoNo it won't. At least not now when we are seeing deformed artefacts in videos like this one: [0] [0] https://old.reddit.com/r/StableDiffusion/comments/1244h2c/will_smith_eating_spaghetti/ https://old.reddit.com/r/StableDiffusion/comments/1244h2c/wi...
- SV_BubbleTime 4y agoFunnier than almost any network TV I’ve seen in years.
- ftxbro 4y agosome call it deformed artefacts some call it meme magic
- rvz 4y agoAs long as 'O̶p̶e̶n̶AI.com' is in last place and loses the race to the bottom, I'm fine with it. Good for anything that is actually open source and doesn't close off research, like Text2Video.
- sockaddr 4y agoI think there's a boob on that camel's hip.
- dragonwriter 4y agoIt will create an entirely new low end to the industry (which may be used in prototyping workflows by the high end), but the high end is going to want both more detailed control of the avatars (I would say models, but that means different things in the colliding domains) and already does skeletal animation with many more control points. You want to upend the VFX industry, you probably want text-to-3d-model and text-for-script-for-existing-animation-systems, that still supports either of those being finetuned manually with existing tools.
- pjonesdotca 4y agoFirst, tweening. If you can get an AI to do the job of a tweener you've already massively impacted the VFX/Animation pipeline.
- CyberDildonics 4y agoThere is no "job of a tweener" in computer animation. Geometry transformations are interpolated and always have been. Where are you getting this idea?
- DrewADesign 4y agoOK... This won't upend the VFX industry. Maybe a subsequent generation that is vastly more precise, but this isn't even ballpark. The stick figure results are vastly less impressive than current methods made with less time and nearly as little effort. Look at Unreal Engine's talk at GDC and then consider that movie studios generally won't use those tools for anything visually important because they just aren't as good as pre-rendered 3D. Even in places like The Mandalorian it was only used for background imagery.
- londons_explore 4y agoSomeone will see these results, and think "if only it was 10x better, I could be rich by selling to the vfx industry". They'll then start training a 10x bigger model (there is plenty of video training data out there). And in 3 months, it will be upending the vfx industry...
- DrewADesign 4y agoOnce again, the hubris of developers when trying to evaluate art just kills me. The most important part of art is deciding exactly what needs to be in the image and how it should look-- not the act of actually creating it. Even using plain language, describing these things in plain English with the precision needed would take way longer than current experts take to make them using existing tools, and the refinement process is full of uncertainty. At most, getting a base layer for people to modify using the professional tools of the trade. Much like development work, the final 10% takes 90% of the time. In dev, you're figuring out how to make it. Usually in VFX, it's figuring out how it should look. That's a fundamentally human process. Whether it accomplishes its goal is evaluated by humans and it needs to satisfy human tastes. No matter how good the tools get at creating images, they just won't be able to speak to human needs and emotions like people will, maybe ever.
- satvikpendem 4y ago> they just won't be able to speak to human needs and emotions like people will, maybe ever. Until the AI knows what you want and like by analyzing your browser and viewing history. There is a part in the Three Body Problem trilogy where the aliens create art that's on par if not better than human-made art, and humans are shocked because the aliens do not even resemble any sort of human intelligence at all.
- reasonabl_human 4y agoControlNet is the new development that will really allow us to guide diffusion model outputs more granularly - first I’m seeing it used against video generation
- mightytravels 4y agoWow - this looks awesome! Love the video2video.
- adeon 4y agoIt kinda works. I tried "kermit doing push ups" and it looks like kermit. And Kermit sorta looks like he is in push up position. Other than that it looks a lot like those early image generation AIs that vaguely look like what you asked but is all mutated. Animation itself is not very good for this prompt. But hey, still pretty good for the early days. Maybe need to figure out how to prompt engineer it and tune it. Seems like it's very heavily based on existing image models? Wondering how easy it is to adapt to other image models. I think I need to read the paper. https://imgur.com/a/h3ciJNn https://imgur.com/a/h3ciJNn
- spokeonawheel 4y agoI tried "spongebob hugging a cactus with his butt hanging out" and it said "error". perhaps I dont understand this tool
- adeon 4y agoYou are welcome: https://imgur.com/a/bjfjhQq https://imgur.com/a/bjfjhQq
- HopenHeyHi 4y agoThanks for the nightmare fuel. And complete absence of butts. :/
- whoomp12342 4y agothat is completely horrific lol
- bunnywantspluto 4y agoCan we run this on a CPU?
- novaRom 4y agoYes, you can run any inference with any model on CPU, but here some examples: To create a single frame with Stable Diffusion 4-5B parameters, 512x512, 20 iterations takes 5-30 minutes depending on your CPU. On any modern GPU it's only 0.1-20 seconds! Similarly with LLMs, to produce one token with a transformer of 30-50 layers 7-12B parameters you will wait several CPU minutes while it takes few seconds on a Pascal-generation GPU and tiny fraction of second on Ampere.
- probablynish 4y agoThe *.cpp adaptations of popular models appear to have optimized them for the CPU, for example llama.cpp and alpaca.cpp let me generate several tokens in a matter of seconds.
- gpderetta 4y ago> To create a single frame with Stable Diffusion 4-5B parameters, 512x512, 20 iterations takes 5-30 minutes depending on your CPU Depends a lot on the cpu. Are you specifically talking about Text2Video or SD in general? IIRC, last time I tried SD on my CPU (10 core 10850k, not exactly cutting edge) it did take less than one minute for more than 20 iters. This was about 4-5 months ago, things might have gotten better. The GPU (even a vintage 1070) was faster still of course.
- tjoff 4y agoThat is much better than expected, depending on how easy it is to setup it is very much worthwhile if you don't have access to a decent GPU.
- dcreater 4y agoCan this get the ggerganov treatment so that I can run it on Apple silicon?
- crakenzak 4y agowith llama.cpp getting GPT4all inference support the same day it came out, I feel like llama.cpp might soon become a general purpose high performance inference library/toolkit. excited!
- ggerganov 4y agoHeh, things seem to be moving in this direction, but I think it's still a very long way to go. But who knows - the amount of contributions to the project keep growing. I guess when we have a solid foundation for LLM inference we can think about supporting SD as well
- reasonabl_human 4y agoThanks for your work in this space, it’s incredible!
- dcreater 4y agoAbsolutely, its just unbelievable what this one person has enabled. ChatGPT isn't mind blowing as much as ggerganov is.
- Labo333 4y agoSD models are much smaller so you can probably already run it with the github code. Image models work well at least: https://huggingface.co/docs/diffusers/optimization/mps https://huggingface.co/docs/diffusers/optimization/mps It is even possible to run them in the browser: https://stablediffusionweb.com/ https://stablediffusionweb.com/
- patientplatypus 4y ago[dead]
- bdcravens 4y agoI entered "A dachshund doing a backflip" It was a dachshund. Doing some sort of a dance in the air. Nothing resembling a flip. https://imgur.com/a/eqHpuo7 https://imgur.com/a/eqHpuo7
- leereeves 4y agoThe examples on GitHub are so much better than the examples here. I wonder if the authors are cherry picking or we just don't know how to get good results. Why are people downvoting this? The examples on GitHub are clearly doing the action described (unlike the dachsund above) and don't have the wild deformations that the "Spongebob hugging a cactus" or "Will Smith eating spaghetti" animations have.
- gremlinsinc 4y agoit only matters to the inpatient among us, give it a few weeks and things will only get better. The pace of ai innovation is frightening but exciting too.
- nullsense 4y agoThey're probably cherry picking. I know on ModelScope text2video I have to generate 10 clips per prompt to get 2 or 3 usable clips and out of that will come one really good one. But it's fast enough to do that I just sort of accept that as the workflow and generate in batches of 10. I assume it's likely the same for this. Can't wait to try it though.
- throw101010 4y agoIn my experience AI generative processes always involve a lot of cherry picking, rarely have I generated 1 image or 1 text and it was perfect or very representative of what can be attained with some refining. Seems fair to choose good results when you try to demonstrate what the software "can" do. Maybe a mention of the process often being iterative would be the most honest, but I think anyone familiar with these tools assume this by now.
- godelski 4y agoThis is, unfortunately, standard practice on generative papers. Best is to show cherry picked and random selection. But if you don't cherry pick you don't get published. Even having random samples can put you in a tough position. To be fair, metrics for vision are limited and many don't know how to correctly inturpret them. You both want to know what's the best images the network can produce a well as the average. Reviewers, unfortunately, just want to reject (no incentive you accept). That said, it can still be a useful tool because you know the best (or near) that the model can produce. This is also why platforms like HuggingFace are so critical. I do think we should ask more of the research community but just know that this is in fact normal and even standard. Most research judge by things like citations and just reading the works themselves. The more hype in a research area the more noisy the signal for good research. (This is good research imo fwiw)
- voxelghost 4y agowhat a time to be alive!
- RajSinghLA 4y agoDear fellow scholars
- crakenzak 4y agothis is two minute papers, with Dr. Karoly Zsolnai-Feher
- detrites 4y agoFor the adventurous among us, here's that boolean you might want to change: https://github.com/Picsart-AI-Research/Text2Video-Zero/blob/main/text_to_video/text_to_video_pipeline.py#L45 https://github.com/Picsart-AI-Research/Text2Video-Zero/blob/...
- stevefan1999 4y agoCome on, somebody please make this an environmental variable
- totetsu 4y agoThis is exactly the kind of thing the Taoist philosophers were warning us about.
- belter 4y agoYou are the wind beneath my wings...
- novaRom 4y agoIs there any estimate how much training data someone like MindGeek has?
- astrange 4y agoMore important question is who has the annotation budget.
- SentinelLdnma 4y agoEconomists will ponder for decades why LLMs never increased labor productivity.
- FractalHQ 4y agoWhat was it? That link returns a 404 now.
- rottc0dd 4y agoThis should help: https://web.archive.org/web/20230329124453/https://github.com/Picsart-AI-Research/Text2Video-Zero/blob/main/text_to_video/text_to_video_pipeline.py https://web.archive.org/web/20230329124453/https://github.co... Check line 45
- nprateem 4y agoSo long giphy
- evolveyourmind 4y agoSeems very limited. I wonder if the same can be achieved with just stable diffusion and neighbor latent walks with very small steps. On the other hand the interpolation techniques with the GigaGAN txt2img produce much higher quality “videos” than this
- valine 4y agoRight, stable diffusion and control net depth network could give very good results if you have a source video.
- deleted 4y ago[deleted]
- mnemotronic 4y ago[edit] _Almost_ everything I try except the predefined examples returns "error". "a pencil with wings" returns something nothing like a pencil that does, in fact, have wings. There will be a learning curve.
- rasz 4y agoat least two of the example dogs have 5 paws
- rexreed 4y agoQuestion about models and weights. When an organization says they release the weights, how is that different from when an organization releases a model, say Whisper from OpenAI? In what way are the model releases different in these cases?
- superb-owl 4y agoThe model is just an untrained architecture. You'd need to spend a lot of money (a) gathering data, and (b) running GPUs to train it. The weights are the fruition of training. They make the model actually useful.
- rexreed 4y agoWhen you download a trained model for use by Python, I'm assuming the model contains both the architecture (the neural net or even a boosted tree) as well as the weights / tree structure that makes the model actually usable in inference. When organizations release a trained model, I'm assuming that the weights are necessary to make use of that model? If not, then are they not really releasing the model, but just the architecture and training data?
- freeone3000 4y agoUsually not the training data either!
- vdfs 4y agoAs i understand it: - Model is the code - Data is the text/images used for traing - Weights are the training results For example Lucene, models will be the java library, data is text data like wikipedia and weights are the lucene index. if you have all the 3 you can start searching right away, if you have model+data you have to generate the index which can take a lot of time, training/indexing take more than searching or using the model. if you have just he model you need to get your own data and run training on it
- leodriesch 4y agoOpenAI also released the weights for Whisper. Some model releases, like LLaMa from Meta, just contain the code for training and the data. You can train the weights yourself, but for LLaMa that takes multiple weeks on very expensive hardware. (LLaMa also released the weights for researchers and they leaked online, but the weights are not open-source.) When a company releases a model including the weights, you can download their pre-trained weights and run inference with them, without having to train by yourself.
- lifeisstillgood 4y agoWeirdly Corridor digital had a AI generated video and they suffered from what is slightly happening here - the image of the bear / panda or whatever is a different animal each time (ie it's a panda, just a different one "hallucinated" each frame. corridor digital handled it by training their model on specific images of specific people - and so they effectively said "video of the panda called phil that we have trained you on images of" Clearly this is not possible here - so I am missing how they got it close
- jonas-w 4y agoI see that there is a 12GB VRAM requirement. Can my 6GB GPU do anything to at least provide some performance advantages as instead of running it entirely on the CPU?
- EGreg 4y agoI want to output an animated GIF. What is the command? Do I run pix2pix at the end? Also can I somehow have more frames and set the GIF speed?
- tehsauce 4y agoI haven't seen a single example from this model that demonstrates video with any time of time continuity. It appears every frame is independent to each other.
- mlboss 4y agoPose based controlnet looks like it demonstrates continuity
- fauxpause_ 4y agoI feel like they really missed an opportunity to make their example Horse in Motion rather than Horse Galloping on a Street