3 ms·
ML models used in this way will struggle with scene to scene consistency... For example, when you generate someone riding a horse, and later want the same char
by londons_explore 5y ago
ML models used in this way will struggle with scene to scene consistency...
For example, when you generate someone riding a horse, and later want the same character to be at home drinking a coffee. It'll be hard to get the model to render the same face, mannerisms and style for the same person in both cases.
I think that alone will take years to properly solve.
- dougabug 5y agoNot so much. Check out E3GAN from Gordon Wetzstein’s lab at Stanford: https://matthew-a-chan.github.io/EG3D/ https://matthew-a-chan.github.io/EG3D/. In particular, note how they are able to apply a process called GAN Inversion to find a point in the 3D GAN’s representation space to generate a 3D model of a person’s face (one of the collaborators) from a single query image.
- londons_explore 5y agoBut it's much trickier to find the point in a GAN's representation space for "the exact same fluffy pink earrings that the actress was wearing in the last scene". Mostly because one can't write code to judge equal-ness of pink earrings. Or the fact the trash can in the background had a lid in the last scene and is now missing the lid. Or the fact the brick wall has turned into a painted wall. etc. Having said that, I'm sure some filmmakers will go for it anyway - after all, human-made films have some consistency issues, and many viewers simply don't care.
- notahacker 5y agoI think the main thing is it becomes a programming problem rather than an AI limitation. From what I've seen it's now possible for the AI to generate EarringsAsset57 and KitchenWall51 and then to ask the AI to draw the actor wearing EaringsAsset57 in front of Kitchenwall51 and for stills at least, it can produce acceptable results
- dougabug 5y agoYou can decompose an entire scene into a collection of 3D GANs, each of which is fit by GAN Inversion to the object instance which best explains what is seen in the images. Also, we absolutely can tell the difference between trash cans with and without lids, as well as brick walls vs painted walls. This falls under many areas of ML-based Computer Vision, such as metric, contrastive, and zero shot machine learning, also open world object detection / recognition. These frame level semantic discrepancies can be detected by discriminative DNNs, which we can use to provide a temporal consistency loss to guide the generator to produce temporally consistent videos.
- webmaven 5y ago> But it's much trickier to find the point in a GAN's representation space for "the exact same fluffy pink earrings that the actress was wearing in the last scene". Not so much, since you can supply an image of the desired item and get back a bunch of vectors to reuse.
- tomatowurst 5y agoSeems like this will also get to a point where we would just take a photo and animate it completely as a 3D mesh. Like that picture, just on a few photos, at some point, would generate a photorealistic, ray-traced 3D asset and also a fully interactable 2D motion paired with GPT-4 or 5 where you could have indistinguishable level of human like interaction (ex. Fable studio) After a couple dozen more papers and prototypes, we could have ultrasound haptic feedback and some 3D holographic projector where we achieve a true augmented reality. Going a bit further with this idea, you could essentially create augmented 3D objects that would be indistinguishable from the real thing until you apply enough force enough to break through some upper limit of ultrasound generated haptic pressure (as to not hurt the end user through some legislative regulation). For example, opposing magnets that can be pushed together past a certain point where the illusion of tactility is gone. What would a reality like that be like? Where you could not only have limitless content media generator but you could conjure up those assets in your living room and have them interact kinetically with the real world. We are talking 3D holographic assets with ultrasound haptic simulator that can apply pressure to any object from all directions. https://www.youtube.com/watch?v=fwTcfwbrNO4 https://www.youtube.com/watch?v=fwTcfwbrNO4 https://www.youtube.com/watch?v=jDRTghGZ7XU https://www.youtube.com/watch?v=jDRTghGZ7XU We might not even need expensive robots in the future. Just conjure up a 3D human via some ultrasound/holograph projector which can apply enough force to objects.
- ben_w 5y agoFor film-length consistency, I wouldn’t be surprised by it taking years to solve, despite what @dougabug says in a sibling comment. But “years” isn’t very much time. (I have the mental image of it being as easy to make a film as it was for the character of Mariner in that episode of Star Trek Lower Decks where she makes a filmified version of the ship, in the holodeck, in a few seconds; but the more I think about it, the more I realise I think I don’t feel the normal drives for expensive economic signalling in a way that would allow me to actually predict what people would do given the technology).
- l33tman 5y agoYou need to loop back the generated output to provide context for the next output, just like how GPT-3 works. This at least establishes the needed data to train for consistent outputs. Given a deep well-trained network, the high-level meta-data that needs to be consistent across scenes is just a small set and the network will discover this just as GPT-3 did.
- dougabug 5y agoRight. In the literature this is typically referred to as autoregressive sequence generation, and you can use it generate a whole story consistent with a given prompt (of any length), or even a whole image where you give it the first N seed pixels.
- notahacker 5y agoConsistency in stills seems to be solvable with the current generation models. You can now get an AI to draw Brad Pitt on a horse wearing a Stetson and Brad Pitt in Central Perk drinking a coffee with the right source data, or move a subject into a different setting by tweaking text descriptions enough, and it seems they often get details like shadows and background borders right. I think it'll struggle enormously with subtle details of mannerism in film and coherently fleshing out stuff where it hasn't got a lot of data in its database (getting Brad Pitt or New York from multiple angles is a lot easier than getting a consistent thisactordoesnotexist or novel fantasy setting). But I think the bigger barrier to adoption outside CGI studios (for producing content above meme-level quality) is that actually describing enough details to get everything to cohere even in static images will be beyond the average person.
- techdragon 5y agoThat’s because Brad Pitt is a named celebrity with lots of photos from many angles that has been included by name in the training data. It can get Brad Pitt like content by name. If I say “5 foot 6 inch tall older Italian woman <insert rest of scene description here>” I’m going to get an absolutely randomised person as the focus each time, the mode has no memory and this has been the BIGGEST limitation in all these transformer based architectures. There’s no memory of each output, they are all discrete and as much as you can kinda fudge it with GPT-3 by including part of the previous paragraph or sentences in your prompt to try and keep it on track, it can, does, and inevitably will drift horribly off track unless you manually coach it over and over micromanaging it’s output to keep it useful and sensible. With GPT it’s each text prompt, with DALL-E it’s each text or image prompt, and with a hypothetical video version it would be the same limit, each prompt has no memory of the previous and so it can’t implicitly reuse anything created by a previous prompt. You have to hack that in sound the core transformer architecture to make a facsimile of memory by things like pasting in extra data or teaching DALL-E 2 to edit a picture you upload and then uploading the picture you just generated using it, video would be no different and consequently this approach will not scale efficiency beyond individual items of content like an arbitrary sized block of text, a single photo or small group of photo edits, or a single “shot” of video content containing a scene or two worth of contiguous narrative flow.