3 ms·

That’s because Brad Pitt is a named celebrity with lots of photos from many angles that has been included by name in the training data. It can get Brad Pitt lik

That’s because Brad Pitt is a named celebrity with lots of photos from many angles that has been included by name in the training data. It can get Brad Pitt like content by name. If I say “5 foot 6 inch tall older Italian woman <insert rest of scene description here>” I’m going to get an absolutely randomised person as the focus each time, the mode has no memory and this has been the BIGGEST limitation in all these transformer based architectures. There’s no memory of each output, they are all discrete and as much as you can kinda fudge it with GPT-3 by including part of the previous paragraph or sentences in your prompt to try and keep it on track, it can, does, and inevitably will drift horribly off track unless you manually coach it over and over micromanaging it’s output to keep it useful and sensible. With GPT it’s each text prompt, with DALL-E it’s each text or image prompt, and with a hypothetical video version it would be the same limit, each prompt has no memory of the previous and so it can’t implicitly reuse anything created by a previous prompt. You have to hack that in sound the core transformer architecture to make a facsimile of memory by things like pasting in extra data or teaching DALL-E 2 to edit a picture you upload and then uploading the picture you just generated using it, video would be no different and consequently this approach will not scale efficiency beyond individual items of content like an arbitrary sized block of text, a single photo or small group of photo edits, or a single “shot” of video content containing a scene or two worth of contiguous narrative flow.

dougabug 5y ago
To create specific people and spatiotemporal consistent content, we would use conditional image generation where we condition on both the text and some seed vector, and proceed autoregressively. If we wanted a random person, we would use a random seed vector. If we wanted a specific person, we would use 3D GAN Inversion to find the seed vector which would give us the person in a provided query image. We would also likely condition the latent space so we could retarget facial expressions from one person to another by manipulating the differences between latent codes.
wrycoder 5y ago
You are reminding me of the Many Worlds interpretation of quantum mechanics.
notahacker 5y ago
My point isn't that the procedurally generated woman is as easy for the AI to conceive in many contexts as Brad, but that there are ways of explicitly reusing parts of images and combining an actor, a hat and a location in a novel but acceptable way is a tough challenge text prompt driven neural networks appear to now perform well at. Manually tagging items in each frame and requesting that they're in the next frame with text prompts doesn't scale, but adding a degree of automation to classifying parts of a frame and updating the data corpus and text prompt for the next frame doesn't seem an insurmountable problem. Many existing CGI techniques don't scale well (even human CGI artists working on Hollywood movies expect instructions on frame composition far more explicit than a director's comments and storyboard; I had a flatmate who was that translation layer). I don't see us getting coherent cinematography from feeding it Hollywood scripts soon if ever and agree there will be significant uncanny valley issues with human mannerisms, but we can do a lot of convincing and selective creative work with text prompts that we couldn't until recently. Especially in combination with other tools, so if DALL-E's concept of geometry is nowhere near capable of generating the side profile of procedurally generated 5'6" Italian woman, we create the woman in a specialised tool for 3D procedural face generation, and feed DALL-E as many images with different angles and lighting as it possesses of Brad. I'd agree we're a long way from doing without human curation and letting the neural networks do all the heavy lifting, but it looks like its now a curation and classification problem rather than a "neural networks can't discern that" problem.

dougabug 5y ago

To create specific people and spatiotemporal consistent content, we would use conditional image generation where we condition on both the text and some seed vector, and proceed autoregressively. If we wanted a random person, we would use a random seed vector. If we wanted a specific person, we would use 3D GAN Inversion to find the seed vector which would give us the person in a provided query image. We would also likely condition the latent space so we could retarget facial expressions from one person to another by manipulating the differences between latent codes.

wrycoder 5y ago

You are reminding me of the Many Worlds interpretation of quantum mechanics.

notahacker 5y ago

My point isn't that the procedurally generated woman is as easy for the AI to conceive in many contexts as Brad, but that there are ways of explicitly reusing parts of images and combining an actor, a hat and a location in a novel but acceptable way is a tough challenge text prompt driven neural networks appear to now perform well at. Manually tagging items in each frame and requesting that they're in the next frame with text prompts doesn't scale, but adding a degree of automation to classifying parts of a frame and updating the data corpus and text prompt for the next frame doesn't seem an insurmountable problem. Many existing CGI techniques don't scale well (even human CGI artists working on Hollywood movies expect instructions on frame composition far more explicit than a director's comments and storyboard; I had a flatmate who was that translation layer). I don't see us getting coherent cinematography from feeding it Hollywood scripts soon if ever and agree there will be significant uncanny valley issues with human mannerisms, but we can do a lot of convincing and selective creative work with text prompts that we couldn't until recently. Especially in combination with other tools, so if DALL-E's concept of geometry is nowhere near capable of generating the side profile of procedurally generated 5'6" Italian woman, we create the woman in a specialised tool for 3D procedural face generation, and feed DALL-E as many images with different angles and lighting as it possesses of Brad. I'd agree we're a long way from doing without human curation and letting the neural networks do all the heavy lifting, but it looks like its now a curation and classification problem rather than a "neural networks can't discern that" problem.