9 ms·

The state of AI for hand-drawn animation inbetweening

262 points by luu 3y ago

Olesya000 3y ago
[dead]
shaileshm 3y ago
Great article! This is one of the most overlooked problems in generative AI. It seems so trivial, but in fact, it is quite difficult. The difficulty arises because of the non-linearity that is expected in any natural motion. In fact, the author has highlighted all the possible difficulties of this problem in a much better manner. I started with some simple implementation by trying to move segments around the image using some segmentation mask + ROI. That strategy didn't work out, probably because of some mathematical bug or data insufficiency data. I suspect the later. The whole idea was to draw a segmentation mask on the target image, then draw lines that represent motion and give options to insert keyframes for the lines. Imagine you are drawing a curve from A to A. You divide the curve into A, A_1, A_2... B. Now, given the input of segmentation mask, motion curve, and whole image, we train some model to only move the ROI according to the motion curve and keyframe. The problem with this approach is in sampling the keyframe and matching consistencies --making sure RoI represents same object-- across subsequent keyframes. If we are able to solve some form of consistency, this method might be able to give enough constraints to generate viable results.
- yosefk 3y ago
  I currently shelved 3K more words of why it's hard if you're targeting real animators. One point is that human inbetweeners get "spacing charts" showing how much each part should move, even though they understand motion very well, because the key animator wants to control the acting
xrd 3y ago
This was really fun. It captured a lot of thinking on a topic I've been interested in for a while as well. The discussion about converting to a vector format was an interesting diversion. I've been experimenting with using potrace from inkscape to migrate raster images into SVG and then use animation libraries inside the browser to morph them, and this idea seems like it shares some concepts. One of my favorite films is A Scanner Darkly, and that used a technique called rotoscoping which I recall was a combination of hand tracing animation and computers then augmenting it, or vice versa. It sounded similar. The Wikipedia page talks about the director Richard Linklater and also the MIT professor Bob Sabiston who pioneered that derivative digital technique. It was fun to read that. https://en.m.wikipedia.org/wiki/Rotoscoping https://en.m.wikipedia.org/wiki/Rotoscoping https://en.m.wikipedia.org/wiki/Bob_Sabiston https://en.m.wikipedia.org/wiki/Bob_Sabiston
- whywhywhywhy 3y ago
  > and that used a technique called rotoscoping Technically it's "interpolated rotoscoping" using a custom tool called Rotoshop, which takes vector shapes drawn over footage then smoothly animates between the frames giving a distinct dream-like look to it. Rotoscoping is where you work to a traditional animation framerate drawing over live action but each frame is a new drawing and doesn't have the signature shimmery look Scanner Darkly and Waking Life so I think it's worth pointing out the distinction. https://en.wikipedia.org/wiki/Rotoshop https://en.wikipedia.org/wiki/Rotoshop
kleiba 3y ago
A blog post from the same guy that used to maintain the C++ FQA! https://yosefk.com/c++fqa/ https://yosefk.com/c++fqa/
- AlexandrB 3y ago
  One of my favourite bits of content written about C++. Highly recommended.
nicklecompte 3y ago
Great read - I learned quite a bit about this. The only quibble I had is at the end, and it's a very general layperson's view: > But what’s even more impressive - extremely impressive - is that the system decided that the body would go up before going back down between these two poses! (Which is why it’s having trouble with the right arm in the first place! A feature matching system wouldn’t have this problem, because it wouldn’t realize that in the middle position, the body would go up, and the right arm would have to be somewhere. Struggling with things not visible in either input keyframe is a good problem to have - it’s evidence of knowing these things exist, which demonstrates quite the capabilities!).... This system clearly learned a lot about three-dimensional real-world movement behind the 2D images it’s asked to interpolate between. I think that's an awfully strong conclusion to draw from this paper - the authors certainly don't make that claim. The "null hypothesis" should be that most generative video models have a ton of yoga instruction videos shot very similarly to the example shown, and here the AI is simply repeating similar frames from similar videos. Since this most likely wouldn't generalize to yoga videos shot at a skew angle, it's hard to conclude that the system learned anything about 3D real-world movement. Maybe it did! But the study authors didn't come to that conclusion, and since their technique is actually model-independent, they wouldn't be in a good position to check for data contamination / overfitting / etc. The authors seem to think the value of their work is that generative video AI is by default past->future but you can do future->past without changing the underlying model, and use that to smooth out issues in interpolation. I just don't think there's any rational basis for generalizing this to understanding 3D space itself. This isn't a criticism of the paper - the work seems clever but the paper is not very detailed and they haven't released the code yet. And my complaint is only a minor editorial comment on an otherwise excellent writeup. But I am wondering if the author might have been bedazzled by a few impressive results.
- yosefk 3y ago
  You're technically correct, there's no basis to argue that a "3D representation" was learned as opposed to "enough 2D projections to handle the inputs in question." That said, the hand which did not exist in either of the original 2D frames makes an appearance. I think calling wherever it was pulled out of "the 3rd dimension" is not wrong; it was occluded in both 2D inputs and functionally you had to know about its existence in the 3rd dimension to show it even if you technically did it by learning how pixels look across frames. You can also see much more 3D ish things in the paper, with 2 angles of a room and video created moving the camera between them. Of course in some sense it adds to my point without detracting from yours...
chris_st 3y ago
Not sure who would fund this research? Perhaps the Procreate Dreams folks [0]. I'm sure they'd love to have a functional auto-tween feature. 0: https://procreate.com/dreams https://procreate.com/dreams
aredox 3y ago
Now that's a great use for AI! Inbetweening has always been a thankless job usually outsources to sweatshop-like studios in Vietnam and - even recently - North Korea (Guy Delisle's "Pyongyang" comic is about his experience as a studio liaison there). And AI has less room to hallucinate - it is more a kind of interpolation - even if in this short curt example, the AI still "morphs" instead of cleanly transitioning. The real animation work and talent is in keyframes, not in the inbetweening.
- yosefk 3y ago
  Actually inbetweening is really hard (and I think requires talent) and used to be a big way to learn enough to become a key animator. And I would worry about AI eliminating this learning route if classical animation wasn't struggling to survive at all
  - aredox 3y ago
    It's hard like translating novels - you have to match someone else's style, which is why it's thankless. I don't know if that's really a good pathway to become a key animator - how many inbetweeners are there for one key animator?
- commieneko 3y ago
  As an animator for 40 plus years, I can tell you that in-betweening is a very difficult job. The fact that it's often cheaply outsourced is more of a factor that the people paying for the animation simply don't care about the quality. The results are seldom good. As to how much poor quality in-betweening hurts the performance to the audience is a complicated discussion. Animation that is _very_ bad can often be well accepted if other factors compensate (voice acting, design, direction, etc.) A good in-betweener is not simply interpolating between the keys. For hand drawn animation at least, there's a lot more going on than that. We'll leave out any discussion of breakdowns here. For one it's a difficult concept, much more difficult than 'tweening to explain. The other is that different animators will give different opinions on what a breakdown is or does. I will say, though, I think that properly tagged breakdown drawings could significantly improve the performance of ai generated in-betweens. Anyone who is seriously interested in the process should read the late, great Richard William's book, _The Animator's Survival Kit_. This is especially true for those who want to "augment" the process with machine learning. The book is very readable, even for non-artists. And he gets into the nitty gritty of what makes a good performance, and the mechanics behind it. Edit: Another good resource, and relevant to 3D animation as well, is Raf Anzovin's _Just To Do Something Bad_ blog. He has many posts on what he calls "ephemeral rigging" that are absolutely fascinating. Be aware that the information is diffused through out the blog and not presented in a form for teaching. His opinions are fairly controversial in the field. But I think he is onto something. (https://www.justtodosomethingbad.com/ https://www.justtodosomethingbad.com/)
empath-nirvana 3y ago
The results are actually shockingly bad, considering that I think this should be _easier_ than producing a realistic image from scratch, which ai does quite well. I don't have more than a fuzzy idea of how to implement this, but it seems to me that key frames _should_ be interchangeable with in between frames, so you want to train it so that if you start with key frames and generate in-between frames, and then run the in-between frames through the ai, it should regenerate the keyframes.
- yosefk 3y ago
  It's counterintuitive but less so considering that it's way easier for a human to draw something from scratch than to inbetween 2 key frames as well! (I guess we're used to machines and people struggling at opposite things so this is counter counter intuitive, or something...) Animation key frames are not interchangeable with inbetween frames since the former try to show the most body parts in "extreme" positions though it's not always possible for all parts due to so called overlapping action. This is not to say you can't generate plausible "extremes" from inbetweens; acting wise key frames definitely have the most weight. AI being good at stills is true, though it takes a lot of prompting and cherry picking quite often; most results I get out of naively prompting the most famous models are outright terrifying.
- BlueTemplar 3y ago
  Same, I would have thought that edge detection would have been among the first problems to get solved !
- 6gvONxR4sf7o 3y ago
  > I think this should be _easier_ than producing a realistic image from scratch Think of this in terms of constraints. An image from scratch has self consistency constraints (this part of the image has to be consistent with that part) and it may have semantic constraints (if it has to match a prompt). An animation also has the self consistency constraints, but also has to be consistent with other entire images! The fact that the images are close in some semantic space helps, but all the tiny details become so important to get precisely correct in a new way. Like, if a model has some weird gap where it knows how to make an arm at 45 degrees and 60 degrees, but not 47, then that's fine for from-scratch generation. It'll just make one like it knows how (or more precisely, like it models as naturally likely). Same with any other weird quirks of what it thinks is good (naturally likely): It can just adjust to something that still matches the semantics but fits into the model's quirks. No such luck when now you need to get details like "47 degrees" correct. It's just a little harder without some training or modeling insight into how an arm at 45 degrees and 47 degrees are really "basically the same" (or just that much more data, so that you lose the weird bumps in the likelihood). I wouldn't be surprised if "just that much more data" ends up being the answer, given the volume of video data on the internet, and the wide applicability of video generation (and hence intense research in the area).
atseajournal 3y ago
Animation has to be the most intriguing hobby I'm never planning on engaging with, so this kind of blog post is great for me. I know hand-drawn 2D is its own beast, but what's your thought on using 3D datasets for handling the occlusion problem? There's so much motion-capture data out there -- obviously almost none of it has the punchiness and appeal of hand-drawn 2D, but feels like there could be something there. I haven't done any temporally-consistent image gen, just playing around with StableDiffusion for stills, but the ControlNets that make use of OpenPose are decent. 3D is on my mind here because the Spiderverse movies seemed like the first demonstration of how to really blend the two styles. I know they did some bespoke ML to help their animators out by adding those little crease-lines to a face as someone smiles... pretty sure they were generating 3d splines however, not raster data. Anyway, I'm saving the RSS feed, hope to hear more about this in the future!
- yosefk 3y ago
  The 2nd paper actually uses a 3D dataset, though it explicitly doesn't attempt to handle occlusion beyond detecting it. I sort of hope you can handle occlusion based on learning 2D training data similarly to the video interpolation paper cited at the end. If 3D is necessary, it's Not Good for 2D animation... AI for 3D animation is big in its own right; these puppets have 1 billion controllers and are not easy for humans to animate. I didn't look into it deeply because I like 2D more. (I learned 3D modeling and animation a bit, just to learn that I don't really like it...)
- wongarsu 3y ago
  Maybe there is also value in 2d datasets that aren't hand drawn. A lot of TV shows are made in Toon Boom or Adobe Animate (formerly Macromedia Flash). Those also do automatic inbetweening, but with a process that's closer to CSS animations: everything you want to move independently if it's own vector that can be moved, rotated and squished, and the software just interpolates the frames in between with your desired easing algorithm. That's a lot of data that's available on those original project files that's nontrivial to infer from the final product
  - yosefk 3y ago
wantsanagent 3y ago
Frankly I'm surprised this isn't much higher quality. The hard thing in the transformers era of ML is getting enough data that fits into the next token or masked language modeling paradigm, however in this case, inbetweening is exactly that task and every hand-drawn animation in history is potential training data. I'm not surprised that using off the shelf diffusion models or multi-modal transformer models trained primarily on still images would lead to this level of quality, but I am surprised if these results are from models trained specifically for this task on large amounts of animation data.
- yosefk 3y ago
  They're indeed not diffusion models, though they are trained on animation data as well as specifically designed for it (the raster papers at least.) I'm very hopeful wrt diffusion, though I'm looking at it and it's far from straightforward. One problem with diffusion and video is that diffusion training is data hungry and video data is big. A lot of approaches you see have some way to tackle this at their core. But also, AI today is like 80s PCs in some sense: both clearly the way of the future and clumsy/goofy, especially when juxtaposed with the triumphalism you tend to hear all around
TiredGuy 3y ago
The state of the art of 3d pose estimation and pose transfer from video seems to be pretty accurate. I wonder if another approach might be to infer a wireframe model for the character, then tween that instead of the character itself. It would be like the vector approach described in the article but with much, much fewer vertices, then once you have the tween, use something similar to pose transfer to map the most recent character's frame depiction to the pose. Training on a wireframe model seems like it would be easier, since there are plenty of wireframe animations out there (at least for humans) you could use and remove in-between frames to try inferring them.
_akhe 3y ago
Really cool read, I liked seeing all the examples. I wonder if it would be beneficial to train on lots of static views of the character too - not just the frames - so that permanent features like the face gets learned as a chunk of adjacent pixels, so when you go to make a walking animation, the relatively low amount of training data on moving legs in comparison to the high repetition of faces would cause only the legs to blur unpredictably, where the faces would be more in tact - the overall result might be a clearer looking animation.
- yosefk 3y ago
  Almost certainly a good idea. I'm about to start trying things in this direction
Solvency 3y ago
i don't get it. we essentially figured this out in the first Matrix by having a bunch of cameras film an actor and then used interpolation to create a 360 shot from it. why can't this basic idea be applied to simple 2d animation over two decades later?
- criddell 3y ago
  What was interpolated in the Matrix? I was under the impression they were creating 1 second of 24 fps video by combining images shot on 24 individual cameras.
- deleted 3y ago
  [deleted]
meindnoch 3y ago
Convert frames to SVG, pass SVG as text to ChatGPT, then ask for the SVG of in-between frames. Simple as.
brcmthrowaway 3y ago
If an AI could ever capture the charm of the original hand drawn animation, then it's over for us
- yosefk 3y ago
  If an AI can't make animators more productive, it's as close to over for hand drawn animation as it has been for the last decade

262 points by luu 3y ago

[dead]

Great article! This is one of the most overlooked problems in generative AI. It seems so trivial, but in fact, it is quite difficult. The difficulty arises because of the non-linearity that is expected in any natural motion. In fact, the author has highlighted all the possible difficulties of this problem in a much better manner. I started with some simple implementation by trying to move segments around the image using some segmentation mask + ROI. That strategy didn't work out, probably because of some mathematical bug or data insufficiency data. I suspect the later. The whole idea was to draw a segmentation mask on the target image, then draw lines that represent motion and give options to insert keyframes for the lines. Imagine you are drawing a curve from A to A. You divide the curve into A, A_1, A_2... B. Now, given the input of segmentation mask, motion curve, and whole image, we train some model to only move the ROI according to the motion curve and keyframe. The problem with this approach is in sampling the keyframe and matching consistencies --making sure RoI represents same object-- across subsequent keyframes. If we are able to solve some form of consistency, this method might be able to give enough constraints to generate viable results.

yosefk 3y ago
I currently shelved 3K more words of why it's hard if you're targeting real animators. One point is that human inbetweeners get "spacing charts" showing how much each part should move, even though they understand motion very well, because the key animator wants to control the acting

xrd 3y ago

This was really fun. It captured a lot of thinking on a topic I've been interested in for a while as well. The discussion about converting to a vector format was an interesting diversion. I've been experimenting with using potrace from inkscape to migrate raster images into SVG and then use animation libraries inside the browser to morph them, and this idea seems like it shares some concepts. One of my favorite films is A Scanner Darkly, and that used a technique called rotoscoping which I recall was a combination of hand tracing animation and computers then augmenting it, or vice versa. It sounded similar. The Wikipedia page talks about the director Richard Linklater and also the MIT professor Bob Sabiston who pioneered that derivative digital technique. It was fun to read that. https://en.m.wikipedia.org/wiki/Rotoscoping https://en.m.wikipedia.org/wiki/Rotoscoping https://en.m.wikipedia.org/wiki/Bob_Sabiston https://en.m.wikipedia.org/wiki/Bob_Sabiston

whywhywhywhy 3y ago
> and that used a technique called rotoscoping Technically it's "interpolated rotoscoping" using a custom tool called Rotoshop, which takes vector shapes drawn over footage then smoothly animates between the frames giving a distinct dream-like look to it. Rotoscoping is where you work to a traditional animation framerate drawing over live action but each frame is a new drawing and doesn't have the signature shimmery look Scanner Darkly and Waking Life so I think it's worth pointing out the distinction. https://en.wikipedia.org/wiki/Rotoshop https://en.wikipedia.org/wiki/Rotoshop

kleiba 3y ago

A blog post from the same guy that used to maintain the C++ FQA! https://yosefk.com/c++fqa/ https://yosefk.com/c++fqa/

AlexandrB 3y ago
One of my favourite bits of content written about C++. Highly recommended.

nicklecompte 3y ago

Great read - I learned quite a bit about this. The only quibble I had is at the end, and it's a very general layperson's view: > But what’s even more impressive - extremely impressive - is that the system decided that the body would go up before going back down between these two poses! (Which is why it’s having trouble with the right arm in the first place! A feature matching system wouldn’t have this problem, because it wouldn’t realize that in the middle position, the body would go up, and the right arm would have to be somewhere. Struggling with things not visible in either input keyframe is a good problem to have - it’s evidence of knowing these things exist, which demonstrates quite the capabilities!).... This system clearly learned a lot about three-dimensional real-world movement behind the 2D images it’s asked to interpolate between. I think that's an awfully strong conclusion to draw from this paper - the authors certainly don't make that claim. The "null hypothesis" should be that most generative video models have a ton of yoga instruction videos shot very similarly to the example shown, and here the AI is simply repeating similar frames from similar videos. Since this most likely wouldn't generalize to yoga videos shot at a skew angle, it's hard to conclude that the system learned anything about 3D real-world movement. Maybe it did! But the study authors didn't come to that conclusion, and since their technique is actually model-independent, they wouldn't be in a good position to check for data contamination / overfitting / etc. The authors seem to think the value of their work is that generative video AI is by default past->future but you can do future->past without changing the underlying model, and use that to smooth out issues in interpolation. I just don't think there's any rational basis for generalizing this to understanding 3D space itself. This isn't a criticism of the paper - the work seems clever but the paper is not very detailed and they haven't released the code yet. And my complaint is only a minor editorial comment on an otherwise excellent writeup. But I am wondering if the author might have been bedazzled by a few impressive results.

yosefk 3y ago
You're technically correct, there's no basis to argue that a "3D representation" was learned as opposed to "enough 2D projections to handle the inputs in question." That said, the hand which did not exist in either of the original 2D frames makes an appearance. I think calling wherever it was pulled out of "the 3rd dimension" is not wrong; it was occluded in both 2D inputs and functionally you had to know about its existence in the 3rd dimension to show it even if you technically did it by learning how pixels look across frames. You can also see much more 3D ish things in the paper, with 2 angles of a room and video created moving the camera between them. Of course in some sense it adds to my point without detracting from yours...

chris_st 3y ago

Not sure who would fund this research? Perhaps the Procreate Dreams folks [0]. I'm sure they'd love to have a functional auto-tween feature. 0: https://procreate.com/dreams https://procreate.com/dreams

aredox 3y ago

Now that's a great use for AI! Inbetweening has always been a thankless job usually outsources to sweatshop-like studios in Vietnam and - even recently - North Korea (Guy Delisle's "Pyongyang" comic is about his experience as a studio liaison there). And AI has less room to hallucinate - it is more a kind of interpolation - even if in this short curt example, the AI still "morphs" instead of cleanly transitioning. The real animation work and talent is in keyframes, not in the inbetweening.

yosefk 3y ago
Actually inbetweening is really hard (and I think requires talent) and used to be a big way to learn enough to become a key animator. And I would worry about AI eliminating this learning route if classical animation wasn't struggling to survive at all
- aredox 3y ago
  It's hard like translating novels - you have to match someone else's style, which is why it's thankless. I don't know if that's really a good pathway to become a key animator - how many inbetweeners are there for one key animator?
commieneko 3y ago
As an animator for 40 plus years, I can tell you that in-betweening is a very difficult job. The fact that it's often cheaply outsourced is more of a factor that the people paying for the animation simply don't care about the quality. The results are seldom good. As to how much poor quality in-betweening hurts the performance to the audience is a complicated discussion. Animation that is _very_ bad can often be well accepted if other factors compensate (voice acting, design, direction, etc.) A good in-betweener is not simply interpolating between the keys. For hand drawn animation at least, there's a lot more going on than that. We'll leave out any discussion of breakdowns here. For one it's a difficult concept, much more difficult than 'tweening to explain. The other is that different animators will give different opinions on what a breakdown is or does. I will say, though, I think that properly tagged breakdown drawings could significantly improve the performance of ai generated in-betweens. Anyone who is seriously interested in the process should read the late, great Richard William's book, _The Animator's Survival Kit_. This is especially true for those who want to "augment" the process with machine learning. The book is very readable, even for non-artists. And he gets into the nitty gritty of what makes a good performance, and the mechanics behind it. Edit: Another good resource, and relevant to 3D animation as well, is Raf Anzovin's _Just To Do Something Bad_ blog. He has many posts on what he calls "ephemeral rigging" that are absolutely fascinating. Be aware that the information is diffused through out the blog and not presented in a form for teaching. His opinions are fairly controversial in the field. But I think he is onto something. (https://www.justtodosomethingbad.com/ https://www.justtodosomethingbad.com/)

empath-nirvana 3y ago

The results are actually shockingly bad, considering that I think this should be _easier_ than producing a realistic image from scratch, which ai does quite well. I don't have more than a fuzzy idea of how to implement this, but it seems to me that key frames _should_ be interchangeable with in between frames, so you want to train it so that if you start with key frames and generate in-between frames, and then run the in-between frames through the ai, it should regenerate the keyframes.

yosefk 3y ago
It's counterintuitive but less so considering that it's way easier for a human to draw something from scratch than to inbetween 2 key frames as well! (I guess we're used to machines and people struggling at opposite things so this is counter counter intuitive, or something...) Animation key frames are not interchangeable with inbetween frames since the former try to show the most body parts in "extreme" positions though it's not always possible for all parts due to so called overlapping action. This is not to say you can't generate plausible "extremes" from inbetweens; acting wise key frames definitely have the most weight. AI being good at stills is true, though it takes a lot of prompting and cherry picking quite often; most results I get out of naively prompting the most famous models are outright terrifying.
BlueTemplar 3y ago
Same, I would have thought that edge detection would have been among the first problems to get solved !
6gvONxR4sf7o 3y ago
> I think this should be _easier_ than producing a realistic image from scratch Think of this in terms of constraints. An image from scratch has self consistency constraints (this part of the image has to be consistent with that part) and it may have semantic constraints (if it has to match a prompt). An animation also has the self consistency constraints, but also has to be consistent with other entire images! The fact that the images are close in some semantic space helps, but all the tiny details become so important to get precisely correct in a new way. Like, if a model has some weird gap where it knows how to make an arm at 45 degrees and 60 degrees, but not 47, then that's fine for from-scratch generation. It'll just make one like it knows how (or more precisely, like it models as naturally likely). Same with any other weird quirks of what it thinks is good (naturally likely): It can just adjust to something that still matches the semantics but fits into the model's quirks. No such luck when now you need to get details like "47 degrees" correct. It's just a little harder without some training or modeling insight into how an arm at 45 degrees and 47 degrees are really "basically the same" (or just that much more data, so that you lose the weird bumps in the likelihood). I wouldn't be surprised if "just that much more data" ends up being the answer, given the volume of video data on the internet, and the wide applicability of video generation (and hence intense research in the area).

atseajournal 3y ago

Animation has to be the most intriguing hobby I'm never planning on engaging with, so this kind of blog post is great for me. I know hand-drawn 2D is its own beast, but what's your thought on using 3D datasets for handling the occlusion problem? There's so much motion-capture data out there -- obviously almost none of it has the punchiness and appeal of hand-drawn 2D, but feels like there could be something there. I haven't done any temporally-consistent image gen, just playing around with StableDiffusion for stills, but the ControlNets that make use of OpenPose are decent. 3D is on my mind here because the Spiderverse movies seemed like the first demonstration of how to really blend the two styles. I know they did some bespoke ML to help their animators out by adding those little crease-lines to a face as someone smiles... pretty sure they were generating 3d splines however, not raster data. Anyway, I'm saving the RSS feed, hope to hear more about this in the future!

yosefk 3y ago
The 2nd paper actually uses a 3D dataset, though it explicitly doesn't attempt to handle occlusion beyond detecting it. I sort of hope you can handle occlusion based on learning 2D training data similarly to the video interpolation paper cited at the end. If 3D is necessary, it's Not Good for 2D animation... AI for 3D animation is big in its own right; these puppets have 1 billion controllers and are not easy for humans to animate. I didn't look into it deeply because I like 2D more. (I learned 3D modeling and animation a bit, just to learn that I don't really like it...)
wongarsu 3y ago
Maybe there is also value in 2d datasets that aren't hand drawn. A lot of TV shows are made in Toon Boom or Adobe Animate (formerly Macromedia Flash). Those also do automatic inbetweening, but with a process that's closer to CSS animations: everything you want to move independently if it's own vector that can be moved, rotated and squished, and the software just interpolates the frames in between with your desired easing algorithm. That's a lot of data that's available on those original project files that's nontrivial to infer from the final product
- yosefk 3y ago

wantsanagent 3y ago

Frankly I'm surprised this isn't much higher quality. The hard thing in the transformers era of ML is getting enough data that fits into the next token or masked language modeling paradigm, however in this case, inbetweening is exactly that task and every hand-drawn animation in history is potential training data. I'm not surprised that using off the shelf diffusion models or multi-modal transformer models trained primarily on still images would lead to this level of quality, but I am surprised if these results are from models trained specifically for this task on large amounts of animation data.

yosefk 3y ago
They're indeed not diffusion models, though they are trained on animation data as well as specifically designed for it (the raster papers at least.) I'm very hopeful wrt diffusion, though I'm looking at it and it's far from straightforward. One problem with diffusion and video is that diffusion training is data hungry and video data is big. A lot of approaches you see have some way to tackle this at their core. But also, AI today is like 80s PCs in some sense: both clearly the way of the future and clumsy/goofy, especially when juxtaposed with the triumphalism you tend to hear all around

TiredGuy 3y ago

The state of the art of 3d pose estimation and pose transfer from video seems to be pretty accurate. I wonder if another approach might be to infer a wireframe model for the character, then tween that instead of the character itself. It would be like the vector approach described in the article but with much, much fewer vertices, then once you have the tween, use something similar to pose transfer to map the most recent character's frame depiction to the pose. Training on a wireframe model seems like it would be easier, since there are plenty of wireframe animations out there (at least for humans) you could use and remove in-between frames to try inferring them.

_akhe 3y ago

Really cool read, I liked seeing all the examples. I wonder if it would be beneficial to train on lots of static views of the character too - not just the frames - so that permanent features like the face gets learned as a chunk of adjacent pixels, so when you go to make a walking animation, the relatively low amount of training data on moving legs in comparison to the high repetition of faces would cause only the legs to blur unpredictably, where the faces would be more in tact - the overall result might be a clearer looking animation.

yosefk 3y ago
Almost certainly a good idea. I'm about to start trying things in this direction

Solvency 3y ago

i don't get it. we essentially figured this out in the first Matrix by having a bunch of cameras film an actor and then used interpolation to create a 360 shot from it. why can't this basic idea be applied to simple 2d animation over two decades later?

criddell 3y ago
What was interpolated in the Matrix? I was under the impression they were creating 1 second of 24 fps video by combining images shot on 24 individual cameras.
deleted 3y ago
[deleted]

meindnoch 3y ago

Convert frames to SVG, pass SVG as text to ChatGPT, then ask for the SVG of in-between frames. Simple as.

brcmthrowaway 3y ago

If an AI could ever capture the charm of the original hand drawn animation, then it's over for us

yosefk 3y ago
If an AI can't make animators more productive, it's as close to over for hand drawn animation as it has been for the last decade