4 ms·

Wow, this looks like the beginning of the end for in-between animation. A large chunk of an anime's budget goes to in-betweening. Essentially human interpolati

Wow, this looks like the beginning of the end for in-between animation. A large chunk of an anime's budget goes to in-betweening. Essentially human interpolation of graphics between two key frames (Its usually like 6-12 inbetweens per key frame). People hate this job, and it is highly unproductive, so generally outsourced by Japan to other countries. Western animation decided to abandon it altogether, and move first to flash, then to 3d animation. But in retrospect that was a mistake, as it lost so much of the creative flexibility of 2d animation. Anime today is substantially bigger than western animation as a result. Crunchyroll has 13 million subscribers. AI will solve the problems with 2d animation. Something like SORA fine-tuned on anime-keyframe data like this. Can probably easily solve in-betweening. Then the 2d animation workflow will dominate 3d. Its so much easier to just draw a beginning and end key-frame, then have the AI fill it in. Than to model and rig and render a entire scene.

CactusOnFire 2y ago
This is one job I am actually hopeful will be fully automated. I've heard before that the people doing this 'tween' frames make less than minimum wage on contract. That being said, I'm sure this is going to take a while to fully replace the manual efforts. There will probably be awkward phase in between the outset and the perfect modelling efforts, and I'm sure lower budget shows (or ones looking to cut corners) will be the early adopters.
- anonylizard 2y ago
  Anime already cuts corners by using cheap 3d models in place of 2d hand drawn objects. The more rigid the object, the better this works. 3d cars look better than 2d cars, even in an otherwise 2d show. Mechs look a bit worse. And human characters look horrible. Yet anime studios still do it. Including for critical highlight scenes like dance scenes (Check out Love live dance scens), because it is so, so hard to draw humans dancing. So if anime studios are willing to do something, that looks obviously bad, as a widespread practice. There'll be 0 barriers to AI inbetweening adaptation, which would likely look BETTER than human inbetweening within a year of release. AI anime art has already wiped out the lower-end of patreon artists, and is heavily impacting the mid-tier. Because AI has gotten more technically proficient than the average mid-tier artist. Pretty much only the higher-end can hold their heads above water. Or they have to transition to drawing comics with storylines, instead of just simple images.
  - GaggiX 2y ago
    >3d cars look better than 2d cars, even in an otherwise 2d show. This is quite debatable, if you notice that the car is a 3D object, then something is already wrong.
    - Hamuko 2y ago
      Usually you can tell by the fact that nothing is wrong, since the 3D model is very consistent and on model. A hand-drawn car is usually not that.
      - GaggiX 2y ago
        The same could be said for a 3D human character, it's very consistent because it's a 3D model but it's horrible to look at.
        mlyle 2y ago
        His point, I believe: artistically interpreting the motion and shape of humans or objects with larger moving parts makes animation look more on-style. But for "boring" rigid objects, there's less of this advantage; hence, the consistency benefits often are more important.
        numpad0 2y ago
        unless it's Miyazaki films - in which case most humans are intended to be lifeless rigid objects and every machines are to be lifelike animations (/s)
        speedgoose 2y ago
        Do you notice when Bluey characters are animated from 3D or 2D? The software they use allows to do 2D drawings from 3D animated models. https://www.celaction.com/en/celaction2d/ https://www.celaction.com/en/celaction2d/
        2y ago
- guyomes 2y ago
  > I've heard before that the people doing this 'tween' frames make less than minimum wage on contract. An employee of an animation company describes in a comic book his experience on working with people drawing the in-between frames. They were paid literally with rice bags [1]. [1]: https://en.wikipedia.org/wiki/Pyongyang:_A_Journey_in_North_Korea https://en.wikipedia.org/wiki/Pyongyang:_A_Journey_in_North_...
mcphage 2y ago
I wondered about that—they stated that they processed videos to pull out keyframes. It makes me wonder how much that will or won't help with in-betweening.
- astrange 2y ago
  Keyframes in digital video means a different thing from keyframes in animation; they could be related but usually wouldn't be. You can hide animation keyframes with a number of techniques, for instance by having different "keyframes" for different parts of a scene so the video doesn't show them all at the same instant. (Rough generalization, but anime is more likely to do this with foreground vs background elements, while Western animation would do this with different characters. In anime the keyframe artists tend to draw every character in the frame, and put more of a personal touch on them, so it's easier to see their individual styles.) Since this is a "sakuga" dataset, Mitsuo Iso is one of the most famous examples of really natural looking movement here. https://www.youtube.com/watch?v=NTMJ8dGFUkc https://www.youtube.com/watch?v=NTMJ8dGFUkc When they're bad at it you get this effect where characters constantly seem to "settle" into a keyframe pose that looks realistic but too static, and then immediately go back into inbetweening that moves a lot but isn't physically possible. I feel like B-grade Disney stuff is the worst here but don't have an example on hand.
deleted 2y ago
[deleted]
zbirkenbuel 2y ago
Can't have a conversation about inbetween frames without sharing this[1] Noodle video. Now granted the interpolation there is pretty rigid, and the goal of this dataset will be to train an LLM replicate hand tweening, but I can't help but feel there's going to be a massive uncanny valley. [1]https://youtu.be/_KRb_qV9P4g?si=vtDvfLU6XYz5QwWV&t=355 https://youtu.be/_KRb_qV9P4g?si=vtDvfLU6XYz5QwWV&t=355

CactusOnFire 2y ago

This is one job I am actually hopeful will be fully automated. I've heard before that the people doing this 'tween' frames make less than minimum wage on contract. That being said, I'm sure this is going to take a while to fully replace the manual efforts. There will probably be awkward phase in between the outset and the perfect modelling efforts, and I'm sure lower budget shows (or ones looking to cut corners) will be the early adopters.

anonylizard 2y ago
Anime already cuts corners by using cheap 3d models in place of 2d hand drawn objects. The more rigid the object, the better this works. 3d cars look better than 2d cars, even in an otherwise 2d show. Mechs look a bit worse. And human characters look horrible. Yet anime studios still do it. Including for critical highlight scenes like dance scenes (Check out Love live dance scens), because it is so, so hard to draw humans dancing. So if anime studios are willing to do something, that looks obviously bad, as a widespread practice. There'll be 0 barriers to AI inbetweening adaptation, which would likely look BETTER than human inbetweening within a year of release. AI anime art has already wiped out the lower-end of patreon artists, and is heavily impacting the mid-tier. Because AI has gotten more technically proficient than the average mid-tier artist. Pretty much only the higher-end can hold their heads above water. Or they have to transition to drawing comics with storylines, instead of just simple images.
- GaggiX 2y ago
  >3d cars look better than 2d cars, even in an otherwise 2d show. This is quite debatable, if you notice that the car is a 3D object, then something is already wrong.
  - Hamuko 2y ago
    Usually you can tell by the fact that nothing is wrong, since the 3D model is very consistent and on model. A hand-drawn car is usually not that.
    - GaggiX 2y ago
      The same could be said for a 3D human character, it's very consistent because it's a 3D model but it's horrible to look at.
      - mlyle 2y ago
        His point, I believe: artistically interpreting the motion and shape of humans or objects with larger moving parts makes animation look more on-style. But for "boring" rigid objects, there's less of this advantage; hence, the consistency benefits often are more important.
        numpad0 2y ago
        unless it's Miyazaki films - in which case most humans are intended to be lifeless rigid objects and every machines are to be lifelike animations (/s)
      - speedgoose 2y ago
        Do you notice when Bluey characters are animated from 3D or 2D? The software they use allows to do 2D drawings from 3D animated models. https://www.celaction.com/en/celaction2d/ https://www.celaction.com/en/celaction2d/
        2y ago
guyomes 2y ago
> I've heard before that the people doing this 'tween' frames make less than minimum wage on contract. An employee of an animation company describes in a comic book his experience on working with people drawing the in-between frames. They were paid literally with rice bags [1]. [1]: https://en.wikipedia.org/wiki/Pyongyang:_A_Journey_in_North_Korea https://en.wikipedia.org/wiki/Pyongyang:_A_Journey_in_North_...

mcphage 2y ago

I wondered about that—they stated that they processed videos to pull out keyframes. It makes me wonder how much that will or won't help with in-betweening.

astrange 2y ago
Keyframes in digital video means a different thing from keyframes in animation; they could be related but usually wouldn't be. You can hide animation keyframes with a number of techniques, for instance by having different "keyframes" for different parts of a scene so the video doesn't show them all at the same instant. (Rough generalization, but anime is more likely to do this with foreground vs background elements, while Western animation would do this with different characters. In anime the keyframe artists tend to draw every character in the frame, and put more of a personal touch on them, so it's easier to see their individual styles.) Since this is a "sakuga" dataset, Mitsuo Iso is one of the most famous examples of really natural looking movement here. https://www.youtube.com/watch?v=NTMJ8dGFUkc https://www.youtube.com/watch?v=NTMJ8dGFUkc When they're bad at it you get this effect where characters constantly seem to "settle" into a keyframe pose that looks realistic but too static, and then immediately go back into inbetweening that moves a lot but isn't physically possible. I feel like B-grade Disney stuff is the worst here but don't have an example on hand.

deleted 2y ago

[deleted]

zbirkenbuel 2y ago

Can't have a conversation about inbetween frames without sharing this[1] Noodle video. Now granted the interpolation there is pretty rigid, and the goal of this dataset will be to train an LLM replicate hand tweening, but I can't help but feel there's going to be a massive uncanny valley. [1]https://youtu.be/_KRb_qV9P4g?si=vtDvfLU6XYz5QwWV&t=355 https://youtu.be/_KRb_qV9P4g?si=vtDvfLU6XYz5QwWV&t=355