6 ms·

Not going to bother with this until the temporal cohesion issue is solved. The results are cool, but the variety in frames makes it look like a very specific an

Not going to bother with this until the temporal cohesion issue is solved. The results are cool, but the variety in frames makes it look like a very specific and distracting art style instead of true animation.

colordrops 3y ago
Did you scroll down and see the different methods used, specifically method 5? Not perfect, but getting pretty close.
- justinclift 3y ago
  It's a lot better than earlier work, but still not really at the level it needs to get to.
- moonchrome 3y ago
  The last one looks the creepiest - like a dystopian matrix glitch. Could be a cool effect to illustrate that.
- rpastuszak 3y ago
  The light reflection on the shoulder doesn't make much sense. I think this is less pronounced when soft light is involved. I wonder if sharp light and reflections will take longer to be handled properly. Every frame on the shoulder looks ok on its own, but when animated it feels like it was taken in a different environment than the rest. I suspect depth maps (SD2 already supports them) could be used to achieve that in the future. I wonder if a diffusion model could accept an "onion skin" noise, so the transitions between frames would be less jarring. Can someone with more knowledge than me explain what's the most promising approach here?
  - bondarchuk 3y ago
    >The light reflection on the shoulder doesn't make much sense. People will really grasp at anything to find something to critique in generative AI, huh?
    - SirMaster 3y ago
      Wait, should we not critique? OK i'll fix that. It's perfect. Stop all work and development on it now guys, it's perfect. It has no flaws and makes no mistakes. Amazing I guess.
      - rpastuszak 3y ago
        I think @bondarchuk took my comment out of context, quoting just the first line. No big deal, it's just hard to respond to that. I don't even see an issue with that particular artefact, I think it's an interesting problem from a technical pov, hence literally every other part of my comment.
        bondarchuk 3y ago
        It just seemed like a bit of a nitpick in the context of the huge temporal consistency issues/fixes, of course you're all free to discuss whatever you want and I agree it's somewhat interesting.
        rpastuszak 3y ago
        Haha, nope, not my intention at all. Although I understand the expectation give the average style of discourse on this site.
- llm_nerd 3y ago
  The last one is largely the original video with a cartoon filter and basically anime facetune. It is educational and interesting, but it looks the best because it's actually doing the least, though even then you get the temporal artifacts on the hair and so on. It is improving super rapidly and it will be a pretty seamless soon enough in all likelihood. We are in the intermediate stage right now, and the results are going to be incredible.
andybak 3y ago
Mark my words. In a few years people will be writing filters and trying to figure out how to recreate the look of early AI animation.
- raincole 3y ago
  ...why? Do you expect GitHub and HuggingFace to go under in a few years?
  - mdp2021 3y ago
    The poster is just saying that "the "glitch" of today will be the nostalgic style of tomorrow".
    - andybak 3y ago
      Exactly. I have no idea what the other commenters thought I meant.
      - theobromananda 3y ago
        They are saying that there is no need to figure out how to recreate "the "glitch" of today that will be the nostalgic style of tomorrow" as the "old"/current models are all available on GitHub and huggingface, and will be in the future. There is no need to figure out how to recreate this style as you will be able to find it on these platforms in the future. The commenter understood you, but you did not understand the commenter.
        andybak 3y ago
        Yes. Because of the way reply notifications come in I thought it was a reply to a different comment. But I still think that's missing the point of my (not entirely serious) comment. People still have access to analogue amplifiers but digital simulations of them are still developed. As models and workflows improve, if people want a flickery old-school look then they may well simulate it rather than go through the hassle of running old tools that might not mesh well with newer workflows.
objclxt 3y ago
> the variety in frames makes it look like a very specific and distracting art style instead of true animation. Well, what is "true animation"? There are number of types of traditional animation where you draw both the characters and the backgrounds from scratch for every frame, so you get this variety naturally. Or you have more practical techniques like strata-cut animation[1], where again you get what we might think of today as a very "ML-like" effect of the background constantly shifting. All of these techniques are very time intensive, so you don't often see them in commercial animation. [1]:https://en.wikipedia.org/wiki/Strata-cut_animation https://en.wikipedia.org/wiki/Strata-cut_animation
- bondarchuk 3y ago
  In this case we have objects disappearing and reappearing, and the design of clothing and objects (headphones/tiara) being significantly different from frame to frame, pieces of sleeves blinking in and out of existence, headphones becoming tiaras becoming helmets... I'm sure you can find examples of such things in traditional animation but there's clearly something peculiar going on here due to the frames being processed mostly standalone, and it's clearly specific to this exact way of processing. Even if you "draw both the characters and the backgrounds from scratch for every frame", normally the animator would at least have the previous frames and/or overall design at his disposal as a reference. What's happening here is more like those crowdsourcing experiments where you let many people draw a single frame without knowing what the others are up to.
  - uwagar 3y ago
    you'd think a real AI would be aware of all these issues and not let this happen.
woctordho 3y ago
I've had the idea for a long time but never managed to try it: Alias-free convolutions (like StyleGAN3) may help with temporal cohesion. The StyleGAN3 project page shows some good videos: https://nvlabs.github.io/stylegan3/ https://nvlabs.github.io/stylegan3/
nialv7 3y ago
NVIDIA has already published something that solves the coherence problem, and IIUC it's even based on stable diffusion. Hopefully the community will reproduce it soon. https://research.nvidia.com/labs/toronto-ai/VideoLDM/ https://research.nvidia.com/labs/toronto-ai/VideoLDM/
beepbooptheory 3y ago
But isn't also the case you have so much control over the individual frames themselves anyway? Why draw the line where you are? Just based on how these things inherently work, I don't see the specificity problem getting solved anytime soon, short of making a whole new set of weights from some whole new cultural timeline of artistic output. But its not an issue really! This is just what AI art is. It is something different from just painting: somewhat less and somewhat more in different dimensions. You trade in individual expression and authorship for bricolage and universality. That's cool, but it will always be "specific." If you just want pretty anime girls or the like, you'd get a lot farther with dumber tools like motion-detection on spine-based animation anyway.

colordrops 3y ago

Did you scroll down and see the different methods used, specifically method 5? Not perfect, but getting pretty close.

justinclift 3y ago
It's a lot better than earlier work, but still not really at the level it needs to get to.
moonchrome 3y ago
The last one looks the creepiest - like a dystopian matrix glitch. Could be a cool effect to illustrate that.
rpastuszak 3y ago
The light reflection on the shoulder doesn't make much sense. I think this is less pronounced when soft light is involved. I wonder if sharp light and reflections will take longer to be handled properly. Every frame on the shoulder looks ok on its own, but when animated it feels like it was taken in a different environment than the rest. I suspect depth maps (SD2 already supports them) could be used to achieve that in the future. I wonder if a diffusion model could accept an "onion skin" noise, so the transitions between frames would be less jarring. Can someone with more knowledge than me explain what's the most promising approach here?
- bondarchuk 3y ago
  >The light reflection on the shoulder doesn't make much sense. People will really grasp at anything to find something to critique in generative AI, huh?
  - SirMaster 3y ago
    Wait, should we not critique? OK i'll fix that. It's perfect. Stop all work and development on it now guys, it's perfect. It has no flaws and makes no mistakes. Amazing I guess.
    - rpastuszak 3y ago
      I think @bondarchuk took my comment out of context, quoting just the first line. No big deal, it's just hard to respond to that. I don't even see an issue with that particular artefact, I think it's an interesting problem from a technical pov, hence literally every other part of my comment.
      - bondarchuk 3y ago
        It just seemed like a bit of a nitpick in the context of the huge temporal consistency issues/fixes, of course you're all free to discuss whatever you want and I agree it's somewhat interesting.
        rpastuszak 3y ago
        Haha, nope, not my intention at all. Although I understand the expectation give the average style of discourse on this site.
llm_nerd 3y ago
The last one is largely the original video with a cartoon filter and basically anime facetune. It is educational and interesting, but it looks the best because it's actually doing the least, though even then you get the temporal artifacts on the hair and so on. It is improving super rapidly and it will be a pretty seamless soon enough in all likelihood. We are in the intermediate stage right now, and the results are going to be incredible.

andybak 3y ago

Mark my words. In a few years people will be writing filters and trying to figure out how to recreate the look of early AI animation.

raincole 3y ago
...why? Do you expect GitHub and HuggingFace to go under in a few years?
- mdp2021 3y ago
  The poster is just saying that "the "glitch" of today will be the nostalgic style of tomorrow".
  - andybak 3y ago
    Exactly. I have no idea what the other commenters thought I meant.
    - theobromananda 3y ago
      They are saying that there is no need to figure out how to recreate "the "glitch" of today that will be the nostalgic style of tomorrow" as the "old"/current models are all available on GitHub and huggingface, and will be in the future. There is no need to figure out how to recreate this style as you will be able to find it on these platforms in the future. The commenter understood you, but you did not understand the commenter.
      - andybak 3y ago
        Yes. Because of the way reply notifications come in I thought it was a reply to a different comment. But I still think that's missing the point of my (not entirely serious) comment. People still have access to analogue amplifiers but digital simulations of them are still developed. As models and workflows improve, if people want a flickery old-school look then they may well simulate it rather than go through the hassle of running old tools that might not mesh well with newer workflows.

objclxt 3y ago

> the variety in frames makes it look like a very specific and distracting art style instead of true animation. Well, what is "true animation"? There are number of types of traditional animation where you draw both the characters and the backgrounds from scratch for every frame, so you get this variety naturally. Or you have more practical techniques like strata-cut animation[1], where again you get what we might think of today as a very "ML-like" effect of the background constantly shifting. All of these techniques are very time intensive, so you don't often see them in commercial animation. [1]:https://en.wikipedia.org/wiki/Strata-cut_animation https://en.wikipedia.org/wiki/Strata-cut_animation

bondarchuk 3y ago
In this case we have objects disappearing and reappearing, and the design of clothing and objects (headphones/tiara) being significantly different from frame to frame, pieces of sleeves blinking in and out of existence, headphones becoming tiaras becoming helmets... I'm sure you can find examples of such things in traditional animation but there's clearly something peculiar going on here due to the frames being processed mostly standalone, and it's clearly specific to this exact way of processing. Even if you "draw both the characters and the backgrounds from scratch for every frame", normally the animator would at least have the previous frames and/or overall design at his disposal as a reference. What's happening here is more like those crowdsourcing experiments where you let many people draw a single frame without knowing what the others are up to.
- uwagar 3y ago
  you'd think a real AI would be aware of all these issues and not let this happen.

woctordho 3y ago

I've had the idea for a long time but never managed to try it: Alias-free convolutions (like StyleGAN3) may help with temporal cohesion. The StyleGAN3 project page shows some good videos: https://nvlabs.github.io/stylegan3/ https://nvlabs.github.io/stylegan3/

nialv7 3y ago

NVIDIA has already published something that solves the coherence problem, and IIUC it's even based on stable diffusion. Hopefully the community will reproduce it soon. https://research.nvidia.com/labs/toronto-ai/VideoLDM/ https://research.nvidia.com/labs/toronto-ai/VideoLDM/

beepbooptheory 3y ago

But isn't also the case you have so much control over the individual frames themselves anyway? Why draw the line where you are? Just based on how these things inherently work, I don't see the specificity problem getting solved anytime soon, short of making a whole new set of weights from some whole new cultural timeline of artistic output. But its not an issue really! This is just what AI art is. It is something different from just painting: somewhat less and somewhat more in different dimensions. You trade in individual expression and authorship for bricolage and universality. That's cool, but it will always be "specific." If you just want pretty anime girls or the like, you'd get a lot farther with dumber tools like motion-detection on spine-based animation anyway.