6 ms·

Someone must be doing this for animation, right? I want to watch all the animated classics at full frame rate. Purists may gasp but I think that it won't be lon

Someone must be doing this for animation, right? I want to watch all the animated classics at full frame rate. Purists may gasp but I think that it won't be long before these models can do a lot better than mere interpolation, preserving the original feel of sequences despite the increased frame rate and avoiding the "soap opera" effect.

teraflop 4y ago
The "soap opera" effect is mainly a perceptual response to the high frame rate itself, even if there are no interpolation artifacts. That's why it has its name: soap operas are typically filmed at 60fps, with cameras that cameras one full frame every 1/60th of a second.
- AstralStorm 4y ago
  More specifically, the reaction to lack of blur. You can get 60 fps upconverted footage from motion interpolation to not exhibit the effect by adding copious gaussian motion blur. The same problem (plus perceived judder due to low fps) appears if you play a video on a screen equipped with correctly tuned strobe backlight. It's one of the reasons as to why simple kernel interpolation (blurmotion) with most kernels does not exhibit this effect either.
  - ubercow13 4y ago
    Why would there be a lack of blur? The 24fps (say) footage will start with more blur as the shutter speed will be slower and this blur will remain when interpolated to 60fps. It would surely have a physically impossibly high amount of blur after interpolation.
    - dagmx 4y ago
      Interpolation doesn’t change shutter angle which is what causes motion blur to have a specific aesthetic look, so when interpolated it looks wrong for the frame rate. You’re right in that it doesn’t change the amount of blur, I’m not sure why the person you’re replying to thinks that. But it does change the aesthetic of the blur. Perhaps they’re thinking of the issue with the Hobbit where the higher frame rate at a given shutter angle (180 let’s say) that shortened the blur compared to a traditional 24 fps film. They adjusted this for the second hobbit film to leave the shutter open longer (not a 360 shutter, but something like a 270) to bring back some of that feeling.
- taneq 4y ago
  A similar thing happened with The Hobbit, iirc. It was shot in 60fps and everyone said the CGI was too obvious, I think we’ve just learned to associate high frame rates with computer games.
  - cma 4y ago
    Hobbit was 48hz (to be a multiple of the more standard 24 letting it also work on 24-only projection systems I am assuming).
    - ace2358 4y ago
      And this was because there were two cameras for the whole 3D craze. One for each eye. Though I don’t think I saw the 48fps in cinemas in Australia
  - breatheoften 4y ago
    My big problem with the high frame rate in the hobbit was the slack of camera stability -- I think whatever image stabilization solution they used performed very poorly for that film -- not sure what went wrong but always hypothesized it was related to the higher capture rate. It felt like the camera was always bouncing and moving around randomly in a way I found very displeasing...
  - Gordonjcp 4y ago
    I have to admit, watching it in 48fps (pretty sure it was 48 and not 60) the CGI sequences did look very much like video game cutscenes.
- leeoniya 4y ago
  > The "soap opera" effect is mainly a perceptual response to the high frame rate itself, even if there are no interpolation artifacts. also, i think the 3:2 pulldown (24->29.97fps) and interlacing that typically causes jitter/blurring is absent at framerates that are even multiples (30->30 or 30->60)
- modeless 4y ago
  Sure, but most hand drawn animation is done at 12 FPS or lower. Even just upsampling from 12 to 24 would be awesome and wouldn't get into the territory where the soap opera effect is caused by the framerate itself.
  - dagmx 4y ago
    This would change the acting choices the animator made however. to take my response from elsewhere , consider a single drawn frame impact that is held for 2 or more frames. That timing is integral to the performance. When you interpolate, it no longer holds the right amount so the performance changes. People may scoff at that, but it’s a huge deal for artists. It was hell when we had dailies and motion smoothing on one of the TVs turned on (if we weren’t using screening rooms). Suddenly all the notes were different and acting didn’t feel right.
  - deleted 4y ago
    [deleted]
- Gordonjcp 4y ago
  Not quite! They capture one full frame every 1/30th of a second in the US and Japan, and 1/25th everywhere else. They capture one *field* every 1/50th of a second, and interlace it. The reason for this is that it was hard to scan a full 625 lines in 1/50th of a second, and it would have required ludicrous bandwidth for the AM transmitters of the day and massive output valves to drive the scan coils in TV sets which would then produced huge flyback spikes. So the decision was to scan 312.5 lines per field, offset slightly, with each pair of fields forming one 1/25th of a second frame. Telecine was easy in the UK because you could speed the film up a little and then each TV frame would be two fields per film frame. If you shoot 50i you can deinterlace to 25p by combining the two sets of scan lines from each field with a bit of clever processing to hide the "combing" caused by things moving between fields, or you can double it up to 50p by just copying each line out twice with a bit of clever processing to smooth out the steps. I could post some video clips showing this if you were interested. Of course modern digital TV cameras are capable of shooting 60p as you say, but anyone shooting "telenovela" stuff is going to be shooting 60i.
  - warmcompress 4y ago
    Thank you!! Our contemporary 60hz world seems to have retconned broadcast video standards.
    - Gordonjcp 4y ago
      It gets worse because for NTSC colour they changed it to 59.94Hz so that the colour subcarrier could be an odd multiple of the scan rate and just fit into the 4MHz-wide channels. This is why all your video editing software has got those ballachey 59.94/24.97 modes. It gets worse because for telecine you need to need to show two frames of 24fps film for every three frames of video, so you need to slow your film down to 23.976fps so it fits. Ballache.
tehnub 4y ago
Frame timing is an integral, creative aspect of the medium, not something to be fixed. 1. https://www.idtech.com/blog/what-does-animating-on-ones-twos-and-threes-mean https://www.idtech.com/blog/what-does-animating-on-ones-twos... 2. https://www.awn.com/tooninstitute/lessonplan/timing.htm#:~:text=In%20film%20animation%20there%20are,)%20%2D%20it%20is%20called%20one's https://www.awn.com/tooninstitute/lessonplan/timing.htm#:~:t.... 3. Search for "A good example would be this cut, and especially the first four shots." in https://animetudes.com/2021/03/06/the-kanada-style-in-context/ https://animetudes.com/2021/03/06/the-kanada-style-in-contex...
- modeless 4y ago
  > Frame timing is an integral, creative aspect of the medium > not something to be fixed. The first does not imply the second in every case. There are many, many instances where a creator with unlimited resources would have chosen to drawn more frames. And now we can. We don't have to indiscriminately interpolate everything to 120 Hz, but I think it's clearly possible to make huge improvements to a whole lot of hand drawn animation.
  - tehnub 4y ago
    Sure, maybe a creator can utilize AI interpolation as a tool. I understood your comment to be about applying AI interpolation to works already completed.
    - modeless 4y ago
      Both.
- satvikpendem 4y ago
  As someone who already interpolates media with SVP4, I remain unconvinced by those arguments. My media looks amazing running at 120 hz, I don't get headaches by jarring panning shots that look closer to a slideshow than a movie. I recently watched Avatar 2 in IMAX 3D with 48 FPS, and it looked incredible. If only all media in the future could let go of the arbitrary 24 frame limit, which was made a hundred years ago because filmmakers wanted to conserve their film usage.
dagmx 4y ago
Adding frames isn’t simply an act of interpolation when it comes to animation. You’d need to understand acting choices as well. The timing of motion and the importance of how long a frame is held for is important to how characters are portrayed and the impact of actions. Take a punch or a bounce. The time of contact may be one drawn frame but it may be on screen longer to sell it. What should interpolation do here? If if it smooths everything but the impact, then the impact will feel too sticky and lose its force. Similarly if it smooths the impact as well then it goes by too quick and it feels too weak. Animation is full of these choices and it’s a common problem in video games for example where animators animate at a given frame rate but games can run at variable sampling rates along their animation, causing unwanted artifacts of motion or ruin the performance.
ipsum2 4y ago
DAIN[0] is very popular for increasing framerate of videos, including animations. Examples: https://www.youtube.com/watch?v=IK-Q3EcTnTA https://www.youtube.com/watch?v=IK-Q3EcTnTA https://www.youtube.com/watch?v=unRdyQYwL5k https://www.youtube.com/watch?v=unRdyQYwL5k https://www.youtube.com/watch?v=pJFHg5UmqDQ https://www.youtube.com/watch?v=pJFHg5UmqDQ [0]: https://github.com/baowenbo/DAIN https://github.com/baowenbo/DAIN
loudmax 4y ago
Just for fun, I used ffmpeg's `minterpolate` (https://ffmpeg.org/ffmpeg-filters.html#minterpolate https://ffmpeg.org/ffmpeg-filters.html#minterpolate) feature to raise the framerate of my Mad Max 2 Road Warrior DVD from 24 to 100 fps. The results were interesting. If you pause the video and step through one frame at a time, it's very clear which frames are from the original film and which frames were conjured by the algorithm. There's no AI involved, so the algorithm doesn't really understand background from foreground, and objects in the artificial frames are distorted and unnatural. But at 100 fps you don't notice this, or at least I didn't, and motion seemed very smooth and fluid. The soap opera effect definitely held. It didn't feel like watching a classic movie from 1982, it felt much more like watching a contemporary reality TV program. There was definitely a sense of realism, but the realism was not that you're witnessing an actual post-apocalyptic society clinging to survival. It was realistic in the sense that you felt like you're on a film set, and if you could just peek outside the frame you'd see George Miller and the camera and microphone operators with the lighting and rigging and make-up and costume departments hanging around the catering truck. A lot of people felt that HFR ruined Peter Jackson's The Hobbit series (or was one of the many contributing factors) because it broke the pretense of fantasy. In my experience raising the frame rate did the same for The Road Warrior. Those outlandish costumes looked even sillier at 100 fps, but man those stunts are incredible! You do get a visceral sense of just how dangerous a lot of those stunts are. It's a miracle that nobody was killed.

teraflop 4y ago

The "soap opera" effect is mainly a perceptual response to the high frame rate itself, even if there are no interpolation artifacts. That's why it has its name: soap operas are typically filmed at 60fps, with cameras that cameras one full frame every 1/60th of a second.

AstralStorm 4y ago
More specifically, the reaction to lack of blur. You can get 60 fps upconverted footage from motion interpolation to not exhibit the effect by adding copious gaussian motion blur. The same problem (plus perceived judder due to low fps) appears if you play a video on a screen equipped with correctly tuned strobe backlight. It's one of the reasons as to why simple kernel interpolation (blurmotion) with most kernels does not exhibit this effect either.
- ubercow13 4y ago
  Why would there be a lack of blur? The 24fps (say) footage will start with more blur as the shutter speed will be slower and this blur will remain when interpolated to 60fps. It would surely have a physically impossibly high amount of blur after interpolation.
  - dagmx 4y ago
    Interpolation doesn’t change shutter angle which is what causes motion blur to have a specific aesthetic look, so when interpolated it looks wrong for the frame rate. You’re right in that it doesn’t change the amount of blur, I’m not sure why the person you’re replying to thinks that. But it does change the aesthetic of the blur. Perhaps they’re thinking of the issue with the Hobbit where the higher frame rate at a given shutter angle (180 let’s say) that shortened the blur compared to a traditional 24 fps film. They adjusted this for the second hobbit film to leave the shutter open longer (not a 360 shutter, but something like a 270) to bring back some of that feeling.
taneq 4y ago
A similar thing happened with The Hobbit, iirc. It was shot in 60fps and everyone said the CGI was too obvious, I think we’ve just learned to associate high frame rates with computer games.
- cma 4y ago
  Hobbit was 48hz (to be a multiple of the more standard 24 letting it also work on 24-only projection systems I am assuming).
  - ace2358 4y ago
    And this was because there were two cameras for the whole 3D craze. One for each eye. Though I don’t think I saw the 48fps in cinemas in Australia
- breatheoften 4y ago
  My big problem with the high frame rate in the hobbit was the slack of camera stability -- I think whatever image stabilization solution they used performed very poorly for that film -- not sure what went wrong but always hypothesized it was related to the higher capture rate. It felt like the camera was always bouncing and moving around randomly in a way I found very displeasing...
- Gordonjcp 4y ago
  I have to admit, watching it in 48fps (pretty sure it was 48 and not 60) the CGI sequences did look very much like video game cutscenes.
leeoniya 4y ago
> The "soap opera" effect is mainly a perceptual response to the high frame rate itself, even if there are no interpolation artifacts. also, i think the 3:2 pulldown (24->29.97fps) and interlacing that typically causes jitter/blurring is absent at framerates that are even multiples (30->30 or 30->60)
modeless 4y ago
Sure, but most hand drawn animation is done at 12 FPS or lower. Even just upsampling from 12 to 24 would be awesome and wouldn't get into the territory where the soap opera effect is caused by the framerate itself.
- dagmx 4y ago
  This would change the acting choices the animator made however. to take my response from elsewhere , consider a single drawn frame impact that is held for 2 or more frames. That timing is integral to the performance. When you interpolate, it no longer holds the right amount so the performance changes. People may scoff at that, but it’s a huge deal for artists. It was hell when we had dailies and motion smoothing on one of the TVs turned on (if we weren’t using screening rooms). Suddenly all the notes were different and acting didn’t feel right.
- deleted 4y ago
  [deleted]
Gordonjcp 4y ago
Not quite! They capture one full frame every 1/30th of a second in the US and Japan, and 1/25th everywhere else. They capture one *field* every 1/50th of a second, and interlace it. The reason for this is that it was hard to scan a full 625 lines in 1/50th of a second, and it would have required ludicrous bandwidth for the AM transmitters of the day and massive output valves to drive the scan coils in TV sets which would then produced huge flyback spikes. So the decision was to scan 312.5 lines per field, offset slightly, with each pair of fields forming one 1/25th of a second frame. Telecine was easy in the UK because you could speed the film up a little and then each TV frame would be two fields per film frame. If you shoot 50i you can deinterlace to 25p by combining the two sets of scan lines from each field with a bit of clever processing to hide the "combing" caused by things moving between fields, or you can double it up to 50p by just copying each line out twice with a bit of clever processing to smooth out the steps. I could post some video clips showing this if you were interested. Of course modern digital TV cameras are capable of shooting 60p as you say, but anyone shooting "telenovela" stuff is going to be shooting 60i.
- warmcompress 4y ago
  Thank you!! Our contemporary 60hz world seems to have retconned broadcast video standards.
  - Gordonjcp 4y ago
    It gets worse because for NTSC colour they changed it to 59.94Hz so that the colour subcarrier could be an odd multiple of the scan rate and just fit into the 4MHz-wide channels. This is why all your video editing software has got those ballachey 59.94/24.97 modes. It gets worse because for telecine you need to need to show two frames of 24fps film for every three frames of video, so you need to slow your film down to 23.976fps so it fits. Ballache.

tehnub 4y ago

Frame timing is an integral, creative aspect of the medium, not something to be fixed. 1. https://www.idtech.com/blog/what-does-animating-on-ones-twos-and-threes-mean https://www.idtech.com/blog/what-does-animating-on-ones-twos... 2. https://www.awn.com/tooninstitute/lessonplan/timing.htm#:~:text=In%20film%20animation%20there%20are,)%20%2D%20it%20is%20called%20one's https://www.awn.com/tooninstitute/lessonplan/timing.htm#:~:t.... 3. Search for "A good example would be this cut, and especially the first four shots." in https://animetudes.com/2021/03/06/the-kanada-style-in-context/ https://animetudes.com/2021/03/06/the-kanada-style-in-contex...

modeless 4y ago
> Frame timing is an integral, creative aspect of the medium > not something to be fixed. The first does not imply the second in every case. There are many, many instances where a creator with unlimited resources would have chosen to drawn more frames. And now we can. We don't have to indiscriminately interpolate everything to 120 Hz, but I think it's clearly possible to make huge improvements to a whole lot of hand drawn animation.
- tehnub 4y ago
  Sure, maybe a creator can utilize AI interpolation as a tool. I understood your comment to be about applying AI interpolation to works already completed.
  - modeless 4y ago
    Both.
satvikpendem 4y ago
As someone who already interpolates media with SVP4, I remain unconvinced by those arguments. My media looks amazing running at 120 hz, I don't get headaches by jarring panning shots that look closer to a slideshow than a movie. I recently watched Avatar 2 in IMAX 3D with 48 FPS, and it looked incredible. If only all media in the future could let go of the arbitrary 24 frame limit, which was made a hundred years ago because filmmakers wanted to conserve their film usage.

dagmx 4y ago

Adding frames isn’t simply an act of interpolation when it comes to animation. You’d need to understand acting choices as well. The timing of motion and the importance of how long a frame is held for is important to how characters are portrayed and the impact of actions. Take a punch or a bounce. The time of contact may be one drawn frame but it may be on screen longer to sell it. What should interpolation do here? If if it smooths everything but the impact, then the impact will feel too sticky and lose its force. Similarly if it smooths the impact as well then it goes by too quick and it feels too weak. Animation is full of these choices and it’s a common problem in video games for example where animators animate at a given frame rate but games can run at variable sampling rates along their animation, causing unwanted artifacts of motion or ruin the performance.

ipsum2 4y ago

DAIN[0] is very popular for increasing framerate of videos, including animations. Examples: https://www.youtube.com/watch?v=IK-Q3EcTnTA https://www.youtube.com/watch?v=IK-Q3EcTnTA https://www.youtube.com/watch?v=unRdyQYwL5k https://www.youtube.com/watch?v=unRdyQYwL5k https://www.youtube.com/watch?v=pJFHg5UmqDQ https://www.youtube.com/watch?v=pJFHg5UmqDQ [0]: https://github.com/baowenbo/DAIN https://github.com/baowenbo/DAIN

loudmax 4y ago

Just for fun, I used ffmpeg's `minterpolate` (https://ffmpeg.org/ffmpeg-filters.html#minterpolate https://ffmpeg.org/ffmpeg-filters.html#minterpolate) feature to raise the framerate of my Mad Max 2 Road Warrior DVD from 24 to 100 fps. The results were interesting. If you pause the video and step through one frame at a time, it's very clear which frames are from the original film and which frames were conjured by the algorithm. There's no AI involved, so the algorithm doesn't really understand background from foreground, and objects in the artificial frames are distorted and unnatural. But at 100 fps you don't notice this, or at least I didn't, and motion seemed very smooth and fluid. The soap opera effect definitely held. It didn't feel like watching a classic movie from 1982, it felt much more like watching a contemporary reality TV program. There was definitely a sense of realism, but the realism was not that you're witnessing an actual post-apocalyptic society clinging to survival. It was realistic in the sense that you felt like you're on a film set, and if you could just peek outside the frame you'd see George Miller and the camera and microphone operators with the lighting and rigging and make-up and costume departments hanging around the catering truck. A lot of people felt that HFR ruined Peter Jackson's The Hobbit series (or was one of the many contributing factors) because it broke the pretense of fantasy. In my experience raising the frame rate did the same for The Road Warrior. Those outlandish costumes looked even sillier at 100 fps, but man those stunts are incredible! You do get a visceral sense of just how dangerous a lot of those stunts are. It's a miracle that nobody was killed.