6 ms·

Non-Rectangular Video Cropping with FFMpeg

68 points by lipeltgm 7y ago

rejschaap 7y ago
It has some aliasing artefacts around the edge of the circle, you can see it very clearly during the fade-in. When the video is playing you don't really notice it. My guess is that this is caused by the way the mask is created. I can't try it out right now, but it could probably be solved by setting the RGB channels of the mask image to black.
deleted 7y ago
[deleted]
ArtWomb 7y ago
My new years resolution might just be to gain mastery of ffmpeg. It's incredibly versitile. Combine with python bindings, image magick and opencv and you have a powerful weapon in your arsenal. But I keep running out of memory. I like that the process just dies. And can usually be solved by sequential batching. But how can I automate via predictive instrumentation?
- the8472 7y ago
  ffmpeg can operate in on streams of frames, you don't have to keep everything in memory or operate on discrete batches.
- BubRoss 7y ago
  I don't know what you mean by predictive instrumentation or why you are running out of memory, but don't forget the original interprocess communication, the file system.
  - ArtWomb 7y ago
    Thanks for the replies the8472 and BubRoss! I just mean I'd like to "cap" ffmpeg's memory usage to ensure it never crashes. I'll provide a simple example. Creating an image slideshow from a directory of huge images. I can downscale and compress to jpeg before processing. But even then, besides batching, it always fails. And this is just raw, without any filters or image processing. Have tried all the i/o techniques from the wiki: https://trac.ffmpeg.org/wiki/Slideshow https://trac.ffmpeg.org/wiki/Slideshow All this local laptop based (4GB). I just want to get a better handle before devoting cloud resources (ffmpeg is standard on gcloud serverless) ;)
    - AstralStorm 7y ago
      Sounds like there's a lack of feature in the CLI tool. You will have to write your own, pick a language and it probably has enough of low level ffmpeg bindings to do this. The library supports streaming reads and writes very well, though you may have to provide a custom reader callback or use the old loop APIs.
      - BubRoss 7y ago
        ffmpeg from the command line is very powerful and people use it for what is being described here all the time. Why do you think it is necessary to have to resort to the library and API?
    - kohanz 7y ago
      You should be able to, for example, divide the slideshow into smaller segments, create those and then use the concatenate ability to join them back together. Maybe that's what you were already referring to by 'batching'? It's a completely valid approach.
    - shakna 7y ago
      I think there's something vital missing from this story. I regularly use a laptop with 4GB RAM to generate videos using the sequential method listed there, with 4-8k sized frames, producing video files of hundreds of GB, without a crash. There can be hundreds of thousands of them as well. No batching required. The final command in my pipeline is a simple: ffmpeg \ -threads "$threads" \ -y \ -start_number 0 \ -i 'build_frames'"$unique"'/%09d.png' \ -c:v libvpx-vp9 \ -lossless 1 \ -qscale:v 2 \ -r "$fps" \ build_tmp"$unique"/media.webm But, if you really need to limit ffmpeg's memory consumption, you probably need to look at -max_alloc and -bits_per_raw_sample. It'll be highly specific to your own hardware.
      - ArtWomb 7y ago
        Thnks for the replies all ;) Switching to vp9 (from x264) worked for me!
      - 7y ago
- pselbert 7y ago
  Learning ffmpeg can be daunting but it is a highly useful tool to know. The option intricacies and peculiarities make it extremely hard to “master” though. It requires so much experimentation and experience (trial and error) that achieving mastery in a year seems nigh impossible!
  - deleted 7y ago
    [deleted]
_Gyan_ 7y ago
See https://superuser.com/questions/1270950 https://superuser.com/questions/1270950 for a way to do it without needing to create a masking image.
- lipeltgm 7y ago
  Agreed. The 'gex' filter can readily accomplish the masking if the shape can be represented as an X,Y equation. This method does however work for a more comprehensive list of shapes, titled wrt circular since that seems to be a highly desired form of <i>cropping</i>. Thank you for pointing it out and likely would address the aliasing another reader referenced.
the8472 7y ago
Note that a few video codecs also support alpha these days, which could be used to preserve the mask and overlay the video over some background. It even works on the web: https://files.catbox.moe/uu2ze0.webm https://files.catbox.moe/uu2ze0.webm (needs a non-black background)
- jpincheira 7y ago
  Wow. Pretty cool. Had no idea about this.
- izietto 7y ago
  here with a yellow background (edit: changed to a yellow-to-green circle-shaped radial gradient because why not): https://codepen.io/mdesantis/pen/RwNZEZJ https://codepen.io/mdesantis/pen/RwNZEZJ
  - pbhjpbhj 7y ago
    Maybe, https://codepen.io/pbhj/pen/dyPzwEj https://codepen.io/pbhj/pen/dyPzwEj instead - centre the video and put the gradient on the body to emphasise the matting?
- pbhjpbhj 7y ago
  Firefox (Linux): I had to open dev tools, remove the bg-image that Firefox shows around media links, and then adjusting the bg-color showed the effect. Otherwise it just looked like a normal video.
vernie 7y ago
I think it's more appropriate to call this matting. I came in thinking they'd figured out how to coax FFmpeg into encoding non-rectangular frame geometries.
GJR 7y ago
Fill and key is basic video manipulation. As to the aliasing, they also need to apply some filtering...
batt4good 7y ago
Is there any way to provide a dynamic mask? For instance, if I wanted to blur a dynamic portion of each fram, like a face or logo? I've been trying to find higher-performance methods of doing this with live footage that don't require openCV. Seems like FFmpeg doesn't have a great way to do this or I just haven't found it yet? I guess I'm also looking for frameworks or tools that can handle live video "manipulation" like rendering shapes or graphics. OpenCV is painfully slow for most of this stuff, everything else is seemingly completely proprietary. But nVidia seems to have some incredible tools baked into their latest stream manipulation API's.
rambojazz 7y ago
Some day someone will write a simpler replacement for ffmpeg's user interface. A tool that you have to read the manual every time you want to write (or even just understand) a command is not a good tool.
- _Gyan_ 7y ago
  The CLI syntax design reflects the N-inputs --> N-outputs capability and no. of options reflects the number of components e.g. there are 300+ demuxers and 400+ filters available. This is not to say there isn't scope for improvement. A couple of areas for improvement is the consistency of options naming across filters that implement the same feature within those filters, and a clearer syntax to identify the target of an option i.e. protocol, demuxer, decoder..etc
  - the8472 7y ago
    perhaps a declarative config file instead of CLI arguments would help. With a config file you could lay things out in a more tree-like manner, have lists etc. And it's not just N inputs, M outputs. You also have a processing graph for the intermediate filters. So ideally you want graph nodes represented by some identifier which can then be referenced by other nodes.
    - _Gyan_ 7y ago
      You also have a processing graph for the intermediate filters. So ideally you want graph nodes represented by some identifier which can then be referenced by other nodes. Unless I misunderstand you, this is already the case. Simple or complex filtergraphs are declared as a string* by the user as an arg to -vf/-af (per-output) or -filter_complex/-lavfi (global). Filter outputs can have link labels, which allow their consumption by other filters. *can also be read from a file
      - the8472 7y ago
        Yeah, but that's just part of the overall parameters. with a structured file format (toml, yaml, json... pick your poison) the filter arguments could also be structured themselves instead of having to be a single string, just another nesting level among the other params. The shell only so many ways of structuring things until you run out of seprator chars. And already has to deal with the impedance mismatch of translating space-separated args into an array of zero-terminated strings which results in its own escaping rules.

68 points by lipeltgm 7y ago

It has some aliasing artefacts around the edge of the circle, you can see it very clearly during the fade-in. When the video is playing you don't really notice it. My guess is that this is caused by the way the mask is created. I can't try it out right now, but it could probably be solved by setting the RGB channels of the mask image to black.

deleted 7y ago

[deleted]

ArtWomb 7y ago

My new years resolution might just be to gain mastery of ffmpeg. It's incredibly versitile. Combine with python bindings, image magick and opencv and you have a powerful weapon in your arsenal. But I keep running out of memory. I like that the process just dies. And can usually be solved by sequential batching. But how can I automate via predictive instrumentation?

the8472 7y ago
ffmpeg can operate in on streams of frames, you don't have to keep everything in memory or operate on discrete batches.
BubRoss 7y ago
I don't know what you mean by predictive instrumentation or why you are running out of memory, but don't forget the original interprocess communication, the file system.
- ArtWomb 7y ago
  Thanks for the replies the8472 and BubRoss! I just mean I'd like to "cap" ffmpeg's memory usage to ensure it never crashes. I'll provide a simple example. Creating an image slideshow from a directory of huge images. I can downscale and compress to jpeg before processing. But even then, besides batching, it always fails. And this is just raw, without any filters or image processing. Have tried all the i/o techniques from the wiki: https://trac.ffmpeg.org/wiki/Slideshow https://trac.ffmpeg.org/wiki/Slideshow All this local laptop based (4GB). I just want to get a better handle before devoting cloud resources (ffmpeg is standard on gcloud serverless) ;)
  - AstralStorm 7y ago
    Sounds like there's a lack of feature in the CLI tool. You will have to write your own, pick a language and it probably has enough of low level ffmpeg bindings to do this. The library supports streaming reads and writes very well, though you may have to provide a custom reader callback or use the old loop APIs.
    - BubRoss 7y ago
      ffmpeg from the command line is very powerful and people use it for what is being described here all the time. Why do you think it is necessary to have to resort to the library and API?
  - kohanz 7y ago
    You should be able to, for example, divide the slideshow into smaller segments, create those and then use the concatenate ability to join them back together. Maybe that's what you were already referring to by 'batching'? It's a completely valid approach.
  - shakna 7y ago
    I think there's something vital missing from this story. I regularly use a laptop with 4GB RAM to generate videos using the sequential method listed there, with 4-8k sized frames, producing video files of hundreds of GB, without a crash. There can be hundreds of thousands of them as well. No batching required. The final command in my pipeline is a simple: ffmpeg \ -threads "$threads" \ -y \ -start_number 0 \ -i 'build_frames'"$unique"'/%09d.png' \ -c:v libvpx-vp9 \ -lossless 1 \ -qscale:v 2 \ -r "$fps" \ build_tmp"$unique"/media.webm But, if you really need to limit ffmpeg's memory consumption, you probably need to look at -max_alloc and -bits_per_raw_sample. It'll be highly specific to your own hardware.
    - ArtWomb 7y ago
      Thnks for the replies all ;) Switching to vp9 (from x264) worked for me!
    - 7y ago
pselbert 7y ago
Learning ffmpeg can be daunting but it is a highly useful tool to know. The option intricacies and peculiarities make it extremely hard to “master” though. It requires so much experimentation and experience (trial and error) that achieving mastery in a year seems nigh impossible!
- deleted 7y ago
  [deleted]

_Gyan_ 7y ago

See https://superuser.com/questions/1270950 https://superuser.com/questions/1270950 for a way to do it without needing to create a masking image.

lipeltgm 7y ago
Agreed. The 'gex' filter can readily accomplish the masking if the shape can be represented as an X,Y equation. This method does however work for a more comprehensive list of shapes, titled wrt circular since that seems to be a highly desired form of <i>cropping</i>. Thank you for pointing it out and likely would address the aliasing another reader referenced.

the8472 7y ago

Note that a few video codecs also support alpha these days, which could be used to preserve the mask and overlay the video over some background. It even works on the web: https://files.catbox.moe/uu2ze0.webm https://files.catbox.moe/uu2ze0.webm (needs a non-black background)

jpincheira 7y ago
Wow. Pretty cool. Had no idea about this.
izietto 7y ago
here with a yellow background (edit: changed to a yellow-to-green circle-shaped radial gradient because why not): https://codepen.io/mdesantis/pen/RwNZEZJ https://codepen.io/mdesantis/pen/RwNZEZJ
- pbhjpbhj 7y ago
  Maybe, https://codepen.io/pbhj/pen/dyPzwEj https://codepen.io/pbhj/pen/dyPzwEj instead - centre the video and put the gradient on the body to emphasise the matting?
pbhjpbhj 7y ago
Firefox (Linux): I had to open dev tools, remove the bg-image that Firefox shows around media links, and then adjusting the bg-color showed the effect. Otherwise it just looked like a normal video.

vernie 7y ago

I think it's more appropriate to call this matting. I came in thinking they'd figured out how to coax FFmpeg into encoding non-rectangular frame geometries.

GJR 7y ago

Fill and key is basic video manipulation. As to the aliasing, they also need to apply some filtering...

batt4good 7y ago

Is there any way to provide a dynamic mask? For instance, if I wanted to blur a dynamic portion of each fram, like a face or logo? I've been trying to find higher-performance methods of doing this with live footage that don't require openCV. Seems like FFmpeg doesn't have a great way to do this or I just haven't found it yet? I guess I'm also looking for frameworks or tools that can handle live video "manipulation" like rendering shapes or graphics. OpenCV is painfully slow for most of this stuff, everything else is seemingly completely proprietary. But nVidia seems to have some incredible tools baked into their latest stream manipulation API's.

rambojazz 7y ago

Some day someone will write a simpler replacement for ffmpeg's user interface. A tool that you have to read the manual every time you want to write (or even just understand) a command is not a good tool.

_Gyan_ 7y ago
The CLI syntax design reflects the N-inputs --> N-outputs capability and no. of options reflects the number of components e.g. there are 300+ demuxers and 400+ filters available. This is not to say there isn't scope for improvement. A couple of areas for improvement is the consistency of options naming across filters that implement the same feature within those filters, and a clearer syntax to identify the target of an option i.e. protocol, demuxer, decoder..etc
- the8472 7y ago
  perhaps a declarative config file instead of CLI arguments would help. With a config file you could lay things out in a more tree-like manner, have lists etc. And it's not just N inputs, M outputs. You also have a processing graph for the intermediate filters. So ideally you want graph nodes represented by some identifier which can then be referenced by other nodes.
  - _Gyan_ 7y ago
    You also have a processing graph for the intermediate filters. So ideally you want graph nodes represented by some identifier which can then be referenced by other nodes. Unless I misunderstand you, this is already the case. Simple or complex filtergraphs are declared as a string* by the user as an arg to -vf/-af (per-output) or -filter_complex/-lavfi (global). Filter outputs can have link labels, which allow their consumption by other filters. *can also be read from a file
    - the8472 7y ago
      Yeah, but that's just part of the overall parameters. with a structured file format (toml, yaml, json... pick your poison) the filter arguments could also be structured themselves instead of having to be a single string, just another nesting level among the other params. The shell only so many ways of structuring things until you run out of seprator chars. And already has to deal with the impedance mismatch of translating space-separated args into an array of zero-terminated strings which results in its own escaping rules.