3 ms·

Specification of ffmpeg commands can get quite complex, especially once you get to filters. I never understood why it hasn't grown a DSL to specify the input/o

Specification of ffmpeg commands can get quite complex, especially once you get to filters. I never understood why it hasn't grown a DSL to specify the input/output/transformation. Could be JSON-based for what I care. Then we could have nice things like easy programmatic generation of these scripts and schema verification (for each particular ffmpeg version). Also, when I looked at how LosslessCut (an Electron app) does the timeline preview thumbnails, it just calls the ffmpeg CLI app however many times it needs (once per thumbnail). With the number of heavy users of ffmpeg, I'm wondering how nobody has yet started a project to run ffmpeg in-memory as a server and avoid the process startup cost as I'm sure LosslessCut isn't the only one who does this.

matsz 2y ago
That'd be a great solution, I'm using ffmpeg in my personal media server app to generate thumbnails and transcode media on the fly and I was wondering that too. There are also a few things that I had to patch on top of ffmpeg (for example it doesn't do well with generating singular HLS packets on demand). Would be nice to have a pluggable architecture that can link multiple things together like that. Might implement this myself if I end up having the time for that.
- aargh_aargh 2y ago
  I only found this project focused on streaming but it's unmaintained (some summer of code or thesis project, IIRC): https://trac.ffmpeg.org/wiki/ffserver https://trac.ffmpeg.org/wiki/ffserver
  - matsz 2y ago
    Interesting that it uses a similar configuration syntax to Apache HTTPd.
gcr 2y ago
What do you mean? FFMpeg supports filters via its own DSL; see https://ffmpeg.guide/ https://ffmpeg.guide/ or https://ffmpeg.org/ffmpeg-filters.html https://ffmpeg.org/ffmpeg-filters.html It's not quite JSON and I have to look up the syntax every time I use it, but it's well-defined. The general syntax is: [input1][input2]... filterchain [output1]; [input3] filterchain [output2]; ... to specify chains of filters that link input streams to output streams. For example, the following: ffmpeg -i INPUT -vf " split [main][tmp]; [tmp] crop=iw:ih/2:0:0, vflip [flip]; [main][flip] overlay=0:H/2 " OUTPUT is doing something like this (python pseudocode): main,tmp = input.split(2) flip = (tmp .crop(iw, ih/2, 0, 0) .vflip()) out = overlay([main, flip], 0, H/2) Here's a more complicated example (visualization on https://ffmpeg.guide/graph/demo https://ffmpeg.guide/graph/demo ): ffmpeg -i ./long-video.mp4 -i ./background-music.mp3 -filter_complex "[0:v]trim=duration=30[out0]; [0:a]atrim=duration=15[out1]; [out1][1:a] amix=inputs=2:duration=longest: dropout_transition=2:weights=1 1: normalize=1 [out2]" -map "[out0]" -map "[out2]" ./shorter-video.mp4 Pseudocode: # input files (-i options) in0 = read("./long_video.mp4") in1 = read("./background_music.mp3") # filter graph out0 = in0.video.trim(duration=30) out1 = in0.audio.trim(duration=15) out2 = amix([out1, in1.audio], duration="longest", dropout_transition=2, weights=[1,1], normalize=1) # output stream mapping out_stream[0] = out0 out_stream[1] = out2
zoogeny 2y ago
Seems like writing a script wrapper around ffmpeg could easily solve the first problem. All it needs to do is to read in JSON/YAML/TOML/etc. then parse them into command line equivalents and call ffmpeg. That is a kind of script pretty much any coding LLM could generate. I was watching a streamer trying to use ffmpeg in a streaming configuration and it didn't seem to work very well in that case. However, if someone was belligerent enough, I would guess it would be possible to use libavformat/libavcodec directly to parse out frames from video. For example, the streamer "sphaerophoria" posted a short series where he built a video editor from scratch [1]. I believe the code he wrote could be used as the basis for a custom-built editor to extract thumbnails from videos. 1. https://www.youtube.com/watch?v=l_hD99zpPBY&ab_channel=sphaerophoria https://www.youtube.com/watch?v=l_hD99zpPBY&ab_channel=sphae...
imtringued 2y ago
If they are such a heavy user why don't they just use libavcodec directly?

matsz 2y ago

That'd be a great solution, I'm using ffmpeg in my personal media server app to generate thumbnails and transcode media on the fly and I was wondering that too. There are also a few things that I had to patch on top of ffmpeg (for example it doesn't do well with generating singular HLS packets on demand). Would be nice to have a pluggable architecture that can link multiple things together like that. Might implement this myself if I end up having the time for that.

aargh_aargh 2y ago
I only found this project focused on streaming but it's unmaintained (some summer of code or thesis project, IIRC): https://trac.ffmpeg.org/wiki/ffserver https://trac.ffmpeg.org/wiki/ffserver
- matsz 2y ago
  Interesting that it uses a similar configuration syntax to Apache HTTPd.

gcr 2y ago

What do you mean? FFMpeg supports filters via its own DSL; see https://ffmpeg.guide/ https://ffmpeg.guide/ or https://ffmpeg.org/ffmpeg-filters.html https://ffmpeg.org/ffmpeg-filters.html It's not quite JSON and I have to look up the syntax every time I use it, but it's well-defined. The general syntax is: [input1][input2]... filterchain [output1]; [input3] filterchain [output2]; ... to specify chains of filters that link input streams to output streams. For example, the following: ffmpeg -i INPUT -vf " split [main][tmp]; [tmp] crop=iw:ih/2:0:0, vflip [flip]; [main][flip] overlay=0:H/2 " OUTPUT is doing something like this (python pseudocode): main,tmp = input.split(2) flip = (tmp .crop(iw, ih/2, 0, 0) .vflip()) out = overlay([main, flip], 0, H/2) Here's a more complicated example (visualization on https://ffmpeg.guide/graph/demo https://ffmpeg.guide/graph/demo ): ffmpeg -i ./long-video.mp4 -i ./background-music.mp3 -filter_complex "[0:v]trim=duration=30[out0]; [0:a]atrim=duration=15[out1]; [out1][1:a] amix=inputs=2:duration=longest: dropout_transition=2:weights=1 1: normalize=1 [out2]" -map "[out0]" -map "[out2]" ./shorter-video.mp4 Pseudocode: # input files (-i options) in0 = read("./long_video.mp4") in1 = read("./background_music.mp3") # filter graph out0 = in0.video.trim(duration=30) out1 = in0.audio.trim(duration=15) out2 = amix([out1, in1.audio], duration="longest", dropout_transition=2, weights=[1,1], normalize=1) # output stream mapping out_stream[0] = out0 out_stream[1] = out2

zoogeny 2y ago

Seems like writing a script wrapper around ffmpeg could easily solve the first problem. All it needs to do is to read in JSON/YAML/TOML/etc. then parse them into command line equivalents and call ffmpeg. That is a kind of script pretty much any coding LLM could generate. I was watching a streamer trying to use ffmpeg in a streaming configuration and it didn't seem to work very well in that case. However, if someone was belligerent enough, I would guess it would be possible to use libavformat/libavcodec directly to parse out frames from video. For example, the streamer "sphaerophoria" posted a short series where he built a video editor from scratch [1]. I believe the code he wrote could be used as the basis for a custom-built editor to extract thumbnails from videos. 1. https://www.youtube.com/watch?v=l_hD99zpPBY&ab_channel=sphaerophoria https://www.youtube.com/watch?v=l_hD99zpPBY&ab_channel=sphae...

imtringued 2y ago

If they are such a heavy user why don't they just use libavcodec directly?