3 ms·

Considering how ubiquitous it is, it's a mystery how and why there's so little documentation about how to use the library versions libavcodec and libavformat.

rockwotj 5y ago
I think you're supposed to read the header files? I have no idea how people write ffmpeg stuff. The only good tutorial I've seen is: https://github.com/leandromoreira/ffmpeg-libav-tutorial https://github.com/leandromoreira/ffmpeg-libav-tutorial
- marcan_42 5y ago
  Part of the problem is the API kept breaking every other version; ffmpeg is an amazing project for users but has historically been a nightmare to integrate with and package, for this reason. It's gotten better in the past few years, though, but there's still a lot of legacy codebases that are useless today because they won't build against modern ffmpeg any more (even though the rest of it is fine).
  - _Gyan_ 5y ago
    API/ABI is meant to be preserved within a major version, but not across. The deprecation period is typically two years, but deletion is usually much later. Do you have examples to the contrary from the past few years?
- oefrha 5y ago
  Header files (or Doxygen), and the few official examples. Some of them use old and no longer recommended APIs/patterns, and have obvious problems as soon as you compile and use them without changing anything. I say this as someone who has extended and maintained a libav* language binding with similarly terrible documentation. I tried to write more examples for people. But the API surface is vast and resources are extremely scarce. Sometimes you just have to dig into actual production code of FFmpeg (the CLI side), mpv, etc. to see how other people use it.
- ReactiveJelly 5y ago
  The headers are actually pretty good. I learned to use libav by trial and error over years, and now I'm at the point where I can just look at the headers for a refresher whenever I use it. At a very broad level: - Encoding and decoding are both pipelines - Codecs and formats are both part of a pipeline - An instance of a codec or a format is called a _context_ - The two basic operations for any context are "put data in" and "pull data out" Putting this all together, a lot of tutorials present this as a case of "Just push the input data in, then pull the output data out!" In my experience this is backwards. This leads to "pipeline buckling" where, e.g., you pull a compressed packet from the demuxer and try to push it into the decoder, and the decoder says "hey, slow down, my buffers are full". You have to just awkwardly stick that packet in your pocket until the decoder's buffers are dry again. I have always had better luck pulling data. e.g. If you're decoding, first try to pull a decompressed YUV frame from the decoder. If that fails, then try to get a new packet from the demuxer. Only if that fails (and you're not using the default file I/O), then feed the demuxer another buffer. This way it never buckles. It results in a few redundant "Can you give me anything?" checks when warming up the pipeline, but you never have buffers awkwardly outliving a loop iteration. If a buffer enters your control, it's always because you already found an empty spot in the next stage to immediately dump it into.
  - marcan_42 5y ago
    In my experience, the problem with this is you're still going to run into issues with A/V sync if the input is muxed weird. You can end up having to buffer a ton of A or V before you get the packet for the timestamp in the present. I have an old project using fixed size packet buffers between the demuxer and decoders, and for some inputs it just deadlocks as one side gets full. My workaround to avoid having to buffer a potentially infinite amount of data was to instantiate the demuxer twice, once for A and once for V. Then they can run out of sync with each other in terms of the muxing, and you can keep them in sync in terms of presentation timestamps.
- taneq 5y ago
  We use libffmpeg to stream live video from RTSP CCTV cameras and the main sources I used were https://github.com/mpenkov/ffmpeg-tutorial https://github.com/mpenkov/ffmpeg-tutorial and https://natnoob.blogspot.com.au/2011/04/modify-api-example-to-encode-h264.html https://natnoob.blogspot.com.au/2011/04/modify-api-example-t...
charcircuit 5y ago
I agree. Unless you had a lot of domain knowledge or a lot of time with a lot of weirdly encoded media I would say that it is impossible for someone to make something as seemingly simple as a thumbnail extractor with those libraries (eg. take the first frame or x seconds of a video and write an image file) Your best bet is to just call ffmpeg to do what you need and not have to worry about the weirdness of the libav stuff. Also, stuff like parsing a buffer of memory requires you to write a file driver since the code for libav is tightly coupled to ffmpeg which only handles files and not buffers.
- bick_nyers 5y ago
  I've been struggling with exactly this, setting up ffmpeg to work with buffers instead of files. I'm about ready to just write frames to /tmp or maybe /dev/shm and invoke the CLI. I work with a lot of weirdly encoded media, so I'm thinking that for long-term maintainability it might be better to use the CLI anyways, as you can iterate significantly faster at the cost of control. I'm just not sure what kind of performance overhead it will have writing to a ramdisk vs. using a buffer, as it's a non-negligible amount of memory to read/write so many extra times.
  - charcircuit 5y ago
    >setting up ffmpeg to work with buffers instead of files I'd take a look at https://github.com/FFmpeg/FFmpeg/blob/master/tools/target_dem_fuzzer.c https://github.com/FFmpeg/FFmpeg/blob/master/tools/target_de... In io_seek I'd also clamp pos at the end to be between 0 and the filesize. That might be needed for some files. It doesn't hurt though. >I'm thinking that for long-term maintainability it might be better to use the CLI anyways It's definitely easier that way. You don't have to deal with gotchas like having to set some random flags or needing to specify a filename. It's just so easy to write code that fails on 1% of files which ffmpeg can actually decode.
mikaraento 5y ago
FWIW, I created a production batch transcoding backend using python to construct ffmpeg command lines. If you don't need streaming you may not need to talk to the libraries. (The transcoding was for transforming DVB-T captures to multi-bitrate HLS and MS Smooth Streaming formats as part of a Cloud PVR. It was a separate backup pipeline in case the commercial streaming pipeline failed, which tended to happen a lot initially.)

rockwotj 5y ago

I think you're supposed to read the header files? I have no idea how people write ffmpeg stuff. The only good tutorial I've seen is: https://github.com/leandromoreira/ffmpeg-libav-tutorial https://github.com/leandromoreira/ffmpeg-libav-tutorial

marcan_42 5y ago
Part of the problem is the API kept breaking every other version; ffmpeg is an amazing project for users but has historically been a nightmare to integrate with and package, for this reason. It's gotten better in the past few years, though, but there's still a lot of legacy codebases that are useless today because they won't build against modern ffmpeg any more (even though the rest of it is fine).
- _Gyan_ 5y ago
  API/ABI is meant to be preserved within a major version, but not across. The deprecation period is typically two years, but deletion is usually much later. Do you have examples to the contrary from the past few years?
oefrha 5y ago
Header files (or Doxygen), and the few official examples. Some of them use old and no longer recommended APIs/patterns, and have obvious problems as soon as you compile and use them without changing anything. I say this as someone who has extended and maintained a libav* language binding with similarly terrible documentation. I tried to write more examples for people. But the API surface is vast and resources are extremely scarce. Sometimes you just have to dig into actual production code of FFmpeg (the CLI side), mpv, etc. to see how other people use it.
ReactiveJelly 5y ago
The headers are actually pretty good. I learned to use libav by trial and error over years, and now I'm at the point where I can just look at the headers for a refresher whenever I use it. At a very broad level: - Encoding and decoding are both pipelines - Codecs and formats are both part of a pipeline - An instance of a codec or a format is called a _context_ - The two basic operations for any context are "put data in" and "pull data out" Putting this all together, a lot of tutorials present this as a case of "Just push the input data in, then pull the output data out!" In my experience this is backwards. This leads to "pipeline buckling" where, e.g., you pull a compressed packet from the demuxer and try to push it into the decoder, and the decoder says "hey, slow down, my buffers are full". You have to just awkwardly stick that packet in your pocket until the decoder's buffers are dry again. I have always had better luck pulling data. e.g. If you're decoding, first try to pull a decompressed YUV frame from the decoder. If that fails, then try to get a new packet from the demuxer. Only if that fails (and you're not using the default file I/O), then feed the demuxer another buffer. This way it never buckles. It results in a few redundant "Can you give me anything?" checks when warming up the pipeline, but you never have buffers awkwardly outliving a loop iteration. If a buffer enters your control, it's always because you already found an empty spot in the next stage to immediately dump it into.
- marcan_42 5y ago
  In my experience, the problem with this is you're still going to run into issues with A/V sync if the input is muxed weird. You can end up having to buffer a ton of A or V before you get the packet for the timestamp in the present. I have an old project using fixed size packet buffers between the demuxer and decoders, and for some inputs it just deadlocks as one side gets full. My workaround to avoid having to buffer a potentially infinite amount of data was to instantiate the demuxer twice, once for A and once for V. Then they can run out of sync with each other in terms of the muxing, and you can keep them in sync in terms of presentation timestamps.
taneq 5y ago
We use libffmpeg to stream live video from RTSP CCTV cameras and the main sources I used were https://github.com/mpenkov/ffmpeg-tutorial https://github.com/mpenkov/ffmpeg-tutorial and https://natnoob.blogspot.com.au/2011/04/modify-api-example-to-encode-h264.html https://natnoob.blogspot.com.au/2011/04/modify-api-example-t...

charcircuit 5y ago

I agree. Unless you had a lot of domain knowledge or a lot of time with a lot of weirdly encoded media I would say that it is impossible for someone to make something as seemingly simple as a thumbnail extractor with those libraries (eg. take the first frame or x seconds of a video and write an image file) Your best bet is to just call ffmpeg to do what you need and not have to worry about the weirdness of the libav stuff. Also, stuff like parsing a buffer of memory requires you to write a file driver since the code for libav is tightly coupled to ffmpeg which only handles files and not buffers.

bick_nyers 5y ago
I've been struggling with exactly this, setting up ffmpeg to work with buffers instead of files. I'm about ready to just write frames to /tmp or maybe /dev/shm and invoke the CLI. I work with a lot of weirdly encoded media, so I'm thinking that for long-term maintainability it might be better to use the CLI anyways, as you can iterate significantly faster at the cost of control. I'm just not sure what kind of performance overhead it will have writing to a ramdisk vs. using a buffer, as it's a non-negligible amount of memory to read/write so many extra times.
- charcircuit 5y ago
  >setting up ffmpeg to work with buffers instead of files I'd take a look at https://github.com/FFmpeg/FFmpeg/blob/master/tools/target_dem_fuzzer.c https://github.com/FFmpeg/FFmpeg/blob/master/tools/target_de... In io_seek I'd also clamp pos at the end to be between 0 and the filesize. That might be needed for some files. It doesn't hurt though. >I'm thinking that for long-term maintainability it might be better to use the CLI anyways It's definitely easier that way. You don't have to deal with gotchas like having to set some random flags or needing to specify a filename. It's just so easy to write code that fails on 1% of files which ffmpeg can actually decode.

mikaraento 5y ago

FWIW, I created a production batch transcoding backend using python to construct ffmpeg command lines. If you don't need streaming you may not need to talk to the libraries. (The transcoding was for transforming DVB-T captures to multi-bitrate HLS and MS Smooth Streaming formats as part of a Cloud PVR. It was a separate backup pipeline in case the commercial streaming pipeline failed, which tended to happen a lot initially.)