9 ms·
FFmpeg is getting multithreaded transcoding pipelines
- jbk 3y agoIt's difficult to understand what this is about without the presentation from Anton, at VideoLAN Dev Days 2023, that you can watch here: https://www.youtube.com/watch?v=Z4DS3jiZhfo&t=1221 https://www.youtube.com/watch?v=Z4DS3jiZhfo&t=1221
- bobsmooth 3y agoWhat is difficult to understand? Does multithreaded mean something different in the realm of video transcoding?
- bootloop 3y ago> "multi-threading it's not really all about just multi-threading - I will say more about that later - but that is the marketable term" That's whats said in the video at least in the first 10 seconds so it might be that multi-threading is just a too trivial term for the work here. (But haven't watched the video yet so just an observation.)
- rolandog 3y agoIf I recall correctly, he meant that multi-threading wasn't a direct goal, but a "side effect" of rewriting some parts of the codebase to match the expected conceptual flow, and paying off some re-write debt that had been accumulated after years of new features being "tacked on".
- defrost 3y agoThere are many different types of pipelined processing tasks with many differing kinds of threading approaches, and I guess the video clears up what kinds of approaches work best with transcoding ..
- t43562 3y agoHis work is not primarily about multithreading but about cleaning up ffmpeg to be true to its own architecture so that normal human beings have a chance of being able to maintain it. Things like making data flow one way in a pipeline, separating public and private state and having clearly defined interfaces. Things had got so bad that every change was super difficult to make. Multithreading comes out as a natural benefit of the cleanup.
- pjc50 3y ago^ This: I just spent the ~20 minutes necessary to watch that part of the talk at a reasonable 1.5x speed, and that's the summary. Ffmpeg was suffering from 20 years of incremental change and lack of cleanup/refactoring, and that's what he's spent two years doing. A couple of great lines including "my test for deprecating an option is if it's been broken for years and nobody is complaining, then definitely nobody is using it".
- andrewstuart 3y agoI often look for obscure options to get specialists tasks done in all sorts of software. Better to fix the option. Just because I didn’t go to the very significant effort of complaining doesn’t mean I didn’t want and burn hours trying to use that option.
- pjc50 3y agoThe verdict of the presentation was that many options are (bad) duplicates of the filter graph, and you should configure the software through the filter graph. We saw in openssl what the consequences of never removing any code for decades were. It has a real cost. Always a case-by-case decision, though.
- Zagitta 3y agoYou can't expect open source developers to be omniscient and know you want to use a specific feature if you don't communicate that to them. Would you rather have them add telemetry?
- gbersac 3y agoI love this guy! He's a very talented guy who devoted his life to open source.
- lofaszvanitt 3y agoHope he doesn't have to beg for donations.
- gooseus 3y agoNice, great presentation! Curious what he has in mind for the "dynamic pipelines" and "scripting (Lua?)" he mentions in the "Future directions" section. I'm imagining something more powerful for animating properties?
- dylan604 3y agoMan, i really want to watch this presentation, but the piss poor audio just causes my brain to have a fit. How in today's time is this still possible to screw up so badly?
- jugad 3y agoWhat are you talking about... the audio might not professional studio level 10/10, but I don't see anything significantly wrong with it - given that its more like a standard presentation mic. Its clearly good enough.
- dylan604 3y agoEvery time he turns away from the mic and continues talking while looking at the projection his volume goes way down and at best sounds like a mumble. It is very taxing to keep up with him when he's turned away. It's the wrong mic for the task.
- wildekek 3y agoAgreed. Maybe some are not as sensitive to this, but it is a major energy suck for me. A little post-processing on noise and compression would come a long way. This recording is as raw as the Ramsey meme.
- jbk 3y agoWe're very happy for your volunteering to record our next developer conference, tickets are free!
- dylan604 3y agoif you weren't on a different continent separated by a very large body of water, I'd be there. I'll donate by suggesting the use of a lav mic vs a podium mic. It is very difficult for some people to be able to understand clearly voices that are muddled from off axis audio recording. It's a real condition. I have hard time hearing voices in a crowded room from people across the table from me. We spend time worrying about the aria tags in our mark up, but we just assume that everyone has the same hearing abilities? I get that most people probably don't think about this when they don't have a hearing condition, but to be dismissive about it is an entirely different level of egregiousness. Could my initial criticism have been provided with an entirely different tact, absolutely. But after the mental exhaustion that video was, that was all the energy I could afford at the time.
- aidenn0 3y agoOh, I've run into so many issues related to the "Extras" listed on the slide at ~33m into that video.
- pjc50 3y agoTweet just links to http://ffmpeg.org/pipermail/ffmpeg-devel/2023-November/316552.html http://ffmpeg.org/pipermail/ffmpeg-devel/2023-November/31655... .. which in turn references code at https://git.khirnov.net/libav.git/log/?h=ffmpeg_threading https://git.khirnov.net/libav.git/log/?h=ffmpeg_threading
- kierank 3y agoThe tweet links to the mailing list which is the official source of the patchset (you can choose "Next in Thread" to continue).
- Alifatisk 3y agoI’ve been looking for more ways to speedup the transcoding process, one solution I found was using gpu acceleration, another was using more threads but its hard to find the optimal amount I should provide.
- marcyb5st 3y agoCan't you just use Hyperparameter Optimization to find the best value? Tools like Sherpa or Scikit-optimize can be used to explore a search space of n-threads/types of input/CPU type (which might be fixed on your machine).
- corndoge 3y agoI don't think "just" is appropriate here, that makes it sound like this should be a trivial task for anyone while it is not. Using "just" like this minimizes work and makes people feel stupid which leads to various negative outcomes. Sorry for lecturing
- pjc50 3y agoFor most workloads, setting the number of threads to the number of vCPUs (i.e. count each hyperthreaded core as 2) works. But GPU acceleration is much better if it's available to you.
- cm2187 3y agoThough in my tests I found that gpu acceleration of video decoding actually hurts performance. It seems software decoding is faster than hardware for some codecs. Of course not the case for encoding.
- themoonisachees 3y agoThat heavily depends on the GPU being used and whether or not it has hardware support for your codec. Maybe your GPU is just old/weak compared to your CPU?
- raphaelj 3y agoAbout 2x faster on my 4-cores ARM server, without any significant parallelism overhead: $ time ffmpeg_threading/ffmpeg -i input.mp4 -ar 1000 -vn -acodec flac -f flac -y /dev/null -hide_banner -loglevel quiet 14.90s user 2.08s system 218% cpu 7.771 total $ time ffmpeg -i input.mp4 -ar 1000 -vn -acodec flac -f flac -y /dev/null -hide_banner -loglevel quiet 14.05s user 1.80s system 114% cpu 13.841 total
- cm2187 3y agoBut what part gets multi threading? Because the video compression is already multithreaded. Video decompression I am not sure. And I think anything else is fairly small in comparison in term of performance cost. All improvements are welcome but I would expect the impact to be fairly immaterial in practice.
- jamal-kumar 3y agoFrom what I understand the more you increase multithreading with ffmpeg the more you get quality issues... If they have started to solve that problem then I will be a much happier camper
- jbk 3y ago> From what I understand the more you increase multithreading with ffmpeg the more you get quality issues... No. If you don't max out your CPU, you will not have quality issues.
- raphaelj 3y agoWell, that's the very specific command I'm using in one of my webapps (https://datethis.app https://datethis.app), and it's one of the main performance hotspots, so it's very *not* immaterial.
- timvdalen 3y agoWow, I learned something today, did not know this was a thing!
- kjuulh 3y agoVery nice. Hopefully this will make my small transcoding needs faster for plex (as I don't have hardware transcoding support on my graphics card) =D
- keepamovin 3y agoCan anyone clarify the licensing requirements of large scale ffmpeg deployments? In what cases are fees required?
- hutzlibu 3y ago"FFmpeg is licensed under the GNU Lesser General Public License (LGPL) version 2.1 or later. However, FFmpeg incorporates several optional parts and optimizations that are covered by the GNU General Public License (GPL) version 2 or later. If those parts get used the GPL applies to all of FFmpeg. " http://ffmpeg.org/legal.html http://ffmpeg.org/legal.html Meaning it is free, but if you use some modules, you might have problems mixing it with proprietary code.
- bsenftner 3y agoThe general use case for ffmpeg inside proprietary software is the version of ffmpeg used needs to be statically compiled and linked into the software's executable, or it needs to be a separate executable called by the proprietary software.
- keepamovin 3y agoThanks. We're doing the second but I heard that at a certain scale you might need to pay fees anyway?
- bsenftner 3y agoI used to work at an FR video security company, where our product was in a significant percentage of the world's airports and high traffic hubs. Statically linked ffmpeg for the win.
- Daemon404 3y agoYou have that backwards - it must be dynamically linked. Static linking without providing your source would violate the LGPL.
- 3y ago
- amelius 3y agoIsn't video transcoding easily parallelizable? I mean just split the video into N equal parts (at keyframes) and divide the work.
- xuhu 3y agoNot a video encoding expert, but for live streams you can't merge the output until you process all the N parts, so you introduce delays. And if any part of the input pipeline, like an overlay containing a logo or text, is generated dynamically i.e. not a static mp4, it basically counts as a live stream.
- amelius 3y agoYes, good point about live streams.
- bambax 3y agoWhy not cut the image in rectangles and process those simultaneously? Wouldn't that work for live streams? (There may be artefacts at the seams though?)
- slimscsi 3y agoYes, and we do, but that is not the slow part: https://en.wikipedia.org/wiki/Motion_compensation https://en.wikipedia.org/wiki/Motion_compensation And as for seams: https://en.wikipedia.org/wiki/Deblocking_filter https://en.wikipedia.org/wiki/Deblocking_filter
- bambax 3y agoThanks! Very informative! It's really a fascinating topic.
- MaxikCZ 3y agoYou are right, but first you ideally should perform scene change detection (to put keyframes at propper positions), and that alone takes quite some processing.
- m3kw9 3y agoIs it automatic or does one need to do command line parameter gymnastics?
- sharkski 3y agoAwesome to see improvements to FFmpeg! I'm hoping to see Dolby AC4 support soon.
- jokoon 3y agoAre there any video editing software that take advantage of ffmpeg? I once thought about making something to draw geometry through SVG and use ffmpeg then, or maybe add some UI or whatever, or just to add text, but I never started. Avidemux feels like it's a bit that. Since ffmpeg internals are quite raw and not written to be accessed through a GUI, any video editor based on it would probably be quite clunky and weird and hard to maintain. Maybe an editor that use modules that just build some kind of preview with an command explainer, or some pipeline viewer. ffmpeg is quite powerful, but it's a bit stuck because it only works with a command line, which is fine, but I guess it somehow prevents it from being used by some people. I've already written a python script to take a random amount of clips, and build a mosaic with the xstack filter. It was not easy.
- deleted 3y ago[deleted]
- brucethemoose2 3y agoVapourSynth is intended to be a middle ground you might be seeking. You manipulate the video in Python instead of ffmpeg's CLI, but its often more extensible and powerful than pure ffmpeg due to the extensions: https://vsdb.top/ https://vsdb.top/ https://www.vapoursynth.com/ https://www.vapoursynth.com/ I have seen some niche software built on ffmpeg like losslesscut: https://github.com/mifi/lossless-cut https://github.com/mifi/lossless-cut Staxrip is also big: https://github.com/staxrip/staxrip https://github.com/staxrip/staxrip But I don't know anything "comprehensive."
- deleted 3y ago[deleted]
- jy14898 3y agohttps://shotcut.org/ https://shotcut.org/ also uses ffmpeg
- up_o 3y agoKDEnlive. Fantastic foss video editing software.
- 38 3y agonon crap link: http://farside.link/twitter.com/FFmpeg/status/1721275669336707152 http://farside.link/twitter.com/FFmpeg/status/17212756693367...
- not2b 3y agoProbably would have been better to link to https://ffmpeg.org/pipermail/ffmpeg-devel/2023-November/316552.html https://ffmpeg.org/pipermail/ffmpeg-devel/2023-November/3165... instead of the tweet (or the xit, or whatever they are called now), as the substance in the tweet is the link.
- Sabinus 3y agoI wonder if the X should be pronounced in the Chinese manner, as in Xi Jinping.
- lakpan 3y ago[dead]
- telotortium 3y ago> tweet (or the xit, or whatever they are called now) Just "post". They're no longer limited to 280 characters now either, although longer posts are collapsed by default.
- lost_tourist 3y agoStill limited to 280 unless you're paying for the check mark.
- goeiedaggoeie 3y agoYou could already place demuxing on a different thread as well