14 ms·
FFmpeg lands CLI multi-threading as its "most complex refactoring" in decades
- ElijahLynn 3y agoThis must have been quite the challenge to continually rebase the ongoing changes coming in on the daily. Wow. Now that it is actually in, it should be much easier to go forward. Big win too! This is going to really speed things up!
- anacrolix 3y agoRewrite it in Rust
- qrush 3y agoIs there a recording of this talk from VDD@Dublin? Can't easily find it on the author's site or here https://www.videolan.org/videolan/events/vdd23/ https://www.videolan.org/videolan/events/vdd23/ Update: Found here! https://www.youtube.com/watch?v=Z4DS3jiZhfo&t=1221s https://www.youtube.com/watch?v=Z4DS3jiZhfo&t=1221s
- jaquesy 3y agoAs someone who has only used ffmpeg for very trivial scenarios this was really interesting to listen to. It's nice knowing it's getting a proper refactor (not that I knew much of its internals until now)
- motoboi 3y agoIt’s nuts to think that in the near future LLM will be able to do that refactoring in seconds. All we need is enough context window.
- bigbillheck 3y agoWhy on earth would you possibly think that?
- ctoth 3y agoJust as your human intelligence lead to you writing the same darn comment as another human above you, AI can often write the same code as a human would, without having to even bring creativity into it! For those of us who write code, this can be useful!
- motoboi 3y agoI'm quite confused by the answers I got from this thread. Haven't you tried refactoring with gpt-4 yet?
- bigbillheck 3y ago> Haven't you tried refactoring with gpt-4 yet? I most certainly have not. At work, I do greenfield development in a specialized problem domain, and I would not trust a model (or, for that matter, a junior developer) to do any kind of refactor in an acceptable manner. (That aside, there's no way I'm goingto expose company code to any sort of outside party without the approval of upper management). At home, I program for fun and self-improvement, and a big part of both is thinking hard about problems. Why would I want to wreck that with asking a model to do it for me?
- motoboi 3y agoOh, I understand. What if you have an idea and you tell the computer to implement it and then thoroughly check the code? About the self-improvement part, I suppose you don't operate your graphics card yourself, but delegate it to your driver. The LLM is just another tool.
- smabie 3y agoOh yea your work is far too sophisticated for a llm, got it
- bigbillheck 3y agoSome of us do actual creative work, yes.
- bsdpufferfish 3y agoWhy would you think this is possible?
- motoboi 3y agobased on my current experience with gpt-4. Have you tried some sort of refactoring in it? Because I have been routinely turning serial scripts into parallel ones with success. Couldn't do the same with larger codebases because the context is not enough for the input code and output refactoring.
- thfuran 3y agoYeah, it's nuts to think that.
- motoboi 3y agoI'm genuinely confused about your point of view. Have you tried refactoring with GPT-4? I have been refactoring code using gpt-4 for some months now and the limiting factor have been the context size. GPT-4 turbo now have 128k context and I can provide it with larger portions of the code base for the refactors. When we have millions of tokens of context, based on what I'm experiencing now, I can see that a refactoring like the one made in ffmpeg would be possible. Or not? What am I missing here?
- rocqua 3y agoRefactoring is really rather well defined. It's " just transformations that are invariant w.r.t. the outcome". The reason they are hard to automate is that 'invariant w.r.t. the outcome' is a lot more lenient than most semantic models van handle. But this kind of well-defined task with a slight amount of nuance (and decently checkable) seems pretty well-suited to an LLM.
- nolist_policy 3y agoAt least for the linux kernel, qemu and other large c projects, this is a solved problem with coccinelle[1]. Compared to AI, it has the added benefit of not doing incorrect changes and/or hallucinating stuff or promt injections or ... I guess you could use AI to help create a coccinelle semantic patch. [1] https://en.wikipedia.org/wiki/Coccinelle_(software) https://en.wikipedia.org/wiki/Coccinelle_(software)
- dataangel 3y agoThe part coccinelle does is the part GPT is good at, the problem is neither of them actually reason about the code
- mrintegrity 3y agoParent post is getting down voted to oblivion but it seems a reasonable belief for someone who is not highly engaged with AI. I have only the vaguest understanding of how it works (and it's probably wrong) and to my layman mind it also seems like a totally fair assumption, based on experience as a user and the constant flood of news. Please explain why the suggestion that a future AI / sufficiently advanced LLM could refactor a complex codebase is so preposterous.
- not_the_fda 3y agoBecause refactoring requires understanding, which LLMs completely lack.
- ctoth 3y ago> Because refactoring requires understanding, which LLMs completely lack. Interesting. Here's a sample question I just asked the AI: We are working on a React mapping application. I am going to give you two components from this application. Our goal is to extract functionality to deduplicate code We can make a single base component, extract code to hooks, or any other strategy which you think is appropriate Here is travel.tsx: <snip> Here is heatmap.tsx: <snip> And here is the response: To deduplicate code and improve maintainability between the `TravelMap` and `Heatmap` components in your React mapping application, we can identify common functionalities and extract them into reusable hooks or components. Here's a strategy considering your code: 1. *Extract Common Hooks*: Both components use similar hooks for managing markers, popups, and map references. We can create custom hooks that encapsulate these functionalities. 2. *Create Shared Utility Functions*: Functions like `initialViewState`, which are identical in both components, can be moved to a separate utility file and imported where needed. 3. *Refactor Map Layers and Sources*: The way map layers and sources are handled is quite similar in both components. A custom hook or component to manage map layers and sources would be beneficial. 4. *Generalize Popup Handling*: The popup handling in both components can be abstracted into a custom hook or a higher-order component. 5. *Combine Common Styling*: Shared styles, such as `MAP_CONTAINER_STYLE`, can be moved to a common CSS module or a styled-component that can be reused. 6. *Refactor Event Handlers*: Functions like `onMapClick`, `onMouseMove`, and `onMouseLeave` have similar structures. These can be part of a custom hook or a utility function with configurable parameters. 7. *Legend and Live Region Components*: These components are used in a similar way in both `TravelMap` and `Heatmap`. Ensure they are designed to be reusable with different props. 8. *Refactor Map Initialization*: The way the map is initialized and configured is quite similar. This can be abstracted into a custom component or hook. ### Example Implementations: #### 1. Custom Hook for Map Markers: ```typescript // useMapMarkers.ts import { useState, useCallback } from 'react'; export const useMapMarkers = (initialLat, initialLng, world) => { // Similar marker logic from your components // ... return { marker, onMarkerDrag, onMarkerDragEnd, updateMarkerPosition, // other marker related states and functions }; }; ``` #### 2. Shared Utility Function: ```typescript // utils/mapUtils.ts export const initialViewState = (map, mapType) => { // Implementation }; ``` #### 3. Custom Hook for Map Layers and Sources: ```typescript // useMapLayers.ts import { useState, useEffect } from 'react'; export const useMapLayers = (props) => { // Logic to handle layers and sources // ... return { dataLayer, patternLayer, // other layer related states and functions }; }; ``` #### 4. Generalized Popup Handling: ```typescript // useMapPopup.ts import { useState } from 'react'; export const useMapPopup = () => { // Popup logic from your components // ... return { popupState, openPopup, closePopup, setPopupData, // other popup related states and functions }; }; ``` ### Integration: Once these hooks and utilities are created, integrate them into your `TravelMap` and `Heatmap` components. Replace the duplicated logic with calls to these shared resources. This will significantly reduce code duplication and make future maintenance easier. ### Testing and Refinement: After refactoring, thoroughly test both components to ensure that all functionalities work as expected. Watch out for any edge cases or specific behaviors that might be affected by the refactoring. For those suggestions, I might use five out of eight of them, and probably do one or two things differently. But you cannot, with a straight face, say the model did not understand. It clearly did. It suggested reasonable refactors. If being able to refactor means understanding, I guess we have understanding! I could continue with this conversation, ask it to produce the full code for the hooks (I have in my custom prompt to provide outlines) and once the hooks are complete, ask it to rewrite the components using the shared code. Have you ever used one of these models?
- ctoth 3y agoI see that you got some responses from people who may have not even used gpt-4 as a coding assistant, but I absolutely agree with you. A larger context window, a framework like Aider, and slightly-better tooling so the AI can do renames and other high-level actions without having to provide the entire changeset as patches, and tests. Lots of tests. Then you can just run the migration 15 times, pick from the one which passes all the tests... run another integration pass to merge ideas from the other runs, rinse and repeat. Of course the outer loops will themselves be automated. The trick to this is continuous iteration and feedback. It's remarkable how far I've gotten with GPT using these simple primitives and I know I'm not the only one.
- beeboobaa 3y agoIf you think a large refactor is just renaming some stuff then it makes sense you think this.
- dataangel 3y agoIf you ask GPT to refactor a single threaded program much smaller than the context window that is truly out of sample into a multithreaded program, its often going to fail. GPT has trouble understanding bit masks in single threaded code, let alone multiple threads.
- bigbillheck 3y ago> so the AI can do renames I do renames in a big window with my IDE.
- beeboobaa 3y agoI have some snake oil to sell you
- PaywallBuster 3y agothe pdf with a presentation requires a password
- bsdpufferfish 3y agoIf I'm operating a cloud service like Netflix, then I'm already running thousands of ffmpeg processes on each machine. In other words, it's already a multi-core job.
- platzhirsch 3y agoAs multi-core as Python and Ruby then.
- bsdpufferfish 3y agoYes. The kernel multiplies your efforts for you. It works great for web services.
- sanitycheck 3y agoCurious, what would that many ffmpeg processes be doing at Netflix? I assume new VOD content gets encoded once per format, and the amount of new content added per day is not gigantic. Agree with the general premise, of course, if I've got 10 different videos encoding at once then I don't need additional efficiency because the CPU's already maxed out.
- asveikau 3y agoProbably a lot more than once when you consider that different devices have different capabilities, and that they might stream you different bitrates depending on conditions like your network capability, screen resolution, how much you've paid them.. You could also imagine they might apply some kind of heuristic to decide to re-encode something based on some condition... Like fine tune encoder settings when a title becomes popular. No idea if they do that, just using some imagination.
- The_Colonel 3y agoI assume they re-compress for each resolution / format, quite possibly they also have different bitrate levels per resolution. Potentially even variants tweaked for certain classes of device (in cases this is not already covered by combination of format/resolution/bitrate). I would also assume they re-compress with new advances in video processing (things like HDR, improved compression). Also, their devs likely want fast feedback on changes - I imagine they might have CI running changes on some standard movies, checking various stats (like SNR) for regressions. Everybody loves if their CI finishes fast, so you might want to compress even a single movie in multiple threads.
- brucethemoose2 3y agoMeanwhile, I've been enjoying threaded filter processing in VapourSynth for nearly a decade. Not that this isn't great. Its fantastic. But TBH its not really going to change my workflow of VapourSynth preprocessing + av1an encoding for "quality" video encodes.
- tetris11 3y agoI believe I have too with gstreamer's pipe framework for threading, but ffmpeg's syntax has stuck in my mind far longer than any of the elaborate setups I built with gstreamer. I'm excited for this development
- dylan604 3y agoFFMPEG does so much more than just video encoding. I use ffmpeg all day every day, and only a fraction of the time do I actually make a video.
- m3kw9 3y agoLike what do you do?
- andoma 3y agoOne can use it instead of cat to display text files. Easy syntax to remember. ffmpeg -v quiet -f data -i file.txt -map 0:0 -c text -f data -
- DonHopkins 3y agoGood thing it's now multi-threaded so it can process all those command line arguments in parallel!
- whalesalad 3y agoI'm dying.
- 3y ago
- _8j50 3y agoWhen I stream 4k from my laptop ffmpeg gets very intense about cpu usage to the point fans are constantly at high speed and it's distracting. I hope this helps in some way. I have a fairly decent specces mid-tier laptop.
- deleted 3y ago[deleted]
- hereme888 3y agoI believe ffmpeg can be compiled to support the GPU, if your laptop has one. It works at least for CUDA-enabled GPUs (https://docs.nvidia.com/video-technologies/video-codec-sdk/12.0/ffmpeg-with-nvidia-gpu/index.html https://docs.nvidia.com/video-technologies/video-codec-sdk/1...) Talk with ChatGPT about it and see if you can do it.
- kevincox 3y agoI've always wondered if better multi-core performance can come from processing different keyframe segments separately. IIUC all current encoders that support parallelism work by multiple threads working on the same frame at the same time. Often times the frame is split into regions and each thread focuses on a specific region of the frame. This approach can have a (usually small) quality/efficiency cost and requires per-encoder logic to assemble those regions into a single frame. What if instead/additionally different keyframe segments are processed independently? So if keyframes are every 60 frames ffmpeg will read 60 frames pass that to the first thread, the next 60 to the next thread, ... then assemble the results basically by concatenating them. It seems like this could be used to parallelize any codec in a fairly generic way and it should be more efficient as there is no thread-communication overhead or splitting of the frame into regions which harms cross-region compression. Off the top of my head I can only think of two issues: 1. Requires loading N*keyframe period frames into memory as well as the overhead memory for encoding N frames. 2. Variable keyframe support would require special support as the keyframe splits will need to be identified before passing the video to the encoding threads. This may require extra work to be performed upfront. But both of these seem like they won't be an issue in many cases. Lots of the time I'd be happy to use tons of RAM and output with a fixed keyframe interval. Probably I would combine this with intra-frame parallelization such as process every frame with 4 threads and then run 8 keyframe segments in parallel. This way I can get really good parallelism but only minor quality loss from 4 regions rather than splitting the video into 32 regions which would harm quality more.
- seeknotfind 3y agoVideo codecs often encode the delta from the previous frame, and because this delta is often small, it's efficient to do it this way. If each thread needed to process the frame separately, you would need to make significant changes to the codec, and I hypothesize it would cause the video stream to be bigger in size.
- keehun 3y agoThe parent comment referred to "keyframes" instead of just "frames". Keyframes—unlike normal frames—encode the full image. That is done in case the "delta" you mentioned could be dropped in a stream ending up with strange artifacts in the resulting video output. Keyframes are where the codec gets to press "reset".
- pier25 3y agoSo does this mean that FFMPEG will be able to use multiple cores with all the included codecs? I'm using FFMPEG to encode MP3 with LAME for an audio hosting service and it would be great to improve encode times for long files.
- pseudosavant 3y agoDoubtful. Many codecs like MP3 aren't well suited to efficient multi-threaded encoding.
- mrbuttons454 3y agoWill this allow multiple machines to process the same file? If not is there anything out there that can?
- muragekibicho 3y agoShameless plug. I teach the FFmpeg C api here : https://www.udemy.com/course/part-1-video-coding-with-ffmpeg-and-c-in-replit-online-ide/ https://www.udemy.com/course/part-1-video-coding-with-ffmpeg...
- 3abiton 3y agoThat's such a very niche topic to teach. What usecases?
- muragekibicho 3y agoIt's for engineers tired of memorizing long weird CLI commands. I teach you the underlying C data structures so you can get out of command line hell and make the most out of your time!
- shp0ngle 3y agoI don't know anything about ffmpeg codebase, but I just wonder... how would I go about doing this _slowly_ without completely doing a giant commit that changes everything? The presentation says it's 700 commits. Was that a separate branch? Or was it slowly merged back to the project? Well I can look at github I guess
- shp0ngle 3y agoIt seems ffmpeg uses the mailing list patch way of doing "PRs", which is... well it is what it is. It doesn't help me understand the process unless I just go through all the mailing list archives, I guess.
- asylteltine 3y agoUgh why? That is so old school
- _joel 3y agoLike the linux kernel?
- asylteltine 3y agoWhich is also archaic and old school
- shp0ngle 3y agoI mean they might be used to doing that as ffmpeg is predating github. (and git.) as long as it works for them...
- ajhai 3y agoThis will hopefully improve the startup times for FFmpeg when streaming from virtual display buffers. We use FFmpeg in LLMStack (low-code framework to build and run LLM agents) to stream browser video. We use playwright to automate browser interactions and provide that as tool to the LLM. When this tool is invoked, we stream the video of these browser interactions with FFmpeg by streaming the virtual display buffer the browser is using. There is a noticeable delay booting up this pipeline for each tool invoke right now. We are working on putting in some optimizations but improvements in FFmpeg will definitely help. https://github.com/trypromptly/LLMStack https://github.com/trypromptly/LLMStack is the project repo for the curious.
- bane 3y agoA theory about this that may also affect other older solid software: the assumptions made on where to optimally "split" a problem for multi-threading/processing has likely changed over time. It wasn't that long ago that reading, processing, and rendering the contents of a single image took a noticeable amount of time. But both hardware and software techniques have gotten significantly faster. What may have made sense many years ago (lots of workers on a frame) may not matter today when a single worker can process a frame or a group of frames more efficiently than the overhead of spinning up a bunch of workers to do the same task. But where to move that split now? Ultra-low-end CPUs now ship with multiple cores and you can get over 100 easily on high-end systems, system RAM is faster than ever, interconnect moves almost a TB/sec on consumer hardware, GPUs are in everything, and SSDs are now faster than the RAM I grew up with (at least on continuous transfer). Basically the systems of today are entirely different beasts to the ones commonly on the market when FFmpeg was created. This is tremendous work that requires lots of rethinking about how the workload needs to be defined, scheduled, distributed, tracked, and merged back into a final output. Kudos to the team for being willing to take it on. FFmpeg is one of those "pinnacle of open source" infrastructure components that civilizations are built from.
- MightyBuzzard 3y agoIt's not the codecs that were multithreaded in this release. Pretty much all modern codecs are already multithreaded. What they decided to parallelize is ffmpeg itself. You know, the filter graphs and such. They didn't do anything to the codecs themselves.
- chrismsimpson 3y agoI assume 1 process with 2 threads takes up less space on the die than 2 processes, both single threaded. If this is true, a threaded solution will always have the edge on performance, even as everything scales ad infinitum.
- LtdJorge 3y agoHardware threads are not the same as software threads.
- vfclists 3y agoAll without Rust?
- 71a54xd 3y agoRandom reach here but has anyone here managed to get FFMPeg to render JS text over a video? I've been thinking about this workflow and just haven't quite figured it out yet, only a prototype in MoviePy but I'd like to move away from that.
- sylware 3y agoI think this was not "basic" multi-threading: they were careful about keeping latency as low as possible and some internal modifications of ffmpeg libs had to be done. That said, I don't think we still get input buffering (for HLS).
- Const-me 3y agoIntel Core Duo CPU was released in 2006. By then it was obvious computationally intensive programs need multithreading, these Unix-style processes are no longer adequate. I wonder why did it took so long for FFmpeg? BTW, MS Media foundation is a functional equivalent of FFmpeg. It was released as a part of Windows Vista in 2006, and is heavily multithreaded by design.
- atif089 3y agoDoes it mean that my FFMPEG H264 encoding until now was single threaded?
- urbandw311er 3y agoApparently not — many of the encoders were already multithreaded this was about making the ffmpeg pipeline itself parallelised.
- deleted 3y ago[deleted]
- kuba-orlik 3y agoI routinely use ffmpeg CLI and I see in top that multiple cores are engaged in the processing. Hadn't it been multi-threaded for years? What exactly is changing?
- Shish2k 3y agoMy interested-lay-person understanding: Previously it would read a chunk of file, then decode it, then apply filters, then run multi-threaded encoding, then write a chunk of file. Now it can read the next chunk of file while decoding a chunk while applying filters while running multi-threaded encoding while writing the previous chunk. I expect this will be a big help for complicated setups (applying multiple filters, rendering to multiple outputs with different codecs), but probably not much change for a simple 1-input 1-output transcode.