13 ms·
FFmpeg and a thousand fixes
- samworm 13y agoIt is interesting that YouTube isn't mentioned in this blogpost, despite there being good evidence that ffmpeg has been used there[1]. The fuzz testing they mention is based around constructing malformed (or at least "exotic") input files and then monitoring for failures... ie simulating exactly the kind of attack someone might use against YouTube's transcoding infrastructure. [1] http://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/ http://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/
- chadillac 13y agoI'm not proud of this legacy code, but... it exists because it was a real world issue that would cripple a conversion server in production envs when fed certain files with timing/syncing errors as part of an automated upload and conversion process. When it would crash it would consume 100% of the cores and eat up enough RAM to force swap. This cron has been keeping ffmpeg in check for over 4 years (not 6, whoopsie) in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system. 1 <?php 2 /* 3 ** Cron responsible for detecting failed/hung ffmpeg 4 ** instances and killing them. 5 */ 6 require_once '/lib/class/dbmysqli.php'; 7 8 $output = shell_exec('ps -aeo pid,etime,args | grep ffmpeg | grep -v grep'); 9 10 11 12 13 preg_match_all("/^[ ]{0,}([0-9]*)[ ]{0,}(.*?) .*?([0-9]*?)\-[0-9]*?\.flv /m", $output, $preg_out, PREG_SET_ORDER); 14 15 if (!empty($preg_out)) { 16 17 $db = new DBmysqli('dbmaster'); 18 19 foreach ($preg_out as $process) { 20 $pid = intval($process[1]); 21 $etime = $process[2]; 22 $queue_id = intval($process[3]); 23 24 $etime = intval(str_replace(':','',$etime)); 25 26 // elapsed time >= 60:00 (1 hour) 27 if ($etime >= 6000) { 28 $fail_sql = "DELETE FROM media_upload_queue WHERE queue_id = $queue_id"; 29 shell_exec("kill -9 ".$pid); // kill hung ffmpeg process 30 $db->query($fail_sql); // remove file from DB 31 // debate if life is worth living... 32 } 33 } 34 } 35 ?> Yes I know there are better ways to do this now at the OS level, but it was a quick hack over a half decade ago and continues to work... ain't broke, don't fix it kinda deal.
- rip747 13y ago"I'm not proud of this legacy code," "This cron has been keeping ffmpeg in check for over 6+ years in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system." why do we programmers always feel we need to apologize for something that we did quickly, but has been running without incident for a number of years. take a bow my friend. that was an awesome patch you came up with all those years ago.
- MBCook 13y ago> why do we programmers always feel we need to apologize for something that we did quickly, but has been running without incident for a number of years. The script is only treating the symptom, not the problem. It would be better to detect the files before they waste an hour of time, so you could tell the user instead of having them silently disappear. Maybe there is something you could do to fix the files. The programmer part of me says there is a real fix that needs writing. It's obviously a great script if it's worked that long. Who knows how long it would take to track down and fix the bug(s) causing the issue. If they haven't needed to fix the bug in all these years just writing that script was obviously a good decision.
- zaroth 13y ago> If they haven't needed to fix the bug in all these years just writing that script was obviously a good decision. Exactly this. There are an unbounded number of bugs which will cause this single symptom. I think detecting the symptom is exactly the right solution.
- kbenson 13y agoIn theory. In reality, after treating one or two causes, the occurrence rate may fall to once a month or year. I agree that mitigating the problem is a good first step though.
- burntsushi 13y ago
- tttt1111 13y agoGoogle has never officially acknowledged that YouTube uses ffmpeg. I seem to recall it was because of legal reasons they are not allowed to. So you will never hear a Google employee mention anything about YouTube and ffmpeg ;)
- dpe82 13y agoWe do at least know that they use x264: http://x264licensing.com/adopters http://x264licensing.com/adopters
- nathancahill 13y agoInteresting that they are pushing fixes for both FFmpeg and the libav fork. I guess they decided not to pick sides in that war.
- craigyk 13y agoThat is good. But this reminds me of the first time I installed 'ffmpeg' on Ubuntu only to have it not work with my well-tested parameters and also declare ffmpeg 'deprecated'. Only after some head scratching did I figure out it had installed libav and an ffmpeg wrapper that tried but failed to be compatible with the real ffmpeg interface. I wasn't even aware of the split at the time, but this shenanigan definitely gave me a strong negative initial impression of libav.
- astrange 13y agolibav tries to ship a compatible wrapper called ffmpeg and a redone one called avconv. But ffmpeg itself just ships ffmpeg and puts incompatible changes in there, so you'd actually be worse off. What issues were you having, and with what file types?
- nathancahill 13y agoIncompatible? Do you mean with libav?
- astrange 13y agoBackwards incompatible with previous versions of itself.
- ux 13y agoMost of the old options have been mapped to the new interface (see opt_old2new() for instance). So most of the old commands are still compatible. Libav also has dropped ffmpeg forked copy since a long while, so FFmpeg is actually more retro compatible.
- IceyEC 13y agoI get a kick out of the fact that they recommend not using FFMPEG but they say that Chrome uses it.
- deleted 13y ago[deleted]
- abraham 13y ago> I get a kick out of the fact that they recommend not using FFMPEG More specifically Google is recommending you don't use it to "process untrusted media files"
- ID_HOME 13y agoI don't trust any media files... Especially ones from Google who are doing most of the research... seriously though, media zero-days are unheard of, probably because they would be extremely valuable.
- fletchowns 13y agoThey aren't just flat out not recommending using FFmpeg, just that you need to take appropriate measures when dealing with untrusted media sources.
- jcampbell1 13y agoChrome is probably using a subset that represents 2% of FFMpeg. I'd guess they have thoroughly audited the code paths that are actually used to decode about 10 media different that chrome supports. For instance, one fix mentioned was a fix the demuxer for .4xm files. 4xm is a proprietary container format for a now defunct company. Chrome would never have to worry about this bug, because it doesn't need to support playback of .4xm files.
- chubot 13y agoThis is cool. I wonder how many of the bugs led to code execution? If you managed to execute code, what privileges do you have in Chrome? I hope that Chrome was using OS sandboxing for video playing. After all, if they found 1000 bugs, there are probably a few more zero day's available. Does playing Flash video in Chrome/Firefox end up using ffmpeg? I'm not that knowledgeable about how video works, but there are probably at least two execution contexts: playing .mp4 natively in Chrome, or playing it via a Flash container.
- nathancahill 13y agoPlaying .mp4's natively uses FFmpeg, but Flash video does not. Both plugins are sandboxed in a "privilege separation" environment that they mention in the post.
- jbk 13y ago> I wonder how many of the bugs led to code execution? Quite a few. We're often affected with VLC, and code execution is easy to get to. But with VLC, you're "only" in userland.
- tptacek 13y agoIn other words, you're "only" dealing with a bug that would allow anything you downloaded from BitTorrent to own up your personal account on your computer. :) (I know you know that, I'm just spelling it out for people).
- jbk 13y agoYep, and that's quite dangerous. Especially since you can see often fake videos circulate around, that tell you to download a special software or codec to play them (with malware, of course). WMP also had a scripting/runtime system that could be abused greatly. Be careful what you download :)
- fulafel 13y agoChrome is also in userland - and it has a sandboxing system. Assuming they're sandboxing ffmpeg, these bugs are more risky for VLC users than Chrome users. Plus, Chrome is more diligent with security updates and the auto-update mechanism is fully automatic. A sandboxing system provided by either ffmpeg or VLC would be a very good idea, though it would be some work... encoded data in, decoded frames out via shared memory. Negligible performance impact.
- lhgaghl 13y ago"Until we can declare both projects "fuzz clean" we recommend that people refrain from using either of the two projects to process untrusted media files." BAHAHA What a joke.
- claudius 13y agoOnce again I am happy that ‘Segmentation fault. Core dumped.’ is a perfectly reasonable reaction to malformed user input in my field of work, which only increases my respect for those who write real-world software. Thanks! :)
- tlrobinson 13y agoThis must be one of the reasons it seems like there's a new update to VLC every time I launch it...
- pjmlp 13y agoThe wonders of C: - NULL pointer dereferences, - Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access, - Out-of-bounds reads and writes to stack, heap and static-based arrays, - Invalid free() calls, - Double free() calls over the same pointer, - Division errors, - Assertion failures, - Use of uninitialized memory. But hey, any good programmer always writes perfect C code.
- taeric 13y agoYou forgot: - One hell of an awesome encoder/decoder library that can hit more formats than I knew existed. :)
- atmosx 13y agoThat's not a bug, that's a feature (people just don't realize the POTENTIAL LOL).
- vinkelhake 13y agoReally breaking new ground here. What should FFmpeg have been written in? I can think of a few candidates, but I think they share a number of those "wonders".
- lmm 13y agoThe only options I could think of that're mature enough are OCaml and Haskell. I'm aware of video transcoders in both, though nothing with anything like the format support of ffmpeg.
- colanderman 13y agoOCaml I know doesn't support SIMD acceleration. Not sure about Haskell. You need SIMD support to get anything decently fast in the field of video/audio manipulation.
- pjmlp 13y ago
- lmm 13y agoA thousand, and all the categories they list are things the computer should catch for you. How many more will it take before we switch to better languages?
- lucb1e 13y agoBe aware that these are open-source projects. I don't see you starting a fork in a better language.
- Goopplesoft 13y agoI find this, 'don't complain, fork/fix' mentality ridiculous. Why should he? If his goal is get people off C, communicating/persuading in a developer community comments is probably more effective than him creating a project in some other language that no one would likely ever use.
- zw 13y ago> communicating/persuading Snarky, whiny, bitching comments do not a discourse make.
- pkroll 13y agoIn what way is asking someone, somewhere, to move a project to another, unnamed language, going to be more effective than actually starting a project and putting code into it? Sure, "go do it yourself" is not a pleasant response, but "go take the tens of thousands of lines of code that mostly works, and spend person-years rewriting it in another language so that a certain set of bugs aren't an issue anymore, instead of fixing the mostly working code" is in no way a reasonable request. "What can be done to help reduce the bug count, by a casual user?" might be. Contacting Coverity or another static analysis company that occasionally runs their tools on open source programs to help the world (and get the free press out of it...), might result in a huge list of subtle (and hideously obvious) bugs getting squashed.
- atmosx 13y ago> At Google, security is a top priority - not only for our own products, but across the entire Internet. Hm, privacy was not on top, I guess - or we're toasted.
- justincormack 13y agoYears ago (I think it was ffmpeg) it would crash constantly on tv input, but it ran in a different process and restarted automatically.
- deleted 13y ago[deleted]
- scrabble 13y agoUntil we can declare both projects "fuzz clean" we recommend that people refrain from using either of the two projects to process untrusted media files. You can also use privilege separation on your PC or production environment when absolutely required. Well, it's not like I've got a whole lot of options for video processing. Note that they didn't recommend alternatives that they say are fuzz clean. I use FFmpeg quite extensively and will continue to do so.
- kokey 13y agoIt sounds like about the right level of effort required to maintain something developed by Fabrice Bellard.
- emmelaich 13y agoBit off topic but I may as well bring this up here. 1. flvrunner uses ffmpeg 2. biggest result for flvrunner searches is how to remove flvrunner (including browser toolbars) 3. doubleclick runs webads for flvrunner.com (google own doubleclick) 4. such ads even run on onlinebehavior.com, which is owned by Google's analytics guy. There's a lot of malware on the net that google could do more to reduce. Someone more conspiratorially minded than me might make other deductions.
- CraigJPerry 13y agoSo my paranoia has finally taken over. My first read through and I only take in these features: * I presume all this infra is used for more than just ffmpeg so why this software specifically? Im conscious its included in everthing from web browsers to games. If this was an attack vector, its a BIG vector. * 1121 (or whatever it was) is a curious number to make a post about. Why not at the 1k mark or 1250? * Explicitly calling out use of an unpriv user to a readership likely already well versed in basic security practices but notorious for casually ignoring on personal machines Google, blink twice if theyve got ya tied up in some legal requirement that you cant disclose.
- neves 13y agoThe software testing course in Udacity is an excellent introduction to the fuzzy testing techniques that the article talks about. Here is the link: https://www.udacity.com/course/cs258 https://www.udacity.com/course/cs258