7 ms·
We found a division by zero bug in FFmpeg with a vibecoded fuzzer
- ozereray1 3mo ago[flagged]
- robertlagrant 3mo agoWhat we need is a numeric type that cannot be zero.
- drdaeman 3mo agoWhat we need are refinement types, where there’s a base type and a predicate. F* has this: val (/) : int -> (divisor:int { divisor <> 0 }) -> int
- rhdunn 3mo agoIt would be more flexible for a compiler to reuse the range analysis logic used in optimizations for statically verifiable divide by zeros. That way you could extend it to other things like statically verifiable overflows.
- yeputons 3mo agoAnd also cannot be INT_MIN, otherwise -1 / INT_MIN is undefined behaviour(!) in C and C++.
- winwang 3mo agoEvery day, we stray closer to Haskell. Dare I say it: good!
- duped 3mo agoFor stuff like niche value optimization sure. For practical arithmetic code, nah. Like with this bug, all that changed is that garbage data in gives the user an error that they tried to process garbage data. Adding a new type doesn't make the code better, it just moves the error around. And you really don't want an infix division operator to fail to type check if the right hand side isn't a nonzero type, do you?
- robertlagrant 2mo agoI think moving the error around is a good idea. E.g. integer division would end up with five cases: Int / 0 -> DivisionByZeroError PositiveInt / PositiveInt -> PositiveFloat PositiveInt / NegativeInt -> NegativeFloat NegativeInt / PositiveInt -> NegativeFloat NegativeInt / NegativeInt -> PositiveFloat And the type signatures of those possible return values can drive validation checks upstream of the calculation, so you're not actually ever going to return DivisionByZeroError. You're making sure through validation checks or case logic that that can never be returned.
- roadbuster 3mo agoThe only way to achieve this is to either put a runtime software check on a variable whenever it's assigned/used, or to literally add hardware support in processors themselves which literally throws an interrupt when a "neverShallBeZero" variable is assigned to zero. There's no viable way to statically prove at compile-time that these variables will never become zero at runtime, ultimately forcing a system of endless runtime checks (be it software or hardware)... which is why processors already throw exception interrupts when division by zero is attempted.
- colechristensen 3mo agoYou're kind of saying the only way to do it is in software or hardware :) An alternative https://en.wikipedia.org/wiki/Projectively_extended_real_line https://en.wikipedia.org/wiki/Projectively_extended_real_lin... The projectively extended real line defines division by zero, no reason you couldn't have a floating point type that implemented it. >There's no viable way to statically prove at compile-time that these variables will never become zero at runtime strongly typed programming languages like Ada allow for types which have ranges such as disallowing zero -- but also any arbitrary thing like you can create a floating point "degrees" type which is [0.0, 360.0] or any other ranged type
- inigyou 3mo agoIt's possible, just extremely difficult.
- dabinat 3mo agoIt’s interesting how AI may both raise and lower the quality of software. It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal. Time is much more important for a human developer with a salary.
- Supermancho 3mo agoI don't care if you call it an over-engineered looping machine or what, there are concrete benefits to using LLMs for this. They work faster than developing your own looping algorithm and more often produce useful results than not.
- saghm 3mo agoIt's not even like fuzzers are valuable because of the process they use specifically either; the value is that they produce a concrete input that you can use as a reproducible test case at that point. The value could be produced by gazing into a crystal ball for all I care, as long as I can use what it gives me to reproduce a bug.
- dmix 3mo agoFinding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part.
- deleted 3mo ago[deleted]
- bewareofscams 3mo ago[flagged]
- shevy-java 3mo ago> LLMs do find bugs, do save time They find bugs but whether they save time is nowhere near as clear as you try to insinuate here.
- VCFundedGenYer 3mo agoThe fruits of using LLMs to code. You'll waste far more time finding what it quietly and subtly wrecked than you would have if you just coded it yourself.
- vegnus 3mo agoYou're not reading it right. The bug was found using a vibecoded fuzzer.
- 12j3afAv 3mo agoI wonder from where Claude stole this fuzzer.
- pjankiewicz 3mo agoOr it used something called an "analogy" which is a valid way to solve new problems.
- criddell 3mo agoThe very first line of the bug report: > This is a bug found with our fuzzer: https://github.com/daedalus/fuzzer/ https://github.com/daedalus/fuzzer/
- LoganDark 3mo agoI think they're talking about the misconception that LLMs can only ever regurgitate their training data verbatim enough to constitute mass copyright violation. And that that's therefore "stealing"
- jaggederest 3mo agoThose sneaky LLMs going 7 years into the past and committing as a human: https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/8eda3c7f91e1a5b3917ef87ea3249968d5a9f263 https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/8eda3c7f91e1a5b...
- Surac 3mo agosend patches
- rs_rs_rs_rs_rs 3mo ago...they did.
- ligarota 3mo agoWhere? They only suggested a basic guard, chich can be useless if this case never happens
- 12j3afAv 3mo agoGenerating an incorrect input file seems to be the easiest task of all for any fuzzer. Generating correct input to get deep into the call stack and then finding something is the hard part.
- deleted 3mo ago[deleted]
- akshay_akula 3mo ago[flagged]
- wy35 3mo agoUnrelated to the submitted link -- just checked your comment history and all of your comments are AI-generated like this one. What's the motivation for this?
- f311a 3mo agoHe won’t reply, he’s busy promoting himself and his peojects with AI.
- akshay_akula 3mo agoAlso haven't done any self promo, but yeah i'll stop running my posts thru chat
- bigfishrunning 3mo agoprobably karma farming
- akshay_akula 3mo agoLoL I just sound botted
- ks2048 3mo agoNo doubt fuzzers (vibecoded or otherwise) can be powerful, but can't you just mark all "/" as potential divide by zero errors? I guess sometimes developers think they "know" some variable won't be zero, but unless it checked explicitly or by the compiler, that shouldn't be trusted.
- wvbdmp 3mo agoI mean there could be a guard clause? But yeah, seems like this could be statically evaluated like how some IDEs see a null check and don’t complain about nullability within the same scope.
- dooglius 3mo agoWhat are you suggesting and how would it be different than how SIGFPE already works?
- Someone 3mo ago> but can't you just mark all "/" as potential divide by zero errors? If you’re accepting large false positives rates: yes. If you want users to take your warnings serious: no. (Nitpick: you certainly don’t want to flag _all_ of them. Divisions by non-zero constants definitely should be excluded, for example (integer division by -1 can lead to overflow, but that would be a different warning))
- saghm 3mo agoFuzzers find inputs, not just "potential" errors that aren't triggerable.
- MaxBarraclough 3mo agoIf it's possible for program execution with some particular input to lead to a divide-by-zero, that's a bug, especially if the program is expected to be able to handle malformed inputs, or perhaps even deliberately malicious ones. It's not trivial to determine whether a program does this correctly. If it was, program analysis would be easy. Division can 'go wrong' for certain inputs, but it's not just division. In C, signed integer addition, subtraction, and multiplication, all give undefined behaviour on overflow. As 'Someone' already pointed out, it's not helpful to just flag all uses of the division operator, or of other potentially dangerous operators. Minimising false positives is one of the core challenges of program analysis.
- souvlakee 3mo agoIt is interesting that FFmpeg has its own Git server. Maybe we should move there too?
- TacticalCoder 3mo ago> It is interesting that FFmpeg has its own Git server. Maybe we should move there too? Git is a DVCS. I know many people only ever used Git through Github and forgot what the 'D' in DVCS means but whether or not they remember what the 'D' stands for, running your own Git server is trivial. Especially in this day and age of LLMs were you can just ask: "Clone this repo and convert it to base Git repo and serve it on the LAN PLZ KTHX". The result is going to be more stable than Github and, arguably, more secure too.
- inigyou 3mo agoIf you have SSH access to a server and Git is installed on that server, you can use it as a Git server. No additional setup is required. The Git client knows how to log in and invoke the Git server over SSH.
- snailmailman 3mo agoLots of projects run their own git or forgejo or similar. I run my own private forge, and it has a higher uptime than GitHub. (A shockingly low bar, tbh) It’s surprisingly simple to setup, and the hardware requirements are pretty small for a private or small forge, as it’s usually a relatively small number of users/repos/etc.
- sva_ 3mo agoYou can add several remotes to your git, and I'd recommend you do so.
- cpriest 3mo agoNice find. The interesting part isn't "AI wrote the fuzzer." It's that a cheap random harness still hits classical bugs in ancient parsers. Keep the corpus; throw away the hype.
- whatsThisBtn4 3mo ago[flagged]
- jeffbee 3mo agoI imagine the discussion will center around this application of AI, but to me this is just the Nth proof of the proven fact that you must build ffmpeg, if you insist on using it, with only an allow-list of file formats that you expect to encounter, and not with the kitchen sink of stuff you are never going to need.
- tensegrist 3mo agonote that this seems to be a bug in what i expect (feel free to correct me) is a code path for a little-used codec maybe we'll just see them remove support for these long-tail formats the way linux has been removing drivers for similar reasons https://www.phoronix.com/news/Linux-Retiring-Moxa-Driver https://www.phoronix.com/news/Linux-Retiring-Moxa-Driver
- parl_match 3mo agoit's widely used but in "industry" applications. so ffmpeg is probably being used in a lot of offices (studios) and maybe even being included in end user software.
- inigyou 3mo agoUnderstatement of the year. Almost everything that processes video uses ffmpeg.
- parl_match 2mo ago> Understatement of the year. Normally yes. For VPK, actually, no. The file format that we're describing was part of a proprietary embedded SDK for the PS2. The majority of professional integrations would be using the Sony SDK. Today, a lot of tooling uses FFMPEG for accessibility of those formats. But I wouldn't say it's the majority. Hence why I said "(studios)" after offices, and "probably" and not "majorly": > probably being used in a lot of offices (studios)
- cptroot 3mo agoThis is not a real bug in FFmpeg. This is a demonstration that if you control a custom AVIO module it is possible to crash FFmpeg by giving it bad data.
- inigyou 3mo agoNot custom. It's an existing module for a format called VPK. It's a quite trivial bug though, not exploitable apart from DOS and won't ever happen in a real file.
- VladVladikoff 3mo agoI even question if it is a DOS vector. So the thread crashes and then the system that controls the threads cleans it up and opens a new thread. Seems to be a trivial impact, unless it locks up the thread somehow.
- inigyou 3mo agoThreads don't work that way. A fatal exception on any thread kills the process.
- VladVladikoff 3mo agoAnd the parent will spawn a new process. Unless the server is terribly poorly misconfigured. Edit; for what it’s worth I’ve run a server processing video with FFMPEG for 10 years now, and there’s just so many things that can make FFMPEG crash. All sorts of corrupted videos people upload. If your server doesn’t recovery gracefully from a crashed FFMPEG thread, that’s on you, not FFMPEG.
- LoganDark 3mo agoI thought you meant Disk Operating System until I realized you probably meant DoS
- avadodin 3mo ago
- justonenote 3mo agoWhatever about the specifics of this bug and whether its a useful vector, this is not surprising even in the slightest? My current opinion on LLMs is that they are superhuman in that they lack fatigue, they have close to full knowledge across all subjects which are known to humans at least publicly, and the fact that you can vibe code a harness to look for bugs in a famously complicated C codebase is intern level stuff and hardly news. Smart aspiring blackhats will be targeting tmux next, both with light llm jailbreaks, light supply chain attacks (web search results) and LPEs within certain environments which weren't particularly useful before but with agents running on auto mode for hours become a very valuable springboard. I'm not sure on the quality of tmux code but I know its written in C and is very complex and was not at all designed to defend against this type of threat.
- senordevnyc 3mo agothe fact that you can vibe code a harness to look for bugs in a famously complicated C codebase is intern level stuff and hardly news It seems like this would have been pure fantasy not that long ago though. So why isn’t it noteworthy again? I don’t really follow what you’re complaining about.
- deleted 3mo ago[deleted]
- hnlmorg 3mo agoI don’t think tmux is the most worthwhile target because you’d need the user to either execute code locally (thus negating any point in targeting tmux) or rely on the user curl or cat some compromised document (in which case you’re better off targeting curl or cat).
- justonenote 3mo agothe point is tmux is being used by many developers working in high value targets to automate long running unsupervised agent tasks. you don't need the user to execute code, you need _their agent_ to stumble on the wrong search result or github repo and it wont be noticed for hours that they loaded a persistent threat into your environment.
- skupig 3mo agoAm I missing something? Who cares? This isn't a security issue, it's just an unexploitable crash on bad data.
- inigyou 3mo agoNo, you're not. It's a minor bug, probably with an easy fix, that deserves to be fixed. It's not worthy of front page HN...
- Jaxan 3mo agoI guess it’s submitted for the method rather than the result.
- ramon156 3mo agoeveryone knows HN only accepts security write-ups /s
- mcdow 3mo agothat’s the issue with these newer models. they are able to string together a sequence of “not-serious” bugs in a system that ultimately results in some serious vulnerabilities. it may not be an issue for ffmpeg, but it might be for an application that bundles ffmpeg.
- aeyes 3mo agoA patch for this was submitted in April: https://lists.ffmpeg.org/archives/list/ffmpeg-devel@ffmpeg.org/message/F4FVBV4GOAWHEPVJ4JAJLHQ23UPKDJ6I/ https://lists.ffmpeg.org/archives/list/ffmpeg-devel@ffmpeg.o... Edit: And there was discussion about this back in 2024 as well
- semiquaver 3mo agoOddly enough I can’t access that site, it just heats up my phone solving hashes. Gave up after about a minute and anubis had only made it less than halfway through. I doubt the real bots have any trouble bypassing it.
- demibabs 3mo agoYeah, is it trying to mine bitcoin or something? Anubis usually takes a second but here I waited a minute and got 20% through on a modern phone.
- bulder 3mo agoPresumably they've configured it to use a higher difficulty challenge due to high rates of scraping on their bugtracker
- myng111 3mo agoDifficulty 6 which some parts of FFmpeg use, is about the highest difficulty you can assign with the default Anubis config. For me personally I only serve that difficulty if I'm near certain the user is a bot. Serving it to everyone sure is a choice.
- bartread 3mo ago[flagged]
- Georgelemental 3mo ago> And don't come crying to me about how the problem you're trying to solve is "hard". I don't care: you chose it, you chose to considerably worsen the web browsing experience of millions of people globally, nobody made you. Unfortunately, if you let all the bots in, they overwhelm your servers, and then nobody can access the website.
- BikiniPrince 3mo agoFunny thing, I know I'm brushing up against something in gStreamer developer, but Fable flips out. I have only a loose idea where the issue might be lurking. Next week, I'll apply for the cyber and I suspect I'll find something similar. Right now, it's just annoying and thanks the OpenAI cyber was much easier to get access to.
- 1saadcodes 3mo agoI find it pretty cool that a fuzzer thrown together this way actually found a bug in ffmpeg
- pfdietz 3mo agoThe thing about testing is that each time you produce a new kind of tester you have a chance to find bugs in the blind spots of the previous testing approaches. Diversity makes sense, more so than in software construction.
- dclavijo 3mo agoYeah that was my bet, escaping the local minima imposed by the current state of fuzzing. I am seeing this problem as statistical and information theory problem. Some day someone by chance will create another fuzzer that would find more bugs because of the blindspots in the previous generation including mine.
- driverdan 3mo agoThe README for the fuzzer is an AI slop mess. https://github.com/daedalus/fuzzer/ https://github.com/daedalus/fuzzer/
- aaron695 3mo ago[dead]
- Zebfross 3mo agoWhy submit an issue rather than just making the fix and adding the tests in PR? Seems like they're just making work for the maintainers.
- dclavijo 3mo agoOP here: A bug report just needs a proof of existence for the condition while a bug fix needs a proof of correctness. Sometimes is the best to let the developers who are day to day in the codebase to choose the best fix and if they what to fix it.
- peter_retief 3mo agoBugs days are numbered with AI!
- pfdietz 3mo agoTechnically correct, because there are infinitely many numbers.
- peter_retief 3mo agoTrue and funny.
- dorianmariewo 3mo agogiven enough ai, all bugs are shallow
- thephyber 3mo agoGiven enough AI, all bug fixes have extremely deep carbon footprint.
- throwa356262 3mo agoI am sure the fuzzer is interesting. But this bug feels like something an LLM would flag as a major finding but turns out to be completely benign. Update: I tried to look into the fuzzer but it is hard to get past the AI blabb. Can someone please explain to me what it does beside being structure aware?
- boomlinde 3mo agoIt crashes because of input that should have been rejected for being invalid. How could that be construed as being benign?
- j16sdiz 3mo agoafaict, decoder bugs like these are treated with lowest priority possible. It is not enabled by default. It is used only in video games, which input files are fixed set of asset that came with the game. It can be a crash, yes. but the typical user of this codec won't care.
- throwa356262 3mo agoBecause an attacker would not gain anything he not already has. This is basically local self-DOS.
- boomlinde 3mo agoThat's not a quality of ffmpeg or this bug, but of the application you use it for. If you only expose your ffmpeg-based application to your own input then yes, of course it's a self-DOS. But if you, say, expose it as a web service passing arbitrary user input to ffmpeg, that no longer holds.
- throwa356262 3mo agoEven then it will be a self-dos: the video you uploaded won't be processed.
- sylware 3mo agoThe real core of the issue is actually the complexity/size and core design of media container/codec file formats.
- written-beyond 3mo agoIDK seems like a bug that could've taken a human a few minutes at best to find. I found a bug in SystemD that would crash the daemon because a bad SystemD unit file configuration. That took me like 5 minutes to actually track down in the actual source code. I understand the utility of this though, I just don't see this particular bug and something that would be particularly difficult o find pre LLM era.
- thephyber 3mo agoThis is the wrong mentality. The fuzzer found the bug before any humans did, so there is a mismatch of developers who could find this bug and those who did (without an LLM-coded fuzzer). The value of the fuzzer continues long after it found this one bug. It's worth nothing that in the bug discussion thread, the bug fix author pointed out that it's not easy to set up the config then call the functions in the right order. Your comment assumes that the reader has enough context to read the code and build the finite state automata in their mind. The bug fix reporter's comments suggest that you are assuming things which you shouldn't assume.
- written-beyond 3mo agoBut I am being very specific to this use case, where a division by zero bug was found. Why couldn't you have just grepped through the codebase, found all possible divisions and ensured that they had a check on it to never be less than or equal to 0? The bug I located in SystemD was literally a null ptr exception. All they had to do was perform a null check on a cstring but they hadn't. I don't see the utility of reporting an LLM made fuzzer finding bugs that could be found by a lint rule or static analysis. I will appreciate a post about an LLM fuzzing software to find unique corner cases, which I predict will happen soon, in ACL controlled systems caused by policy shadowing.
- pfdietz 3mo agoBugs are always easier to find in retrospect.
- 3mo ago
- dclavijo 3mo agoOP here: for those interested is not that an LLM found the bugs the fuzzer found them. My take on this matter was to implement as much information theory algorithms as possible, and try to extract as much structure with statistical importance from the binary being fuzzed. Also port as much features from other fuzzers and whie papers on the mater (llms are good at connecting dots across vast codebases and papers). I honestly can not take full credit for this work since I made it with AI, but I has taken two months of my time and 1100+ commits. My developing process was to use several models from several vendors not just Claude that decouples it from a single vendor/model and throws to the flor that llms regurgitate verbatim code. Also the interesting part is the developing pipeline I have had setup my own cicd with my own tool impactguard whitch saved me a couple of times and hard rules on the agent.md(70% to 80% of those rules i wrote them by hand). The pytest testing battery is also interesting, I adopted TDD and to me since I adopted it seems that llms make less bugs. Yes I know the code and the readme might look like ai slop as pointed out earlier but is efective at finding bugs. At the end of the day is all economy: you spend a lot of tokens once and keep the fuzzer forever, not the same as paying every time for tokens to find bugs. As pointed out in the readme, this fuzzer trades speed for edge novelty, maybe there is it's niche. Also we found earlier another bug with this fuzzer https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/23945 https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/23945. For the concerned IMO: rather than the results the methodology is more important. I welcome constructive criticism and feedback. Any input is useful to me.
- nixpulvis 3mo agoI can't speak to the quality of the fuzzer since I haven't used it or looked at it thoroughly, but it does seem to cover a lot of ground on features and interesting concepts. I'll definitely be reading more into what you have here.
- krpovmu 3mo agoDid we find it?, or Did AI do my job?
- thephyber 3mo agoDo you normally get paid to fix bugs in open source repos?
- soiax 3mo agoWhy are people upvoting a unexploitable bug? How is this interesting? There are thounds of these, no one even reports them unless they are exploitable, DoS only.
- 21wa 3mo agoBecause the fuzzer was vibe coded and stolen by Claude! They only need the headline for celebrating another "AI victory" on Twitter, even though the issue was found in 2024 by OSSFuzz: https://ffmpeg.org/pipermail/ffmpeg-devel/2024-November/335598.html https://ffmpeg.org/pipermail/ffmpeg-devel/2024-November/3355...
- ChannelFence 3mo agoTwo months and 1100+ commits to rediscover a bug that was already found in 2024 is probably the funniest possible ending to a "vibecoded fuzzer" story.
- dclavijo 3mo agoNo, the funny thing is that this comment is comming from an account with 1 karma, no submisions and only 1 comment, I wonder if this is your only account and you just came to spill hate or you are using multiple accounts to discredit other peoples work.
- ChannelFence 3mo ago4 now lol. why would my karma matter anyway? this isnt reddit
- dclavijo 3mo agoIt just states your intention, you could just have been constructive with your first comment on HN.
- ChannelFence 3mo agoWhy is it a bad thing I found this funny? Two months and 1.1k+ commits to rediscover an already known bug IS funny. That doesn't mean I hate the person or want to discredit their work. You're reading way too much into a comment :)
- dclavijo 3mo agoHumor is subjective, not objective; if we delve into the subjective realm, I can feel whatever I want, just like you. You haven't offered anything constructive yet. I'm more willing to listen to your ideas if you have any.
- 3mo ago