13 ms·
A title that actually describes the post, mostly paraphrasing the first paragraph: Reasons why this buffer overflow wasn't caught earlier despite doing all the
by lucb1e 5y ago
A title that actually describes the post, mostly paraphrasing the first paragraph:
Reasons why this buffer overflow wasn't caught earlier despite doing all the right things
And then to give those reasons:
- "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could never be discovered."
- "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes"
- "combined [fuzzer] coverage metrics [...]. This data proved misleading, as the vulnerable code is fuzzed extensively but by fuzzers that could not possibly generate a relevant input."
The conclusion is, of course, to fix those problems if your code base also has them, but also "even extremely well-maintained C/C++ can have fatal, trivial mistakes".
- deleted 5y ago[deleted]
- jandrese 5y ago> - "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes" This is the one that seemed short sighted to me. It's a completely arbitrary (and small!) limit that blinded the fuzzer to this very modest sized buffer overflow.
- js2 5y agoThe buffer holds 2K, so this limit alone which exceeds the buffer by 8K'ish didn't blind the fuzzer. It's not clear a larger input would've caught anything due to other "what went wrong" items, specifically "each component is fuzzed independently."
- a-priori 5y agoThe problem is that the search space grows (exponentially?) as you increase the fuzzer’s limit. So there’s a cost, and likely diminishing returns, to raising that limit.
- jandrese 5y agoAre they checking every possible overflow up to the max? Like no overflow at 7377 bytes, lets try 7378... While I can see targeting near natural boundaries (1025 bytes for example), you should be able to skip over most of the search space and verify that it doesn't blow up on enormous values like 16777216 bytes.
- lucb1e 5y agoIt's not that simple though. Single-variable integer overflows can be checked like that, but when the critical byte in a buffer might be at positions 1 through {bufferlength}, you have to do a shotgun approach and see if anything sticks, and at some point the number of possible combinations grows too big even for that. I'm not an expert on fuzzing myself, but generally I do see the point of having a limit here. Why, then, that limit was not chosen to be the max size for each of the length-checked inputs, I don't know. That does seem a bit more obvious, but also I just read this article so I can't prove that I wouldn't have made the same mistake.
- UncleMeat 5y agoCoverage-guided fuzzing dramatically mitigates the exponential nature of the search space. It used to be that searching for magic bits was impossible with fuzzing but now it is nearly trivial.
- moring 5y agoIs it possible to feed a fuzzer with information from static analysis to limit the search space? Such as, if you have a check like "someParameter > 0" in the code, have the fuzzer generate a positive, negative, and zero value for someParameter, but not thousands of them -- at least not based on this check alone -- because they will all behave the same.
- baby 5y agoThere's whitebox fuzzing that's starting to become a thing, when your fuzzer gets stuck you give the non-giving corpus to an SMT solver and it'll try to see if there exists an input that could uncover a new path. I'm really excited about these but haven't really followed the advances.
- jjoonathan 5y agoOh, the fools! If only they'd built it with 6001 hulls! When will they learn?
- 6chars 5y agoThank you! So many hindsight fortune tellers here.
- Terry_Roll 5y agoPredictable Human Behaviour, you can literally count on it! LOL Anyway the post mortem analysis is interesting because it gives away peoples knowledge or lack of. What is it they say? The Devil is in the detail!
- l33t2328 5y agoNot using small bounds on your fuzzing size isn’t exactly something you can only know via hindsight.
- baby 5y agoIf you increase the limit you lose in coverage as you can only do so many exec/s
- rfoo 5y agoWhat's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs. Otherwise, the same story could be told as a generic software testing joke: "unit-tests are short-sighted and coverage lies", i.e. an "extremely well-maintained codebase, with extensive unittest, >98% test coverage and constantly scanned by all-static-analyzers-you-may-come-up" can have fatal, trivial bugs.
- lucb1e 5y ago> What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs. Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buffer overflows are security issues. (Presuming no mitigations, of course, since those are separate from the bug itself.)
- rfoo 5y agoSorry, I mean the special part is "the bug itself is a memory corruption". The second sentence is a quick explanation for those not in our bubble.
- PeterisP 5y agoAs a crude example, there sometimes are off-by-one bugs which allow the buffer to overflow by one byte, and that single overflowing byte is always 0 (the last bye in a zero-terminated string), and it overwrites data in a variable that doesn't affect anything meaningful, giving you a buffer overflow with no security impact.
- trissylegs 5y agoAlthough single zero byte overflows are sometimes exploitable. https://googleprojectzero.blogspot.com/2014/08/the-poisoned-nul-byte-2014-edition.html https://googleprojectzero.blogspot.com/2014/08/the-poisoned-...
- Veserv 5y agoHow do you know that it was actually "extremely well-maintained"? Everybody thought OpenSSL was well-maintained since it was used as critical infrastructure by multi-billion dollar megacorporations, but it was actually maintained by two full-time employees and a few part-time volunteers with maybe a quick once-over before a commit if they were lucky. How about sudo, a blindly trusted extremely sensitive program [1], which is maintained by basically one person who has nearly 3,000,000(!) changes[2] over 30 or so years? Assuming that something is well-maintained because it is important is pure wishful thinking. Absent a specific detailed high quality process, or an audit that they conform to a well-established process that has demonstrably produced objectively high-quality output in a large percentage of audited implementations of that process (thus establishing nearly every instance of the audited process -> high quality output) all evidence indicates that you should assume that these code bases are poorly maintained until proven otherwise[3]. And, even the ones that are demonstrably maintained usually use very low quality processes as demonstrated by the fact that almost nobody working on those projects would be comfortable using their processes on safety-critical systems [4] which is the minimum bar for a high quality process (note the bar is "believe it is okay for safety-critical"). In fact, most would be terrified of the thought and comfort themselves knowing that their systems are not being used in safety-critical systems because they are absolutely not taking adequate precautions, which is a completely reasonable and moral choice of action as they are not designing for those requirements, so it is totally reasonably to use different standards on less important things. [1] https://news.ycombinator.com/item?id=25919235 https://news.ycombinator.com/item?id=25919235 [2] https://github.com/sudo-project/sudo/graphs/contributors https://github.com/sudo-project/sudo/graphs/contributors [3] https://xkcd.com/2347/ https://xkcd.com/2347/ [4] https://xkcd.com/2030/ https://xkcd.com/2030/
- nullityrofl 5y agoTavis explains clearly why he thinks it's well maintained in the post, complete with linkages to source code. To paraphrase: [...] NSS was one of the very first projects included with oss-fuzz [...] [...] Mozilla has a mature, world-class security team. They pioneered bug bounties, invest in memory safety, fuzzing and test coverage. [... all links to evidence ...] Did Mozilla have good test coverage for the vulnerable areas? YES. Did Mozilla/chrome/oss-fuzz have relevant inputs in their fuzz corpus? YES. Is there a mutator capable of extending ASN1_ITEMs? YES. I don't think at any point anyone assumed anything.
- titzer 5y agoThe whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language." No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.
- speed_spread 5y agoAt least Asbestos is inert once in place.
- masklinn 5y agoIsn’t asbestos always inert unless you matchmake it to a ridiculously strong oxidiser? In my recollection it dealt purely physical damage, and its inability to react with much of anything is what leads to its accumulation in dwellings.
- afiori 5y agoI believe they mean inert as it relate to health, not chemically.
- adrianN 5y agoUnfortunately it's too much work to throw away all code written in unsafe languages. So it's valuable to try and improve tools and techniques that make these languages less unsafe.