7 ms·
How we applied fuzzing techniques to cURL
- not2b 3y ago[flagged]
- fermuch 3y agocURL is also embedded in a lot of applications. An exploit within curl could lead to a vulnerability in the host application.
- martinky24 3y agoThe analogy sounds great and catchy (and sufficiently contrarian to get knee jerk upvotes from the site), but it really oversimplifies things and misses the mark on what types of problems fuzzing catches and why that's incredibly important in a project as popular as curl.
- tgv 3y agoIndeed. It's more of an argument not to execute scripts downloaded from the web. Which is valid, but not the point of fortifying curl.
- deleted 3y ago[deleted]
- ape4 3y agoWould it make sense for cURL to detect when its being piped into `sh`? Possibly have it refuse to proceed unless the site is on a whitelist or the user has added an --i-accept-the-risk option.
- tgv 3y agoThat's not possible in general, I think. Can you even get process information out of file descriptors? But the people who write the scripts that embed such curl commands, will add the option directly. And others will put it in the command to copy. Or put | cat | sh in the command. And there's also wget. I think it's not something curl can tackle by itself. Instead, you'd need a file descriptor that informs you about the source, and the shell should refuse not whitelisted sources.
- ape4 3y agoYou're right it subverted in general. isatty() can tell if output is being piped. When piped special safety rules could kick in. Of course they would be false positives but then there could be an option (like I mentioned above).
- not2b 3y agoI agree that fuzzing is a very important technique, and as I said it's worth fixing any issues found, but I still think that many of the most common uses of curl are very problematic. As you can see, I didn't get any knee jerk upvotes.
- masto 3y agoIt's a tired argument. Running any program has risks.
- deleted 3y ago[deleted]
- SamuelAdams 3y agoI am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.
- epistasis 3y agoThere's a tradeoff for sure, but to fully evaluate that tradeoff you must also take into account the future time spent on the code base, including the amount of time needed to debug future features, adopt a new build environment, allow new team members to muck around and see what does and does not work... If it's a small project that won't be extended much, then perhaps then re-runnable unit tests may not make the bar as a net positive tradeoff. But some time must still be spent testing the code, even if it's not tests that are written in code.
- avgcorrection 3y agoThe best-case IMO is a test suite where writing the test is almost as easy (or maybe easier?) than doing the equivalent manual test. The test code is maybe 2–4x longer than the change. The worst case is… arbitrarily bad, to the point of being impossible. Test setup is hell because there are too many dependencies. You have to mock the “real world” since (again) things depend on other things too much and you can’t really do a simple string-manipulating change without setting up a whole environment. Also you introduce bugs in your tests that take longer to debug than the real change you are making. What I feel that we (current project) have fallen into the trap of is that there wasn’t a test suite from the start. Just manual testing. Then when you get to a stage where you feel you need one, the code base is not structured to make automated testing simple.
- bmcniel 3y agoDon't test protect against future unintended regressions? If all code was write-only then testing is probably a waste of time but code changes constantly.
- yjftsjthsd-h 3y ago
- cpill 3y ago[flagged]
- gavinhoward 3y agoGenerating random test cases, seeing what path each random case goes through, and then trying to generate cases to cover as many paths as possible. I love fuzzing.
- Jtsummers 3y agoThrow random inputs at a system and see what happens. The primary purpose is to trigger crashes or going into some other error state or the like. It's useful for checking the stability/robustness of a system. https://en.wikipedia.org/wiki/Fuzzing https://en.wikipedia.org/wiki/Fuzzing
- nickpsecurity 3y agoHere’s a survey of fuzzing techniques: https://arxiv.org/pdf/1812.00140.pdf https://arxiv.org/pdf/1812.00140.pdf One of the earliest examples was a person who fuzzed many applications by getting them to load files he created. His files triggered many vulnerabilities in apps. His method (IIRC): take a file, flip a bit, load it, repeat throughout the file. Amazing how effective it was.
- PoignardAzur 3y agoI don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?
- pstrateman 3y agoThe file format they choose is difficult for a fuzzer to produce valid examples by random chance. The file format isn't what's being fuzzed, so trying to accept as many things as possible as valid is useful. It's a trick to make the fuzzer faster.
- gavinhoward 3y agoNot an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid input in the bunch. Using a custom mutator and returning a dummy value will give the fuzzer a starting point from a valid input and makes generating other valid inputs more likely. For my part, I prefer to use custom mutators to generate valid test cases most of the time, but I want some invalid inputs because error handling is where most bugs are.
- vlovich123 3y agoI personally would separate that into two separate fuzz cases - one that generates only valid inputs and one that generates only invalid inputs and spend more resources on verifying the latter because validating invalid inputs is more important. I didn’t read the article but I like property testing for this where your mutator takes random values and uses that to generate a valid input somehow rather than stubbing in a static value. Where I can use a static value is where I wouldn’t be fuzzing the validation of that value. Of course certs are complicated beasts so I’m sure the cURL people did what made sense to them.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- rurban 3y ago[flagged]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- mikescott4 3y ago[flagged]
- throwaway81523 3y agoLots of good info here about how they added more fuzz tests, but no mention whether the fuzzing actually found bugs?
- userbinator 3y agoIf they did, I'm sure they would mention it.
- costco 3y agoOSS-fuzz is great in many cases but as this post hints at it seems like a lot of the time the fuzzers miss things because no one actually reviews the coverage info. Not in this case, but sometimes it's like someone just found the functions with signatures like parse(unsigned char *data, size_t size) just to get the money from Google for integrating fuzzing. Then again some things are just difficult to fuzz properly. I tried writing a libpurple IRC harness by doing it the "right way" with the 3000 libpurple event callbacks you are supposed to set up and structures you are supposed to allocate, which worked, but it was very slow. I ended up being lazy and only calling irc_parse_msg after modifying the code to remove error logging that required proper libpurple setup.