3 ms·

Please don't start this way. Michal Zalewski didn't write afl fuzzer so it would be used without deep technical knowledge. It's actually not for beginners at al

by dsacco 12y ago

Please don't start this way. Michal Zalewski didn't write afl fuzzer so it would be used without deep technical knowledge. It's actually not for beginners at all despite how powerful it is. As a general rule to start in security work, you will want to know how to program in at least one language competently, then move on to theoretical understanding of vulnerabilities, then finding them painstakingly by hand, and finally by automation. It's perfectly fine to use afl (or other similar tools), but understand that using an automated tool without first knowing how to do things by hand and groking the theory will stunt your growth and make it hard to progress very far. Pop open a weakened VM and exploit a few buffer overflow bugs the old fashioned way before you use a fuzzer. With code review, learn the nuances of each language before you automate your audit. You shouldn't have to rely on an automated tool to find out register globals is on in PHP (as one example).

exDM69 12y ago
I agree with you if we're talking about someone who is just getting started in programming and debugging. It's not a good starting point if you want to understand why these bugs exist or how to exploit them. But if we're talking about someone who is already a competent programmer, perhaps looking to learn a new tool, there's just no point (except perhaps learning) in trying to find bugs manually in production software by reading code or by trying to write test inputs that invoke unwanted behavior. After all, most code works fine at least 99% of the time, finding the 1% manually is a lot of effort. Tools like afl-fuzz or static analysis tools are so much faster (when you throw enough CPU time at them) in finding unwanted behavior that there aren't many reasons not to use them. For learning and getting started with exploits, I like your idea of using some kind of "weakened" VM, but that's a whole different deal than finding and fixing bugs (not even talking about exploiting) in production code (which is where the bounties are).
dalke 12y ago
I realize there's a philosophy that it's best to start from the lowest levels and work one's way up. It's one I can empathize with, in spirit, though I disagree. The same logic means that people shouldn't use static code analysis tools, or valgrind, or even debuggers, until they acquire deep technical knowledge. While I think all of these tools help reinforce the principles. If someone starts with the fixed and unwavering goal of security analysis in mind, then perhaps I can agree with you. If however someone is only curious about security analysis, and finds that spending a year to "grok the theory" is a high barrier, then even clumsy use of semi-automated tools may provide more concrete incentive to learn the underlying skills. While I don't believe you are correct, another question is, how many white hats do we end up with? Even if it takes 2 years to learn security skills by using automation, and only 1 year without automation, if after 10 years there are 500 following your path, while 10,000 following my path, then that's a net gain for the good side, yes? (I assume that there is such a thing as "good enough", and that it's relatively stable. Obviously if there is only a market for 500, and your 500 are always better than my 10,000 then that changes the dynamics.) Finally, it's also good to have even the script kiddies on the side of good than the side of lolz.
- dsacco 12y ago
  >> The same logic means that people shouldn't use static code analysis tools, or valgrind, or even debuggers, until they acquire deep technical knowledge. While I think all of these tools help reinforce the principles. This probably sounds controversial, but I agree. I don't think you should use Valgrind until you understand how Valgrind works. This doesn't mean early C programmers shouldn't use Valgrind - you can read the documentation and theory behind Valgrind in a day. But definitely do that. I think maybe "deep technical knowledge" wasn't the right term for me to use. A better term would be "technical understanding" - know how it works, and know how to find the different classes of bugs it can find, but you do not need to be capable of writing the tool yourself. Now let me clarify this, and my earlier point about afl - I think you should use them, and generously, and pretty much always once you know what you're doing. But if you use them without understanding the fundamentals, you will get caught up in false positives/negatives. Always use it as a supplement, not a crutch. However, I agree with what you're saying about someone's level of dedication to security analysis. Using afl is better than not using afl, so if you're not a dedicated security guy, then you're definitely right that someone should use it. I would still caution anyone that using an automated tool without fully understanding how it works will lead to an incomplete picture of the application's security posture.
  - dalke 12y ago
    Well in that case, I have only a surface level understanding of Valgrind, but I use it for those rare times I code in C. I also don't know how afl works at the code modification level, other than that it asks gcc to add some instrumentation, but I've used it to find bugs. :) I completely agree with your last statement, with one proviso. You will always have an incomplete picture of the application's security posture. The fundamentals include hardware and OS behaviors, power consumption, RF emissions, and other fields which are very far from purity of the programming language model. Acquisition of all of this knowledge is very difficult, but is also rarely needed. I think for most cases it's okay to have an incomplete understanding of the tools. It depends on the threat model.
  - tptacek 12y ago
    This idea of people not using tools until they understand how they work (or in the Matasano shorthand, until they can implement them, in "build your own light saber" fashion) is I think very true of professional testers, but not true for developers. If your full-time job is going to be about developing a radar for assumptions that developers make and a deviousness about breaking those assumptions, tools --- and most especially opaque tools --- totally hamstring you. We both know plenty of testers that develop a kind of blindness from relying on tools too much. But I'm guessing most of HN doesn't really understand the mindset that's involved in pentesting full-time. I certainly wouldn't tell a developer that they should avoid valgrind.

by dsacco 12y ago

exDM69 12y ago

I agree with you if we're talking about someone who is just getting started in programming and debugging. It's not a good starting point if you want to understand why these bugs exist or how to exploit them. But if we're talking about someone who is already a competent programmer, perhaps looking to learn a new tool, there's just no point (except perhaps learning) in trying to find bugs manually in production software by reading code or by trying to write test inputs that invoke unwanted behavior. After all, most code works fine at least 99% of the time, finding the 1% manually is a lot of effort. Tools like afl-fuzz or static analysis tools are so much faster (when you throw enough CPU time at them) in finding unwanted behavior that there aren't many reasons not to use them. For learning and getting started with exploits, I like your idea of using some kind of "weakened" VM, but that's a whole different deal than finding and fixing bugs (not even talking about exploiting) in production code (which is where the bounties are).

dalke 12y ago

I realize there's a philosophy that it's best to start from the lowest levels and work one's way up. It's one I can empathize with, in spirit, though I disagree. The same logic means that people shouldn't use static code analysis tools, or valgrind, or even debuggers, until they acquire deep technical knowledge. While I think all of these tools help reinforce the principles. If someone starts with the fixed and unwavering goal of security analysis in mind, then perhaps I can agree with you. If however someone is only curious about security analysis, and finds that spending a year to "grok the theory" is a high barrier, then even clumsy use of semi-automated tools may provide more concrete incentive to learn the underlying skills. While I don't believe you are correct, another question is, how many white hats do we end up with? Even if it takes 2 years to learn security skills by using automation, and only 1 year without automation, if after 10 years there are 500 following your path, while 10,000 following my path, then that's a net gain for the good side, yes? (I assume that there is such a thing as "good enough", and that it's relatively stable. Obviously if there is only a market for 500, and your 500 are always better than my 10,000 then that changes the dynamics.) Finally, it's also good to have even the script kiddies on the side of good than the side of lolz.

dsacco 12y ago
>> The same logic means that people shouldn't use static code analysis tools, or valgrind, or even debuggers, until they acquire deep technical knowledge. While I think all of these tools help reinforce the principles. This probably sounds controversial, but I agree. I don't think you should use Valgrind until you understand how Valgrind works. This doesn't mean early C programmers shouldn't use Valgrind - you can read the documentation and theory behind Valgrind in a day. But definitely do that. I think maybe "deep technical knowledge" wasn't the right term for me to use. A better term would be "technical understanding" - know how it works, and know how to find the different classes of bugs it can find, but you do not need to be capable of writing the tool yourself. Now let me clarify this, and my earlier point about afl - I think you should use them, and generously, and pretty much always once you know what you're doing. But if you use them without understanding the fundamentals, you will get caught up in false positives/negatives. Always use it as a supplement, not a crutch. However, I agree with what you're saying about someone's level of dedication to security analysis. Using afl is better than not using afl, so if you're not a dedicated security guy, then you're definitely right that someone should use it. I would still caution anyone that using an automated tool without fully understanding how it works will lead to an incomplete picture of the application's security posture.
- dalke 12y ago
  Well in that case, I have only a surface level understanding of Valgrind, but I use it for those rare times I code in C. I also don't know how afl works at the code modification level, other than that it asks gcc to add some instrumentation, but I've used it to find bugs. :) I completely agree with your last statement, with one proviso. You will always have an incomplete picture of the application's security posture. The fundamentals include hardware and OS behaviors, power consumption, RF emissions, and other fields which are very far from purity of the programming language model. Acquisition of all of this knowledge is very difficult, but is also rarely needed. I think for most cases it's okay to have an incomplete understanding of the tools. It depends on the threat model.
- tptacek 12y ago
  This idea of people not using tools until they understand how they work (or in the Matasano shorthand, until they can implement them, in "build your own light saber" fashion) is I think very true of professional testers, but not true for developers. If your full-time job is going to be about developing a radar for assumptions that developers make and a deviousness about breaking those assumptions, tools --- and most especially opaque tools --- totally hamstring you. We both know plenty of testers that develop a kind of blindness from relying on tools too much. But I'm guessing most of HN doesn't really understand the mindset that's involved in pentesting full-time. I certainly wouldn't tell a developer that they should avoid valgrind.