5 ms·

I'm not sure I read your final remark correctly. Do you mean to say that using Skipfish or sqlmap has no place in a professional pen test or that a professional

I'm not sure I read your final remark correctly. Do you mean to say that using Skipfish or sqlmap has no place in a professional pen test or that a professional pen tester should not subject his clients to the raw results from said tools? The latter I agree with, the former I don't. Any automated security scanner has its shortcomings - combining the results of various scanner products generally has a positive effect upon the quality of the test. I've often been in a situation where one tool (commercial or otherwise) indicated or hinted at a vulnerability while all other tools (commercial or otherwise) used did not indicate anything.

tptacek 14y ago
The former. With some very minor exceptions, we don't use automated scanners at all in our practice. We use Burp to set up testing experiments, and we use ruby for automation. We've found that "scanners" make good testers much less sharp; if the tool spits out the names of specific vulnerabilities, it's suspect. But even practices that do use automated scanners won't be relying on Skipfish or sqlmap. This obviously doesn't hold for network pentesters. For tests in which application flaws aren't the whole story, and you're just looking for the fastest way in you can find, sure; sqlmap is valuable there, much more so than Skipfish. We don't do those kinds of tests often.
- jvdongen 14y ago
  Thanks for your clarification. Phrased this way I tend to agree with you.
- ontoillogical 14y ago
  When you say "we use ruby for automation" do you mean using something like buby with Burp, or something custom? I find myself writing a lot of custom fuzzers in python, and am looking to improve my toolkit.
  - tptacek 14y ago
    A long time ago, Eric Monti, then a member of our team and now at a big startup in SF, wrote a Burp/Ruby bridge called Buby. For a long time, and maybe still, another team member --- Timur Duehr, who also wrote the first version of our Ragweed native code debugger DSL --- maintained it. A lot of people on our team use Buby to automate. Another faction of our team uses Mike Tracy's WWMD toolkit, which does some of what Burp does but from an irb prompt. I'm allergic to both approaches and tend to just start from EventMachine and ev-http-request. Good example of the kind of scenario that'll get me into "writing code to test web apps" mode: testing those goofy cryptographic tokens applications send via email to reset passwords.
    - ScottBurson 14y ago
      "Goofy" cryptographic tokens? Is there a better way?
      - tptacek 14y ago
        Not having the tokens be replayable, rewritable to different user IDs, or decryptable would be a good start, right? Yes: the better approach is to have the token be a 256 bit cryptographically strong random number that corresponds to a row in the database with metadata sufficient to expire the token and invalidate it on password reset.
        ScottBurson 14y ago
        Ah. So it's not the use of cryptographic tokens per se that you're objecting to -- just that they're not generated and/or handled correctly.
        tptacek 14y ago
        I object in general to the widespread practice of using tokens that contain semantically interesting information protected by encryption.
        14y ago
  - eru 14y ago
    > I find myself writing a lot of custom fuzzers in python, and am looking to improve my toolkit. Have you looked into Haskell's QuickCheck? It's essentially a fuzzer for unit testing. You might find some interesting ideas there.
    - ontoillogical 14y ago
      Thanks for the tip. I've just been looking at Learn You a Haskell, it's nice to have some specific usecases when learning a new language
spydum 14y ago
I think the point is automated tools are a nice way of directing your pen test, sort of like the way you might use binoculars to scan your surroundings while hiking to spot interesting terrain you might wish to explore.
- tptacek 14y ago
  This is the idea that scares me the most, because while it sounds great, it has an obvious flip side: wherever the scanner doesn't "focus" you is getting less attention.
  - JangoSteve 14y ago
    Could you not do your normal practice, and then have someone else run the automated scanners as a sort of double-check just to make sure you didn't miss anything obvious? Sure, it would likely never find anything, as it's your job to not miss anything obvious, but if they're basically automated and free, seems odd to dismiss them outright. EDIT: Furthermore, how embarrassing would it be if someone hired you and then ran one of these scanners themselves and did find something you missed? Given the magnitude of the potential downside, and the marginal cost of using them as described, it'd almost be more of a business insurance tactic than anything else.
    - tptacek 14y ago
      First, we're a pretty big company, and I'm only one of 3 founders and of 4 practice managers, so you can imagine we've debated this pretty thoroughly. Last point first: Try real hard not to miss stuff. Seriously, that's it. Appsec is a competitive field. Forget scanners: many clients are going to hire a different firm for their next test, and some of those firms are really, really good. When you're testing an app, that is the thing that is animating your work: trying not to leave nuggets for the next team to find and shame you with. Think we're worried about missing things sqlmap finds? Try worrying about what iSec Partners is going to find. To your first point: this is hard to articulate well, but let me take a stab at it: it's very hard to double back over terrain a scanner has already covered. Think of security testing like a treasure hunt. Have you ever had a fun treasure hunt? Can you get a bead on the feeling you had when you started out hunting for stuff? Imagine trying to put the same focus and motivation together if, before the treasure hunt, the organizers announced "we sent a bunch of people out before you to try to make sure there's no treasure".

tptacek 14y ago

The former. With some very minor exceptions, we don't use automated scanners at all in our practice. We use Burp to set up testing experiments, and we use ruby for automation. We've found that "scanners" make good testers much less sharp; if the tool spits out the names of specific vulnerabilities, it's suspect. But even practices that do use automated scanners won't be relying on Skipfish or sqlmap. This obviously doesn't hold for network pentesters. For tests in which application flaws aren't the whole story, and you're just looking for the fastest way in you can find, sure; sqlmap is valuable there, much more so than Skipfish. We don't do those kinds of tests often.

jvdongen 14y ago
Thanks for your clarification. Phrased this way I tend to agree with you.
ontoillogical 14y ago
When you say "we use ruby for automation" do you mean using something like buby with Burp, or something custom? I find myself writing a lot of custom fuzzers in python, and am looking to improve my toolkit.
- tptacek 14y ago
  A long time ago, Eric Monti, then a member of our team and now at a big startup in SF, wrote a Burp/Ruby bridge called Buby. For a long time, and maybe still, another team member --- Timur Duehr, who also wrote the first version of our Ragweed native code debugger DSL --- maintained it. A lot of people on our team use Buby to automate. Another faction of our team uses Mike Tracy's WWMD toolkit, which does some of what Burp does but from an irb prompt. I'm allergic to both approaches and tend to just start from EventMachine and ev-http-request. Good example of the kind of scenario that'll get me into "writing code to test web apps" mode: testing those goofy cryptographic tokens applications send via email to reset passwords.
  - ScottBurson 14y ago
    "Goofy" cryptographic tokens? Is there a better way?
    - tptacek 14y ago
      Not having the tokens be replayable, rewritable to different user IDs, or decryptable would be a good start, right? Yes: the better approach is to have the token be a 256 bit cryptographically strong random number that corresponds to a row in the database with metadata sufficient to expire the token and invalidate it on password reset.
      - ScottBurson 14y ago
        Ah. So it's not the use of cryptographic tokens per se that you're objecting to -- just that they're not generated and/or handled correctly.
        tptacek 14y ago
        I object in general to the widespread practice of using tokens that contain semantically interesting information protected by encryption.
        14y ago
- eru 14y ago
  > I find myself writing a lot of custom fuzzers in python, and am looking to improve my toolkit. Have you looked into Haskell's QuickCheck? It's essentially a fuzzer for unit testing. You might find some interesting ideas there.
  - ontoillogical 14y ago
    Thanks for the tip. I've just been looking at Learn You a Haskell, it's nice to have some specific usecases when learning a new language

spydum 14y ago

I think the point is automated tools are a nice way of directing your pen test, sort of like the way you might use binoculars to scan your surroundings while hiking to spot interesting terrain you might wish to explore.

tptacek 14y ago
This is the idea that scares me the most, because while it sounds great, it has an obvious flip side: wherever the scanner doesn't "focus" you is getting less attention.
- JangoSteve 14y ago
  Could you not do your normal practice, and then have someone else run the automated scanners as a sort of double-check just to make sure you didn't miss anything obvious? Sure, it would likely never find anything, as it's your job to not miss anything obvious, but if they're basically automated and free, seems odd to dismiss them outright. EDIT: Furthermore, how embarrassing would it be if someone hired you and then ran one of these scanners themselves and did find something you missed? Given the magnitude of the potential downside, and the marginal cost of using them as described, it'd almost be more of a business insurance tactic than anything else.
  - tptacek 14y ago
    First, we're a pretty big company, and I'm only one of 3 founders and of 4 practice managers, so you can imagine we've debated this pretty thoroughly. Last point first: Try real hard not to miss stuff. Seriously, that's it. Appsec is a competitive field. Forget scanners: many clients are going to hire a different firm for their next test, and some of those firms are really, really good. When you're testing an app, that is the thing that is animating your work: trying not to leave nuggets for the next team to find and shame you with. Think we're worried about missing things sqlmap finds? Try worrying about what iSec Partners is going to find. To your first point: this is hard to articulate well, but let me take a stab at it: it's very hard to double back over terrain a scanner has already covered. Think of security testing like a treasure hunt. Have you ever had a fun treasure hunt? Can you get a bead on the feeling you had when you started out hunting for stuff? Imagine trying to put the same focus and motivation together if, before the treasure hunt, the organizers announced "we sent a bunch of people out before you to try to make sure there's no treasure".