3 ms·

> Nobody spends 50% of their time writing reports. If anything, it's the exact opposite At my employer we don't, but a friend said that 50% of their time at a

by lucb1e 4y ago

> Nobody spends 50% of their time writing reports. If anything, it's the exact opposite At my employer we don't, but a friend said that 50% of their time at a big multinational who does pentesting and other auditing services was reporting and fighting the MS Word template. Apparently it's not that abnormal and hard enough to change that my friend would rather leave than work with the team to get it fixed. > This "multiple rounds of review" thing is news to me too. Depends on the definition, like do you count your teammates reading what you wrote on top of the boss checking off on the report? If yes, then we have multiple rounds of reviews for every report. If no, then we only have it for certain (not all) public reports. Again, not a weird thing. (To get a bit meta, your writing style across any security-related comments is a bit "this is definitively how it definitely is" whereas... well, in this case it's objectively not true. Some orgs do spend (waste) disproportionate amounts of time on reporting.) > the ordinary process for a project like this with a public report is, you deliver the whole project as if there wasn't a public report, with an internal report, and then the public report is based on the internal one (probably post-remediation) This is also not how we do it. We typically know beforehand if it is going to be public and the report that gets sent to the customer and released to the public is literally the same file. It's not that a second report gets written "based on" the real report. That's how we maintain integrity as a third-party auditor: we don't sugar-coat or alter our reports if it's going to be public. If desired, both for public and nonpublic, we can do a retest and add remediation status. What is different is that we triple check for typos, lines running out of the page, etc., and might be more verbose so a wider audience can understand the findings (some customers are very technical, others not at all; for a public report we mostly switch to non-technical-customer mode). Interesting to know that this is not, at least not ordinarily, the case at places you work(ed).

tptacek 4y ago
I can't speak to the big multinational your friend works at, only NCC, and, I guess, a bunch of boutique-y firms that don't qualify as "big multinationals" the way NCC does. And, just generally: if you're contracting software pentests, you should not be OK with the prospect of your consultants spending 50% of their time in doc. That isn't a norm. You should expect most of the time to be spent looking for the bugs you're paying for them to find. After your edit With respect to your thing about how public reports work, I appreciate the intellectual honesty of the approach your firm takes, and I have real problems with public reports in general (something I've ranted about here before). But at a big US pentest firm I think it'd be hard to do public reports any other way. For one thing: clients routinely dispute findings, and they're often right (they know more about their own code than the testers do). You'd have to let them see the report before you could write a version of the report for the public, so you can scrub out the findings that don't pan out. Another obvious issue is that reports often include tentative findings, which need to be confirmed before the report is finalized. I'm interested, just as a security consulting nerd, in how your process handles these things. But past that, the question that started this thread was whether having Google breathing down your neck would make this a hard engagement to deliver, and I think the answer to that is almost certainly "no". If anything, Google wants good findings, and the stressful part of this engagement would be coming up with a report that never gets anything more than a sev:med. (I don't know, Google's a big company, I haven't done a software assessment in over a year, &c &c).
- lucb1e 4y ago
  > I guess, a bunch of boutique-y firms that don't qualify as "big multinationals" the way NCC does NCC has on the order of 1% of the revenue of this "boutique-y firm". I'd mention the name but then I'd want to check with $friend first in case it would somehow make them identifiable. > you should not be OK with the prospect of your consultants spending 50% of their time in doc Agreed on that
  - tptacek 4y ago
    Just so we're clear: I'm saying my personal experience extends only as far as a single org of NCC's size, and then a variety of boutique-y firms of ~Matasano's size. I have no experience with Deloitte or PwC. I'm not calling your friend's firm a boutique; it's clearly not.
    - lucb1e 4y ago
      Ah, sorry that was a misunderstanding on my part!
- lucb1e 4y ago
  (Sorry about the substantial edit after you apparently already read my comment. Most of the time people don't see it that fast ^^') > at a big US pentest firm I think it'd be hard to do public reports any other way [...] You'd have to let [clients] see the report before you could write a version of the report for the public, so you can scrub out the findings that don't pan out. We do let them see before, also because there is indeed time needed to roll out fixes, but we aren't often asked to make substantial changes (and if they ask, whether we can oblige depends). Sometimes we make a real mistake and then it just looks silly for both parties so that would be removed, but usually things are just factually correct. For example, we can't always know about backports when we see an old version (and PoCs / detailed info is notoriously missing from CVEs, one of my pet peeves), so we recommend they check that the thing is up to date and implement an update procedure if it's not. That advice is valid no matter the actual state. We can mark it as checked and OK as part of a retest (if we got ssh to see for ourselves, or we would write something like "$customer checked and says it was up to date" as the retest description alongside a 'resolved' marker). > Another obvious issue is that reports often include tentative findings, Is this like an int overflow that you're not yet sure is exploitable or so? Or do you have an example of this? I'm not familiar with the concept of a tentative finding unless it's like a little note "this search function at /search.php produces a weird error with input {{1+1}} but I haven't found a useful bug yet, todo investigate more". If it's the latter, we don't report those if we couldn't find anything clearly wrong with it. If it's just not following the spec (HTML entity encoding issues are a common example) but we don't find a security flaw, then it's an informational note at best. Maybe we should be more pushy with these in case it turns out to be exploitable. > the question that started this thread was whether having Google breathing down your neck would make this a hard engagement to deliver, and I think the answer to that is almost certainly "no". I mostly agree there. It's definitely a bit more work with our process because what the test team sends out for internal review is supposed to be basically ready to publish, so they'll do more double checking of the facts before git committing the finding and such. But at the same time, because the routine is almost identical for private and public reports, it's not very special even for big customers that people here would recognize. Is the engagement harder? Slightly, because you don't want to make the firm look stupid by doing something wrong. Does it qualify for the statement "hard to deliver"? Nah, I agree that this answer is "no". > the stressful part of this engagement would be coming up with a report that never gets anything more than a sev:med This. Or even nothing above 'low' if the attack surface is small and the devs are up to date on the relevant security best practices.

by lucb1e 4y ago

tptacek 4y ago

I can't speak to the big multinational your friend works at, only NCC, and, I guess, a bunch of boutique-y firms that don't qualify as "big multinationals" the way NCC does. And, just generally: if you're contracting software pentests, you should not be OK with the prospect of your consultants spending 50% of their time in doc. That isn't a norm. You should expect most of the time to be spent looking for the bugs you're paying for them to find. After your edit With respect to your thing about how public reports work, I appreciate the intellectual honesty of the approach your firm takes, and I have real problems with public reports in general (something I've ranted about here before). But at a big US pentest firm I think it'd be hard to do public reports any other way. For one thing: clients routinely dispute findings, and they're often right (they know more about their own code than the testers do). You'd have to let them see the report before you could write a version of the report for the public, so you can scrub out the findings that don't pan out. Another obvious issue is that reports often include tentative findings, which need to be confirmed before the report is finalized. I'm interested, just as a security consulting nerd, in how your process handles these things. But past that, the question that started this thread was whether having Google breathing down your neck would make this a hard engagement to deliver, and I think the answer to that is almost certainly "no". If anything, Google wants good findings, and the stressful part of this engagement would be coming up with a report that never gets anything more than a sev:med. (I don't know, Google's a big company, I haven't done a software assessment in over a year, &c &c).

lucb1e 4y ago
> I guess, a bunch of boutique-y firms that don't qualify as "big multinationals" the way NCC does NCC has on the order of 1% of the revenue of this "boutique-y firm". I'd mention the name but then I'd want to check with $friend first in case it would somehow make them identifiable. > you should not be OK with the prospect of your consultants spending 50% of their time in doc Agreed on that
- tptacek 4y ago
  Just so we're clear: I'm saying my personal experience extends only as far as a single org of NCC's size, and then a variety of boutique-y firms of ~Matasano's size. I have no experience with Deloitte or PwC. I'm not calling your friend's firm a boutique; it's clearly not.
  - lucb1e 4y ago
    Ah, sorry that was a misunderstanding on my part!
lucb1e 4y ago
(Sorry about the substantial edit after you apparently already read my comment. Most of the time people don't see it that fast ^^') > at a big US pentest firm I think it'd be hard to do public reports any other way [...] You'd have to let [clients] see the report before you could write a version of the report for the public, so you can scrub out the findings that don't pan out. We do let them see before, also because there is indeed time needed to roll out fixes, but we aren't often asked to make substantial changes (and if they ask, whether we can oblige depends). Sometimes we make a real mistake and then it just looks silly for both parties so that would be removed, but usually things are just factually correct. For example, we can't always know about backports when we see an old version (and PoCs / detailed info is notoriously missing from CVEs, one of my pet peeves), so we recommend they check that the thing is up to date and implement an update procedure if it's not. That advice is valid no matter the actual state. We can mark it as checked and OK as part of a retest (if we got ssh to see for ourselves, or we would write something like "$customer checked and says it was up to date" as the retest description alongside a 'resolved' marker). > Another obvious issue is that reports often include tentative findings, Is this like an int overflow that you're not yet sure is exploitable or so? Or do you have an example of this? I'm not familiar with the concept of a tentative finding unless it's like a little note "this search function at /search.php produces a weird error with input {{1+1}} but I haven't found a useful bug yet, todo investigate more". If it's the latter, we don't report those if we couldn't find anything clearly wrong with it. If it's just not following the spec (HTML entity encoding issues are a common example) but we don't find a security flaw, then it's an informational note at best. Maybe we should be more pushy with these in case it turns out to be exploitable. > the question that started this thread was whether having Google breathing down your neck would make this a hard engagement to deliver, and I think the answer to that is almost certainly "no". I mostly agree there. It's definitely a bit more work with our process because what the test team sends out for internal review is supposed to be basically ready to publish, so they'll do more double checking of the facts before git committing the finding and such. But at the same time, because the routine is almost identical for private and public reports, it's not very special even for big customers that people here would recognize. Is the engagement harder? Slightly, because you don't want to make the firm look stupid by doing something wrong. Does it qualify for the statement "hard to deliver"? Nah, I agree that this answer is "no". > the stressful part of this engagement would be coming up with a report that never gets anything more than a sev:med This. Or even nothing above 'low' if the attack surface is small and the devs are up to date on the relevant security best practices.