4 ms·

Never did a security audit, but I did regulatory and financial audits of banks. Most of our work involved looking at last year's work, re-performing it and chan

by jasong 4y ago

Never did a security audit, but I did regulatory and financial audits of banks. Most of our work involved looking at last year's work, re-performing it and changing dates. Writing reports + financial statement notes followed a similar process. Exceptions are immediately escalated to the audit committee and sometimes end up as a small footnote in the public reports. Most of the time it says "we were unable to collect sufficient evidence" to provide assurance. Almost never "this was done wrong". It's interesting to see how the second report differed from their first assessment earlier in the year: https://research.nccgroup.com/2021/04/08/public-report-vpn-by-google-one-technical-security-privacy-assessment/ https://research.nccgroup.com/2021/04/08/public-report-vpn-b... Most of the findings in the first report were "Fixed".

tptacek 4y ago
This is a good call-out. The software security field uses the term "audit" in a deeply weird way. A real audit, like the kind done in a SOC2 assessment, really is about the report, and is more like 90% paperwork than 50%. Here we're talking about software pentesting, and consulting firms have historically used the word "audit" to elevate that work. But these engagements almost never follow the shape of a real audit; there's no spelled-out audit criteria, there's no reconciliation against previous results, and the focus of the engagement is almost purely the exceptions, with little to no documentation of the tests completed without findings.
- thaumasiotes 4y ago
  You've noted elsewhere in the thread that pentesting has the concept of a "retest". A retest purely examines the findings of some original earlier report. Any other vulnerabilities are out of scope, no matter how serious. This seems like a better match to the regulatory audit model, but it's a bad match to the problem people would like to think of audits as solving, which is determining "are there any problems?". The normal pentesting use of "audit" draws on that intuitive meaning of the word; the concept is that you answer whether problems exist. But it's deeply weird anyway, because the answer can't be known, only guessed. I seem to remember PCI being called out as a very different report type from the usual, closer to the regulatory audit model at least in that the process was heavily regulated and old problems had to be fixed and retested. I never did a PCI report; don't hold me to anything.
  - tptacek 4y ago
    Yes, exactly. There is a weird conflict of interest thing happening with a lot of public-facing security assessment work. The client wants a clean bill of health. The delivery consultants can't honestly sell that, at least not without a huge project scope stretching into double-digit person/months. But the firm wants to sell engagements, and public reports are a condition of the engagement. So we have this phenomenon of "audit reports" that are really anything but that. Very few people in the industry know how to read them (for instance, how to locate and evaluate the scope of the project). But they're effectively used as seals of approval by clients. Which creates an even bigger incentive for firms to sell them, to the point where there are firms that almost specialize in doing them. PCI is closer to the audit model, and yet even less effective than the pentest model, because the standardized delivery model created a race to the bottom effect in the market. My oddball position on this stuff: firms shouldn't do external reports at all, and clients should just be able to post their internal reports.

by jasong 4y ago

tptacek 4y ago

This is a good call-out. The software security field uses the term "audit" in a deeply weird way. A real audit, like the kind done in a SOC2 assessment, really is about the report, and is more like 90% paperwork than 50%. Here we're talking about software pentesting, and consulting firms have historically used the word "audit" to elevate that work. But these engagements almost never follow the shape of a real audit; there's no spelled-out audit criteria, there's no reconciliation against previous results, and the focus of the engagement is almost purely the exceptions, with little to no documentation of the tests completed without findings.

thaumasiotes 4y ago
You've noted elsewhere in the thread that pentesting has the concept of a "retest". A retest purely examines the findings of some original earlier report. Any other vulnerabilities are out of scope, no matter how serious. This seems like a better match to the regulatory audit model, but it's a bad match to the problem people would like to think of audits as solving, which is determining "are there any problems?". The normal pentesting use of "audit" draws on that intuitive meaning of the word; the concept is that you answer whether problems exist. But it's deeply weird anyway, because the answer can't be known, only guessed. I seem to remember PCI being called out as a very different report type from the usual, closer to the regulatory audit model at least in that the process was heavily regulated and old problems had to be fixed and retested. I never did a PCI report; don't hold me to anything.
- tptacek 4y ago
  Yes, exactly. There is a weird conflict of interest thing happening with a lot of public-facing security assessment work. The client wants a clean bill of health. The delivery consultants can't honestly sell that, at least not without a huge project scope stretching into double-digit person/months. But the firm wants to sell engagements, and public reports are a condition of the engagement. So we have this phenomenon of "audit reports" that are really anything but that. Very few people in the industry know how to read them (for instance, how to locate and evaluate the scope of the project). But they're effectively used as seals of approval by clients. Which creates an even bigger incentive for firms to sell them, to the point where there are firms that almost specialize in doing them. PCI is closer to the audit model, and yet even less effective than the pentest model, because the standardized delivery model created a race to the bottom effect in the market. My oddball position on this stuff: firms shouldn't do external reports at all, and clients should just be able to post their internal reports.