4 ms·

There are a few bespoke consulting firms that actually try to improve their clients security beyond checklist whack-a-mole and automated scanning. Annoyingly bu

There are a few bespoke consulting firms that actually try to improve their clients security beyond checklist whack-a-mole and automated scanning. Annoyingly bunch of charlatan firms pretend to do this, but just toss an intern with a scanner at the customer and/or double book their staff so they don’t have time to think beyond the basics. This lets them always underbid the firms that do honest work. The problem is that its hard to measure if someone had deep thoughts about the security of a system vs checked a box as it can take years before you notice they did nothing (eg: hired another firm the next time that found a pile of issues). Disclaimer: I work for one of the rare non-checklist/scanner firms.

raesene9 5y ago
The pentesting industry has a big problem with being a good "market for lemons". It's very hard for customers to differentiate the good and bad companies, without having their own internal expertise, and even then you need to go down the line of getting named testers and speaking to each one. Another problem is with how many/most pentest companies report, which is by exception. There's no requirement to state all the tests they did, just the results, so it's hard to tell the difference between "we've got a good system and they didn't find much" and "they didn't do good work and missed things"
- extrapickles 5y ago
  Our reports always have a section on things we looked at or for. Doing so takes a majority of the time it takes to write the report. Since it adds to the cost, firms that don’t do this can underbid the ones that do. For slimmed down reports that are published/given to third parties (given to the customer’s customer) it’s extremely important that the scope of the test is detailed in that letter. The charlatan firms will be happy to omit the fact that they only tested the “about us” page while blind-folded. So if you are reading a “letter of assessment” for something you are thinking of adopting, just look for the scope of the test, and if it sounds reasonable they potentially had a good test. If it’s missing, the test wasn’t worth the electrons in that letter.
dsr_ 5y ago
From the other side of the fence: I've been hiring companies to do external pen tests for fifteen years now. Some of them have been giant corporations with security divisions, some of them have been just past the startup stage, and some of them are recognizable big names in the industry. I've signed one year, two year and three year contracts. A few of them have distinguished themselves, slightly, in the first year of a multi-year contract -- and then regressed to the mean in the rest. Quoted prices vary by a factor of 4, approximately. Work done does not. Quality of work appears to be basically independent of price. Arrogance scales with price, though. Even if you are a non-checklist firm, I can't justify hiring you at a higher price because I can't differentiate you from the bloviators, and basically nothing you say other than "I am ptacek" can change that.
- TecoAndJix 5y ago
  This is a good perspective. I pushed for a higher priced pen test firm in our last engagement due their name/reputation (i was sick of nessus scan outputs) but ultimately their price was just SO MUCH cheaper than the prestigious named one. I couldn't provide a justification to leadership outside of "they have these awesome writeups in their research division". Asking to see a sample report to prospective firm turns out to be a crapshoot as well (but seems to be one of the few things to go off of). Can you provide any insight on how you evaluate?
- extrapickles 5y ago
  Security unfortunately is a creative process, which is hard to get consistent. I would love to use something like statistical product control, but I have yet to see a good way to apply it (or similar techniques) without forcing pen-tests to be “follow the checklist”.
  - extrapickles 5y ago
    I should add that burnout is a massive problem when it comes to quality. Even if you found a good consultant (or group of), for round #2, they could have just finished testing all of your competitors products and are now worn out. Scheduling in variety to try and prevent partial burnout is difficult if everyone in a vertical only wants to use the same person.
jasonladuke0311 5y ago
This is what I am working hard towards, moving to a firm like yours.

raesene9 5y ago

The pentesting industry has a big problem with being a good "market for lemons". It's very hard for customers to differentiate the good and bad companies, without having their own internal expertise, and even then you need to go down the line of getting named testers and speaking to each one. Another problem is with how many/most pentest companies report, which is by exception. There's no requirement to state all the tests they did, just the results, so it's hard to tell the difference between "we've got a good system and they didn't find much" and "they didn't do good work and missed things"

extrapickles 5y ago
Our reports always have a section on things we looked at or for. Doing so takes a majority of the time it takes to write the report. Since it adds to the cost, firms that don’t do this can underbid the ones that do. For slimmed down reports that are published/given to third parties (given to the customer’s customer) it’s extremely important that the scope of the test is detailed in that letter. The charlatan firms will be happy to omit the fact that they only tested the “about us” page while blind-folded. So if you are reading a “letter of assessment” for something you are thinking of adopting, just look for the scope of the test, and if it sounds reasonable they potentially had a good test. If it’s missing, the test wasn’t worth the electrons in that letter.

dsr_ 5y ago

From the other side of the fence: I've been hiring companies to do external pen tests for fifteen years now. Some of them have been giant corporations with security divisions, some of them have been just past the startup stage, and some of them are recognizable big names in the industry. I've signed one year, two year and three year contracts. A few of them have distinguished themselves, slightly, in the first year of a multi-year contract -- and then regressed to the mean in the rest. Quoted prices vary by a factor of 4, approximately. Work done does not. Quality of work appears to be basically independent of price. Arrogance scales with price, though. Even if you are a non-checklist firm, I can't justify hiring you at a higher price because I can't differentiate you from the bloviators, and basically nothing you say other than "I am ptacek" can change that.

TecoAndJix 5y ago
This is a good perspective. I pushed for a higher priced pen test firm in our last engagement due their name/reputation (i was sick of nessus scan outputs) but ultimately their price was just SO MUCH cheaper than the prestigious named one. I couldn't provide a justification to leadership outside of "they have these awesome writeups in their research division". Asking to see a sample report to prospective firm turns out to be a crapshoot as well (but seems to be one of the few things to go off of). Can you provide any insight on how you evaluate?
extrapickles 5y ago
Security unfortunately is a creative process, which is hard to get consistent. I would love to use something like statistical product control, but I have yet to see a good way to apply it (or similar techniques) without forcing pen-tests to be “follow the checklist”.
- extrapickles 5y ago
  I should add that burnout is a massive problem when it comes to quality. Even if you found a good consultant (or group of), for round #2, they could have just finished testing all of your competitors products and are now worn out. Scheduling in variety to try and prevent partial burnout is difficult if everyone in a vertical only wants to use the same person.

jasonladuke0311 5y ago

This is what I am working hard towards, moving to a firm like yours.