4 ms·
From the other side of the fence: I've been hiring companies to do external pen tests for fifteen years now. Some of them have been giant corporations with secu
by dsr_ 5y ago
From the other side of the fence: I've been hiring companies to do external pen tests for fifteen years now. Some of them have been giant corporations with security divisions, some of them have been just past the startup stage, and some of them are recognizable big names in the industry. I've signed one year, two year and three year contracts.
A few of them have distinguished themselves, slightly, in the first year of a multi-year contract -- and then regressed to the mean in the rest.
Quoted prices vary by a factor of 4, approximately. Work done does not. Quality of work appears to be basically independent of price.
Arrogance scales with price, though.
Even if you are a non-checklist firm, I can't justify hiring you at a higher price because I can't differentiate you from the bloviators, and basically nothing you say other than "I am ptacek" can change that.
- TecoAndJix 5y agoThis is a good perspective. I pushed for a higher priced pen test firm in our last engagement due their name/reputation (i was sick of nessus scan outputs) but ultimately their price was just SO MUCH cheaper than the prestigious named one. I couldn't provide a justification to leadership outside of "they have these awesome writeups in their research division". Asking to see a sample report to prospective firm turns out to be a crapshoot as well (but seems to be one of the few things to go off of). Can you provide any insight on how you evaluate?
- extrapickles 5y agoSecurity unfortunately is a creative process, which is hard to get consistent. I would love to use something like statistical product control, but I have yet to see a good way to apply it (or similar techniques) without forcing pen-tests to be “follow the checklist”.
- extrapickles 5y agoI should add that burnout is a massive problem when it comes to quality. Even if you found a good consultant (or group of), for round #2, they could have just finished testing all of your competitors products and are now worn out. Scheduling in variety to try and prevent partial burnout is difficult if everyone in a vertical only wants to use the same person.