2 ms·

Hey Ken, hope you guys the best, good project. As a seasoned (although retired) pen-tester I wanted to say you'll have some serious problems with rating when i

Hey Ken, hope you guys the best, good project. As a seasoned (although retired) pen-tester I wanted to say you'll have some serious problems with rating when it comes to results. When a company hires pen-testers and pen-testers do or do not find stuff, the company has no idea about the coverage. So the pen-test team might have missed many stuff or identified all. I'm sure you've seen in real world even the same members of the same pen-test team might find different issues for the same test. Therefore one of the biggest problems is to actually knowing whether they are good or not at what they do. It's easy to rate communication skills, responsiveness, attitude, report quality etc. But very hard to rate the quality of the results (which is the real reason for carrying out a pen-test). When they don't find something, maybe there really is nothing there. When they found something, maybe there is more there. The customer has no idea at that point. It'll be only a fair amount of time later they'll figure out the coverage / vulnerability finding quality. I'm sure in the long run market will stabilize (assuming you can change your rating for a pen-tester even after a year) but this is something to consider. Update: BTW personally I don't like the idea of logging in via LinkedIn (for finding a security talent), it's feels too intrusive, beside of the personal preference my experience showed me especially security industry don't like SSO style things.

michaelt 11y ago
When they don't find something, maybe there really is nothing there. When they found something, maybe there is more there. Also, if you try to evaluate on "number of issues found" you encourage the testers to report junk issues to get their count up.
kenbaylor 11y ago
Some of this can be covered by the methodology section, where the pen testers shows the approach taken and has an overview of what they did. There's a happy medium between one-line reports 'nothing found' (which encourages questions like 'did you even try?'), to the voluminous crap produced like old vulnerability scanners. Providing the report template in advance may help set expectations. The industry is moving towards standards based pen testing. That has some pros but many cons as well. For the moment, setting expectations and having a thorough debriefing with the customer may have to do. Your question is the scientific one: How do I know what you did was good enough. Just like a patient evaluating a medical professional care, the customer isn't an expert and goes with their gut in some cases. That's why the industry also wants you to use different pen testers. I've seen many a time, when one team finds nothing, and another rips the infrastructure apart. Competency is a variable over time and so is trust. TLDR: The scientific question does not have a simple fix by any means.
- tptacek 11y ago
  I'm not sure I follow here. Is your firm getting access to pentest reports from the consultancies it matches to clients? That seems untenable; clients tend to be very possessive about those reports. Is your firm instead relying on self-assessment by clients? Then, like Ferruh Mavitunah said, I'm not clear how this system can work: most clients aren't qualified to evaluate the effectiveness of a pentest, and will instead evaluate based on soft-skills. I have another business question. The most lucrative clients across the board are "house accounts" that source repeated tests from a single firm. When one of these companies sources a consultancy through your market, what prevents them from bypassing you for all future engagements? This is a race-to-the-bottom problem that plagues freelancer programming markets.

michaelt 11y ago

When they don't find something, maybe there really is nothing there. When they found something, maybe there is more there. Also, if you try to evaluate on "number of issues found" you encourage the testers to report junk issues to get their count up.

kenbaylor 11y ago

Some of this can be covered by the methodology section, where the pen testers shows the approach taken and has an overview of what they did. There's a happy medium between one-line reports 'nothing found' (which encourages questions like 'did you even try?'), to the voluminous crap produced like old vulnerability scanners. Providing the report template in advance may help set expectations. The industry is moving towards standards based pen testing. That has some pros but many cons as well. For the moment, setting expectations and having a thorough debriefing with the customer may have to do. Your question is the scientific one: How do I know what you did was good enough. Just like a patient evaluating a medical professional care, the customer isn't an expert and goes with their gut in some cases. That's why the industry also wants you to use different pen testers. I've seen many a time, when one team finds nothing, and another rips the infrastructure apart. Competency is a variable over time and so is trust. TLDR: The scientific question does not have a simple fix by any means.

tptacek 11y ago
I'm not sure I follow here. Is your firm getting access to pentest reports from the consultancies it matches to clients? That seems untenable; clients tend to be very possessive about those reports. Is your firm instead relying on self-assessment by clients? Then, like Ferruh Mavitunah said, I'm not clear how this system can work: most clients aren't qualified to evaluate the effectiveness of a pentest, and will instead evaluate based on soft-skills. I have another business question. The most lucrative clients across the board are "house accounts" that source repeated tests from a single firm. When one of these companies sources a consultancy through your market, what prevents them from bypassing you for all future engagements? This is a race-to-the-bottom problem that plagues freelancer programming markets.