3 ms·

SOC2 is a joke and pen testers usually do a terrible job. Almost all of SOC2 has nothing to do with whether you built secure software. It seems like pen testers

SOC2 is a joke and pen testers usually do a terrible job. Almost all of SOC2 has nothing to do with whether you built secure software. It seems like pen testers usually just run standard scanners, report all the false positives, and then call the job done. If your software team has been writing millions of lines of code for years, how can you expect someone to find actual security issues in something they have never seen before in just days or weeks?

sam0x17 4y ago
It really wasn't a joke when we went through the process at Arist (YC S20) I can tell you. Pen tester found some really meaningful privilege escalation stuff we were able to fix before launching our new platform that was 100% specific to our system and never would have come up in an automated scanner and went through our entire threat model with great detail. Additionally, the automated scanners at least catch CSRF stuff. Somehow they even missed that here
- Aaronstotle 4y ago
  Unfortunately there is a large quality range for pen-tests. There are some that are meant to check the box and others that actually find vulnerabilities.
  - sam0x17 4y ago
    Right but there is a ring of accountability -- the compliance vendor (Vanta) should only refer startups to good auditors and pen testers. The auditor should verify that the pen tester is reputable, and the pen tester should check for basic things like this that would show up in an automated scan. Every link in the chain seems to have failed here, but Vanta is the one that did the initial referring presumably.
- jabroni_salad 4y ago
  I mainly work with financial institutions and most of them have had their first real hands-on pentest in just the past year despite receiving audits (nessus scans) quarterly to yearly by various entities for decades (FDIC, state department of banking, cyber insurance, and the ATM vendor). There's a lot of stuff that comes up in a properly driven pentest that nessus just doesn't do. Never had a nessus operator pass the hash into domain admin or pretend to be a Spectrum technician and hide in the closet until after hours.
- singron 4y ago
  Who did your pentest? Was that your first one? I'm guessing our codebase is older and bigger if you were S20, and we've had several pentests at this point, so maybe they are scraping the bottom of the barrel.
  - sam0x17 4y ago
    we used Federacy. One big advantage we had though is our app is 100% serverless via Ruby on Jets. So it's a bunch of rails code that runs in lambda, makes the security footprint at lot easier to deal with. No EC2 instances for example.

sam0x17 4y ago

It really wasn't a joke when we went through the process at Arist (YC S20) I can tell you. Pen tester found some really meaningful privilege escalation stuff we were able to fix before launching our new platform that was 100% specific to our system and never would have come up in an automated scanner and went through our entire threat model with great detail. Additionally, the automated scanners at least catch CSRF stuff. Somehow they even missed that here

Aaronstotle 4y ago
Unfortunately there is a large quality range for pen-tests. There are some that are meant to check the box and others that actually find vulnerabilities.
- sam0x17 4y ago
  Right but there is a ring of accountability -- the compliance vendor (Vanta) should only refer startups to good auditors and pen testers. The auditor should verify that the pen tester is reputable, and the pen tester should check for basic things like this that would show up in an automated scan. Every link in the chain seems to have failed here, but Vanta is the one that did the initial referring presumably.
jabroni_salad 4y ago
I mainly work with financial institutions and most of them have had their first real hands-on pentest in just the past year despite receiving audits (nessus scans) quarterly to yearly by various entities for decades (FDIC, state department of banking, cyber insurance, and the ATM vendor). There's a lot of stuff that comes up in a properly driven pentest that nessus just doesn't do. Never had a nessus operator pass the hash into domain admin or pretend to be a Spectrum technician and hide in the closet until after hours.
singron 4y ago
Who did your pentest? Was that your first one? I'm guessing our codebase is older and bigger if you were S20, and we've had several pentests at this point, so maybe they are scraping the bottom of the barrel.
- sam0x17 4y ago
  we used Federacy. One big advantage we had though is our app is 100% serverless via Ruby on Jets. So it's a bunch of rails code that runs in lambda, makes the security footprint at lot easier to deal with. No EC2 instances for example.