6 ms·

Even more damning, looks like they got a SOC-2 Type II via Vanta in July of 2021. I just had to go through SOC-2 Type II at two startups in the last few years s

Even more damning, looks like they got a SOC-2 Type II via Vanta in July of 2021. I just had to go through SOC-2 Type II at two startups in the last few years so I'm very familiar with the requirements. How the hell did they pass multiple penetration tests and a third party audit with such blatent security issues that an automated scan would detect? If anything, I think this says something much more damning about Vanta and/or about whatever penetration testing firm and/or auditor Vanta referred them to. It's on Vanta to make good referrals, it's on the auditor to validate that the pen tester is reputable and that all the requirements are met, and it's on the pen tester to actually check these things. All are at fault. They never should have come close to passing. Details on the audit seem to be sparse so I can't call out the auditor and/or penetration tester but yeah this is a real bad look for the whole industry. This really muddies the water for every security-minded startup that has taken the trouble to go through SOC-2 or ISO 27001. Very disappointing. I guess billion dollar valuation really can buy you a rubber stamp.

makestuff 4y ago
Might be like the credit rating agencies in 2008. If you want to keep your business with venture backed startups, you better play by their rules.
singron 4y ago
SOC2 is a joke and pen testers usually do a terrible job. Almost all of SOC2 has nothing to do with whether you built secure software. It seems like pen testers usually just run standard scanners, report all the false positives, and then call the job done. If your software team has been writing millions of lines of code for years, how can you expect someone to find actual security issues in something they have never seen before in just days or weeks?
- sam0x17 4y ago
  It really wasn't a joke when we went through the process at Arist (YC S20) I can tell you. Pen tester found some really meaningful privilege escalation stuff we were able to fix before launching our new platform that was 100% specific to our system and never would have come up in an automated scanner and went through our entire threat model with great detail. Additionally, the automated scanners at least catch CSRF stuff. Somehow they even missed that here
  - Aaronstotle 4y ago
    Unfortunately there is a large quality range for pen-tests. There are some that are meant to check the box and others that actually find vulnerabilities.
    - sam0x17 4y ago
      Right but there is a ring of accountability -- the compliance vendor (Vanta) should only refer startups to good auditors and pen testers. The auditor should verify that the pen tester is reputable, and the pen tester should check for basic things like this that would show up in an automated scan. Every link in the chain seems to have failed here, but Vanta is the one that did the initial referring presumably.
  - jabroni_salad 4y ago
    I mainly work with financial institutions and most of them have had their first real hands-on pentest in just the past year despite receiving audits (nessus scans) quarterly to yearly by various entities for decades (FDIC, state department of banking, cyber insurance, and the ATM vendor). There's a lot of stuff that comes up in a properly driven pentest that nessus just doesn't do. Never had a nessus operator pass the hash into domain admin or pretend to be a Spectrum technician and hide in the closet until after hours.
  - singron 4y ago
    Who did your pentest? Was that your first one? I'm guessing our codebase is older and bigger if you were S20, and we've had several pentests at this point, so maybe they are scraping the bottom of the barrel.
    - sam0x17 4y ago
      we used Federacy. One big advantage we had though is our app is 100% serverless via Ruby on Jets. So it's a bunch of rails code that runs in lambda, makes the security footprint at lot easier to deal with. No EC2 instances for example.
leetrout 4y ago
One point of clarification: Vanta doesn't do the audits. They connect you with auditors. There is no reason to think you can't buy your way into a SOC 2 cert with minimal effort.
- sam0x17 4y ago
  Right, but Vanta is known for having a relationship with the auditors they use that is cozier than it is between SecureFrame or Drata and their auditors. This combined with the fact that they call out Vanta in their own blog post makes the whole thing even sketchier... almost like Vanta hooked them up with a rubber stamp AND almost like they don't even realize themselves that Vanta doesn't do the audit, which is CRAZY
lvh 4y ago
1) SOC2 is mostly just a rubber stamp and not a guarantee you're doing much of anything particularly well. 2) Reed (CEO) already responded to these claims: https://news.ycombinator.com/item?id=33164969 https://news.ycombinator.com/item?id=33164969 With (2) in mind: no, I don't think it's damning at all. Sure looks that way if you fail to analyze half the claims and take the remaining half out of context :)

makestuff 4y ago

Might be like the credit rating agencies in 2008. If you want to keep your business with venture backed startups, you better play by their rules.

singron 4y ago

SOC2 is a joke and pen testers usually do a terrible job. Almost all of SOC2 has nothing to do with whether you built secure software. It seems like pen testers usually just run standard scanners, report all the false positives, and then call the job done. If your software team has been writing millions of lines of code for years, how can you expect someone to find actual security issues in something they have never seen before in just days or weeks?

sam0x17 4y ago
It really wasn't a joke when we went through the process at Arist (YC S20) I can tell you. Pen tester found some really meaningful privilege escalation stuff we were able to fix before launching our new platform that was 100% specific to our system and never would have come up in an automated scanner and went through our entire threat model with great detail. Additionally, the automated scanners at least catch CSRF stuff. Somehow they even missed that here
- Aaronstotle 4y ago
  Unfortunately there is a large quality range for pen-tests. There are some that are meant to check the box and others that actually find vulnerabilities.
  - sam0x17 4y ago
    Right but there is a ring of accountability -- the compliance vendor (Vanta) should only refer startups to good auditors and pen testers. The auditor should verify that the pen tester is reputable, and the pen tester should check for basic things like this that would show up in an automated scan. Every link in the chain seems to have failed here, but Vanta is the one that did the initial referring presumably.
- jabroni_salad 4y ago
  I mainly work with financial institutions and most of them have had their first real hands-on pentest in just the past year despite receiving audits (nessus scans) quarterly to yearly by various entities for decades (FDIC, state department of banking, cyber insurance, and the ATM vendor). There's a lot of stuff that comes up in a properly driven pentest that nessus just doesn't do. Never had a nessus operator pass the hash into domain admin or pretend to be a Spectrum technician and hide in the closet until after hours.
- singron 4y ago
  Who did your pentest? Was that your first one? I'm guessing our codebase is older and bigger if you were S20, and we've had several pentests at this point, so maybe they are scraping the bottom of the barrel.
  - sam0x17 4y ago
    we used Federacy. One big advantage we had though is our app is 100% serverless via Ruby on Jets. So it's a bunch of rails code that runs in lambda, makes the security footprint at lot easier to deal with. No EC2 instances for example.

leetrout 4y ago

One point of clarification: Vanta doesn't do the audits. They connect you with auditors. There is no reason to think you can't buy your way into a SOC 2 cert with minimal effort.

sam0x17 4y ago
Right, but Vanta is known for having a relationship with the auditors they use that is cozier than it is between SecureFrame or Drata and their auditors. This combined with the fact that they call out Vanta in their own blog post makes the whole thing even sketchier... almost like Vanta hooked them up with a rubber stamp AND almost like they don't even realize themselves that Vanta doesn't do the audit, which is CRAZY

lvh 4y ago

1) SOC2 is mostly just a rubber stamp and not a guarantee you're doing much of anything particularly well. 2) Reed (CEO) already responded to these claims: https://news.ycombinator.com/item?id=33164969 https://news.ycombinator.com/item?id=33164969 With (2) in mind: no, I don't think it's damning at all. Sure looks that way if you fail to analyze half the claims and take the remaining half out of context :)