16 ms·
SOC2: The screenshots will continue until security improves
- mkl95 4y agoYou can be SOC2 compliant and let your employees store passwords in plain text on Slack, Confluence, etc. and those passwords can be things like "password" and be shared with your partners. No 2FA / SSO by the way. And that's just the tip of the iceberg.
- hn_throwaway_99 4y agoThat's not really true. I mean, if any auditor knew you were doing those things, you would fail the audit (or, rather, should fail it, if your auditor knows what they're doing). It's not like it's impossible to hide those things, but that's a whole other issue.
- whatinthenote 4y agoI don't understand why people assume SOC 2 can cover every single possible scenario. Especially scenarios that have nothing to do with actual SOC 2 controls, but the result of lax security culture or bad actors. You can pass a driving test and get a driving license, but you can still drive 90 miles on the freeway and run red lights. Is it the fault of the DMV? The fault of the person who administered the driving test? Well, since people are getting away with bad things, why don't we remove the driving test and abolish the DMV. Also, who is intentionally letting your employees store password in plaintext?
- tptacek 4y agoThis is true. But if you have diverse applications and vendors, SOC2'ing that practice will be a pain in the ass, because you're going to have to document how you manage credentials, access, and onboarding/offboarding for each of them. SSO is good on the merits, but it also simplifies clearing SOC2 audits.
- toomuchtodo 4y agoThis content marketing is so good I don’t even know if I’d call it marketing: it is legit educational with a weak (in a good way) “throw some money at us” CTA. Bravo.
- tptacek 4y agoThe CTA is mostly a joke; the trick is just having fleshed out opinions and sharing them. It’s easy to replicate!
- PyWoody 4y agoWhich is my favorite kind of marketing. Others in this group for me are the Backblaze Drive Stats blogs and any guide that DigialOcean writes.
- alooPotato 4y agoAgree, this is so well written.
- meowface 4y agoIt was written by the person with (by far) the most HN karma, so they definitely know their audience.
- mrkurt 4y agoThomas is also an exceptionally good writer. He was honed in the forges of newsgroups and HN comments. He definitely knows the audience, but writing something like this is a special skill.
- toomuchtodo 4y agoSkill, for sure. But it’s also art.
- TameAntelope 4y agoOkay cool; but where is the report? Surely you didn't write all of this just to make me sign an NDA to see your report...
- d_watt 4y agoI've never met a company that gave SOC 2 out without an NDA. In fact, I'd consider it a negative indicator of maturity to not require it.
- asciimike 4y agoI've met a few where it's not explicitly required on download (e.g. GitHub has their SOC2 available for download on the enterprise admin page, but by the time you're using GHEC you've signed a few pieces of paper), but agreed that most companies aren't giving them away for free.
- d_watt 4y agoThose systems tend to stamp your account information into the PDF, as well.
- TameAntelope 4y agoYeah I was trying to make a joke that didn't land I guess.
- tptacek 4y agoYou can’t publish them; it’s an auditor restriction. You do a separate, expensive report if you want it public. Trust me, you’re not missing much.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- dave_universetf 4y ago
- xpe 4y ago> Everybody would be better off if they stopped believing what they believe about SOC2, and started believing what I believe about SOC2. Since the author is a member of the set "everybody", we have a paradox. :) More seriously, it would not be hard to adjust the language just a little bit by saying, e.g. "Most people would be better off...". Alternatively, the author could adopt a common style used in business communication where the author creates a label for the group that would benefit from the SOC2 knowledge. Perhaps call the combination of "cynics", "customers", and "true believers" the "unwise trifecta" or something. (I admit don't have a catchy term in mind yet.)
- mrkurt 4y agoThe best thing about written English is how malleable it is. Everybody doesn't have to mean everybody.
- CoastalCoder 4y agoMy current view is that ambiguous language only benefits poets and disingenuous scoundrels (politicians, sleazy marketers, etc.) That said, I'm open to being talked out of this viewpoint. Being cynical makes me unhappy.
- mrkurt 4y agoThomas is definitely a disingenuous scoundrel. In all seriousness, I laugh at absurd absolutes. If you show me a sentence with "Most people" and the same sentence with "Everybody", I'm going to smile at the second. But I don't think that means it's ambiguous. "I literally died" makes me laugh but there's no deception.
- xpe 4y agoThis is why we can't have nice things. e.g. remember when "synergy" actually meant something?
- rightbyte 4y ago
- simplecto 4y agoGreat writeup. Dude is totally right about the importance of compliance and its implications. There are other ones as well. * ISO 13485 for medical software * ISO 2700x for process, security, and change management * ISO 55000 for asset management Your eyes might glaze over, but they can really help a startup graduate from "move fast and break things, cowboy shit" to a mature and respectable engineering organization. Congrats on the SOC2 and finding a good spin on the story. It is also an important number of checkboxes you might need to get those juicy Enterprise and Government deals.
- icegreentea2 4y agoPedant mode on (this is standards after all) - ISO 13485 is medical devices in general. You'll end up tacking on something like IEC 62304 for medical software and software in medical devices.
- xpe 4y agoI’m underselling SOC2. It assures some things pentests don't: * consistent policies for who in the company gets access to what ... That's a start. Of course, having policies raises a new question -- how well are they practiced? To what degree are they: a. discoverable via internal tools? b. descriptive and understandable (the policies use terminology that maps to the business processes clearly)? c. measured? d. internalized by employees as part of the culture? e. enforced (as desired)? f. externally reported (e.g. to downstream customers)? g. reviewed when appropriate? h. adjusted or removed as needed? There are some spectacular failure modes of policies in the real word. Here are five attributes that fit together nicely into a mosaic of dysfunction: * Everyone has to take a sleep-inducing thirty minute training. * Policy compliance is "tracked" with a spreadsheet on an ad-hoc basis. * Everyone in the organization has to "check off" that they comply, starting at the bottom and working up the chain to El Jefe. * If something hits the fan and you need someone to blame, follow the paper trail and blame the people who incorrectly attested to policy compliance. * Virtually anyone could fail compliance if you haul out the microscope. This is a feature, not a bug. Now you have a convenient and official way to get rid of people you don't like for arbitrary reasons. This is stylized, yes, but not too far off the mark at some places.
- jsmith45 4y agoMany of those are real concerns. One of the main points of real audits like SOC2 type II are about validating enforcement of the controls outlined as policies. So the policy that only employees with job role Z can sign onto System Y would be checked by summarizing the login audit logs for Y, and verifying that all the names on the list. Other policies might require verification by reviewing all or some auditor selected subset of occurrences to verify. Some details of some policies cannot be fully verified. As long as that risk is known and documented, it is not necessarily a problem. SOC2 audit reports provide a high level version of reporting to downstream customers, without necessarily revealing the full details of the policy. (Sufficiently important customers could always insist on seeing the actual policy documents, if the reports don't satisfy them). But some of your remaining considerations are somewhat outside the scope of SOC2. And they can be tricky problems.
- ceejayoz 4y agoThis matches our experience well. Surprisingly intensive in some unexpected ways, and surprisingly silly and incomplete in others. I get that the Fortune 500s want it, but having now done one I don’t quite know why they value it beyond a checklist item in their own compliance.
- NovemberWhiskey 4y agoYou know how you get software engineering candidates that look awesome on paper, but can't code FizzBuzz? Same reason.
- mjr00 4y agoThis is a great way to put it. People here are bashing SOC2 because it doesn't go in-depth enough, it's just checking for the basics, it doesn't actually stop hackers from accessing insecure AWS buckets or ransomware attacks, etc etc, and they're absolutely right. But it's meant to be a minimum. It verifies that there isn't one copy of the source code on a dev's laptop. It verifies that a dev who gets fired won't be able to log into the production server and delete all data in retaliation. It verifies that an intern isn't able to completely destroy the business by accidentally deleting the production database (because you have routinely tested backups and a documented RTO/RPO, of course). Being able to demonstrate this level of minimum competency is extremely valuable when you're in the B2B world and trying to sell your product to a larger client. The paperwork is a hassle, but if your company is following best practices for development and operations, there shouldn't be much of a step change in what you're actually doing on a day-to-day basis.
- JoshTriplett 4y ago> This is the only issue we ended up having to seriously back-and-forth with our auditors about. We held the line on refusing to do background checks, and ultimately got out of it after tracking down another company our auditors had worked with, finding out what they did instead of background checks, stealing their process, and telling our auditors “do what you did for them”. This worked fine. What process did that company use instead of background checks, that you ended up doing as well?
- tptacek 4y agoThe only reason I didn’t write it out, besides it being boring, is we stole it from someone else who might rather share it themselves.
- InCityDreams 4y agoSo, two reasons?
- parkerhiggins 4y agoInvite them to share?
- paranoidrobot 4y agoI'd really like to have something to point to, when the issue of forcing-background-checks thing comes up again for our SOC2 certification. In one case, I know a company is using reference checks to comply with the "background check" requirement.
- d_watt 4y agoPart of taking on SOC 2 is also choosing whether you want your attitude to be "Let's do the minimum to get past the audit" and "Let's take this framework, and figure out where we can learn from it." The post mentions background checks. On the one hand, I understand there's a real issue with these. On the other, if my PAAS isn't ensuring repeat offender fraudsters don't have access to sensitive data, that's possibly an area of concern. Hopefully the things they took from the other mentioned company do increase due diligence in vetting employees who have access to sensitive/regulated information. Use it as a framework to actually think about BCP, DRP, etc, etc, and it won't be a total waste of time. Edit: Also adding I bring up background checks as an example of learning from vetted practices, rather than trying to attack the decisions of fly. I respect this article, especially where it's easy for people on the internet to criticize decisions, when the reality is security is a series of tradeoffs, and to function as a business means having imperfect processes.
- ceejayoz 4y ago> On the other, if my PAAS isn't ensuring repeat offender fraudsters don't have access to sensitive data, that's possibly an area of concern. I don’t know that there’s much evidence that background checks work for this, though.
- d_watt 4y agoSlightly different, but in a past life working in a field the company had extreme access to people's homes, we definitely weeded out some people who should not be given access to a strangers home using background checks. Again, they're not ideal, and there's a large social concern of a permanent record like that, but you have a duty of care if your customers are trusting you.
- tptacek 4y agoI think they're intrusive and mostly unuseful, but I can at least see the rationale behind it at the kinds of companies that hire people en masse, or the kinds of jobs that people get bonded to do.
- givemeethekeys 4y agoIf I do SOC2, then I have to spend a lot of money. If I don't, then my customers will forgive me in a few weeks and life goes on.
- d_watt 4y agoIf you're in B2B, plenty of larger companies will disqualify for not having SOC2/ISO27001. Also, it can help get you out of repeat security assessment questionnaires, so it can actually give you time back, depending on how many of those you have to field.
- jacobsenscott 4y agoThis. You'll lose more money in lost clients than SOC2 will cost you. It is only really expensive the first time you do it - after that if you just follow your own procedures the annual audits are pretty easy. And yes, being able to just reply to those security questionnaires (do you have armed guards in your data center?) with "see SOC2 report" is gold. Of course if you are in an industry were clients don't ask for soc2, don't do soc2.
- patrakov 4y ago> If you're in B2B, plenty of larger companies will disqualify for not having SOC2/ISO27001. And it's a good question whether you want such larger clients at all. At one of the previous places where I worked, we used to put deliberately bad answers (the worst that our public version of the security policy would allow, not the actual practices) in security forms in order to get rid of too-demanding clients.
- jjav 4y ago> we used to put deliberately bad answers That seems like quite a waste of time. Nobody forces you to take on a customer, so if you don't want them just say no and move on, instead of spending a lot of everyone's time to go through the motions hoping for the deal to break.
- tptacek 4y ago
- hobs 4y agoSOC2 - the thing your CTO tells you important but does everything in their power to ignore.
- spidermanjones 4y ago> but does everything in their power to ignore. aka "make the devops team deal with it even though we're paying double their salary to a 'Security & Compliance Director' who hasn't renewed any of the certs they used to get this job since 1997 and hasn't the foggiest clue how the SIEM works when the auditor shows up"
- philipwhiuk 4y agoI feel this in my veins.
- tptacek 4y agoI'm proud to say that I cost our SRE team less than a day's work from the start of the audit engagement to the end of it. If I'd thought to brag about that in the post, I would have. I cost our bizops person a lot of time though, which I feel bad about.
- orliesaurus 4y agoWait, what screenshots?
- NovemberWhiskey 4y agoIn my experience, massive reams of screenshots for evidence of process / tooling / etc. are the norm with auditors. I have in the past been asked to provide screenshots of source code even.
- jameshart 4y agoYes. Screenshots with the desktop clock included, because that way you can be sure of exactly when the screenshot was taken.
- deathanatos 4y agoLike the sibling, I've been through a SOC2 process, and there were screenshots, lots of screenshots, to try to "prove" things to the auditor. (I say "prove", as sometimes the question is malformed, and so you're trying to make the best of it as you can.)
- ceejayoz 4y agoYes. Screenshots of resolved security tickets, screenshots of 2FA being required in the Google Apps console, etc.
- anewpersonality 4y ago
- iasay 4y agoI worked for a SOC2 and ISO27001 cert outfit. Real security starts with a good security culture. Audit and compliance assures audit and compliance passes not a good security culture. While I respect that the two are independent, being audited doesn’t mean your customers aren’t at risk or you care about them.
- tptacek 4y agoDid you get any other message from this post? I'd like to think I just wrote the most SOC2-cynical "we just got SOC2" post that has ever been written, but if you've got a better example, I'd dearly love to read it.
- iasay 4y agoI would love to write it but I would probably be sued instantly. That's all I'm saying and that says a lot :)
- xorcist 4y agoI have seen that SSO requirement creep into more and more audits. It reminds me of the requirement to change passwords every month, which was everywhere but now suddenly is nowhere to be found. It's weird. It's very convenient for the user, but it's on the opposite side of the convenience/security divide. Hijack one session and re-use it for other purposes. Of course it's preferable to not let systems access credentials but one would have thought that we had yubikeys or cards and whatnot now.
- NovemberWhiskey 4y agoSSO is important because it's the best mechanism to show that employee lifecycle events are being appropriately mapped into access controls. You only activate the SSO account when the employee onboards. You deactivate the account when the employee off-boards. You can suspend the account if the employee is on an extended leave-of-absence. You can do this in one place, as single source of truth, rather than having to demonstrate that all the different identity systems in which Bob is represented have deactivated him when you have to let Bob go. Ideally this happens automatically due to an integration between your HR system and your identity platform. The alternative is that you have some kind of runbook that tells you exactly who to speak to about removing accounts from what systems (etc) which rapidly becomes unmanageable as scope of applications and number of employees increases. Even for small firms, you can run into problems when the person you need to let go is the person that knows how to run the process.
- michaelt 4y agoStill - do you use SSO for your password manager? Edit: For your work password manager.
- NovemberWhiskey 4y agoI don’t use a password manager at work. But for certain operations (e.g. privileged access stuff), we do have step-up to a second authentication factor in addition to SSO.
- 4y ago
- eranation 4y agoGoing through SOC2 as well, I have a similar experience :) > I was surprised by how important this was to our auditors. If they had one clearly articulable concern about what might go wrong with our dev process, it was that some developer on our team might “go rogue” and install malware in our hypervisor build. > ... our auditors cared a lot about unsupervised PRs hitting production. It's indeed a growing concern, and (shameless plug) the main reason we founded arnica.io (help organizations automate that process without hurting development velocity)
- sergiotapia 4y agoI will need something like this in 12 months. Can you add me to your CRM to reach out in a year please? Find me on linkedin, thank you!
- xtracto 4y agoHave a look at https://www.vanta.com/ https://www.vanta.com/ when you actually get involved in the SOC2 dance. A couple of years ago I took a startup through the SOC2 and PCI L1 compliance process "manually". At the same time Vanta was kind of starting. I decided not to use them because I (foolishly in retrospective) wanted to "learn" the SOC2 and PCI cert process by walking through it (kind of how you do derivatives, intergrals, numerical methods in school by hand so that you "grok" them). Since then, I've heard good things about Vanta from a couple of friend CTOs that adopted them. If I had to go through SOC2, PCI or ISO27001 (I did that in yet a previous startup) I would deffinitely go with them.
- ceejayoz 4y agoWe similarly had good results with Vanta (and their recommendation of a Vanta-friendly firm).
- yashap 4y agoThe blog post in general was good/informative, but I gotta say, this quote does reduce my confidence in Fly quite a lot: > To get the merit badge, we also had to document an approval process that ensured changes that hit production were reviewed by another developer. > This isn’t something we were doing prior to SOC2. We have components that are effectively teams-of-one; getting reviews prior to merging changes for those components would be a drag. But our auditors cared a lot about unsupervised PRs hitting production. > We asked peers who had done their own SOC2 and stole their answer: post-facto reviews. We do regular reviews on large components, like the Rust fly-proxy that powers our Anycast network and the Go flyd that drives Fly machines. But smaller projects like our private DNS server, and out-of-process changes like urgent bug fixes, can get merged unreviewed (by a subset of authorized developers). We run a Github bot that flags these PRs automatically and hold a weekly meeting where we review the PRs. Letting code go straight to prod without a review is just IMO a really bad practice. It sounds like they've improved significantly here, but still have plenty of gaps where people can ship to prod with no code review until it's been running in prod for up to a week. This isn't just about stopping bad actors, it's 99% about preventing all sorts of bugs/mistakes/bad ideas from hitting prod. Obviously automated tests are the main line of defence there, but code reviews are very important too. I'm kind of shocked that they want to skip out on this. I personally wouldn't want to rely on a core piece of infrastructure from a team practicing this level of cowboy coding.
- sergiotapia 4y ago>Bottom-line: SOC2 is a weak positive indicator of security maturity, in the same ballpark of significance as a penetration test report (but less significant than multiple pentest reports). Having gone through SOC2 a few times, it's more about opening doors to enterprise customers. The audits are very "grey" and subjective to your compliance auditor. Also, you get the freedom to say we're working on this and it allows you pass certain controls. One final note: watching our CISO go through this I realize it's utterly the most boring, soul crushing job I have ever seen. It's non-stop clerical paperwork that nobody will ever read but everybody demands to cover their ass. Pay your CISO's.
- whatinthenote 4y agoI think one thing that doesn't get covered enough is SOC 2's value in providing additional data for vendor security reviews. That poor CISO that have to work on SOC 2 is probably tasked with reviewing new vendors on a regular basis as well. Sure there are security white papers and pentests (which can come from dubious sources), a SOC 2 report at least serves as a fairly independent assessment of a company's security maturity. Most people don't fully understand the amount of vendors required for a company to operate (take every department you can think of and assume each will have at least 3-5 vendors per quarter).
- InfoSecErik 4y agoA tqbf blog with a link to a blogger agreeing with a Matasano blog about not liking agents? I smell a reference loop!
- mopoke 4y ago> SOC2 is the best-known infosec certification, and the only one routinely demanded by customers Maybe in the US. For the rest of the world, ISO27001 is arguably better known.
- OrvalWintermute 4y agoSOC2 is also one of the weakest. >Developed by the American Institute of CPAs I don't know when CPAs became infosec experts. >Each company designs its own controls to comply with its Trust Services Criteria. Because it depends on self-assertion, SOC2 is generally a weak organizational certification.
- tptacek 4y agoThey're not infosec experts, and don't claim to be.
- bflesch 4y agoSOC2 signals much higher maturity than ISO27001, also in Europe.
- AtNightWeCode 4y agoSOC2 is a deal breaker. It is already a pain to work for large American companies. You are forced to not be honest in the audits. The audits are complete bs by the way. Some external off shored company that wants screenshots of some specifics that they somehow found important. It is more Kafka than actual value. Miss those days, not… It is better how it works in EU. Track money and stock activity + GDPR. SOC does not prevent Enrons. I mean, even if there was a complete ledger of everything ever done in a company you could still not prevent Enrons. The fraud part should have been detected anyway. And would have in EU, I would say. Edit: D'oh! I mixed it up with SOX that is mandatory.
- NovemberWhiskey 4y ago>SOC does not prevent Enrons. SOC does not prevent Enrons; SOX does ... that's a different audit, and a different process, though. >The fraud part should have been detected anyway. And would have in EU, I would say. cough ... Wirecard ...
- AtNightWeCode 4y agoD'oh! You are right of course. I mixed it up with SOX that is mandatory.
- whatinthenote 4y ago
- CoastalCoder 4y ago> You are forced to not be honest in the audits. Not sure I agree. The question is, what are you willing to sacrifice in order to be honest.
- michaelt 4y agoWhen an audit imposes a dumb requirement that nobody will benefit from, and it's easier to change jobs than to do the dumb thing, you have the theoretical option to be honest and do it. But if it's easier to change jobs than to be honest - is the option to be honest one that will be taken by any rational person?
- mixmastamyk 4y ago> We moved everything we could to our Google SSO. I realize this is for employees and it is hard to escape their gravitational pull, but I hope no customer data is going to Google unless asked for.
- michaeldwan 4y agoThis is for employees only. We don't collect more than necessary for customers, and if we did we sure as hell wouldn't be sending it to Google.
- mixmastamyk 4y agoThanks, and right... I don't expect maliciousness on fly's part. But you'd be surprised (or not) how many goog products phone home with anything they can find. In fact it might be called a "business model" of some sort.
- ceejayoz 4y agoGoogle's SSO can't really phone home. You're either using SAML or OAuth; in either scenario, the information flow is Google --> the app you're SSOing into; name, email, and user group information. If you're SSOing into, say, AWS, Google doesn't get any access or private info out of AWS in the flow.
- tptacek 4y agoYou get that the norm in hosting providers is hosting applications themselves on Google and Amazon, right?
- mixmastamyk 4y agoI don’t trust anything with google on it, is that clear enough?
- mixmastamyk 4y ago
- mnd999 4y ago
- CoastalCoder 4y ago> The whole industry is a fucking racket. That seems plausible, but can you flesh out the argument?
- vosper 4y agoNot OP, but from my experience with SOC2 the auditing and compliance can be a complete joke. There are auditors who're entirely satisfied so long as an engineer dismisses all the dependabot alerts in Github (even if they're all dismissed with the "don't have bandwidth to fix this" option). > SOC2 is a weak positive indicator of security maturity [quoted from TFA] I'd argure it's no indicator at all.
- mnd999 4y agoIt’s an indicator that you paid your protection money to the industry audit mafia. Racket, like I said.
- xtracto 4y ago> SOC2 is the best-known infosec certification, and the only one routinely demanded by customers. In the US... and I think that SOC2 is not an "infosec" certification per se but more of a generic policies and procedures certification. When I sucessfully took a startup to SOC2 certification, we were doing PCI DSS Level 1 compliance at the same time. SOC2 was very light on InfoSec per se (it was more "bureaucracy"), whereas PCI leaned heavily into InfoSec itself. Of course that was for an American company, the rest of the world uses ISO 27001
- NovemberWhiskey 4y agoSOC2 is absolutely an infosec certification; it's just one that's premised around the idea that that the service organization should have its own mechanisms for achieving security goals and then demonstrate compliance with them. This is different from PCI-DSS which is single-domain-specific and highly prescriptive at a technical level as a result.
- chaps 4y agoThe ssh transaction/repl log thing is frightening to me. I get it, but when I've needed to get creds for a user to log into a service, one of my go-tos has been to pull shell history on all the hosts they log into since they've almost definitely typed their password prematurely at some point. So, logging all that sounds like a really fun way of making a password store!
- tptacek 4y agoThe trick here is not having any passwords that anyone ever types.
- NovemberWhiskey 4y agoRight - for example: openssh supports GSSAPI, which you can make work (e.g.) with your Active Directory or other Kerberos implementation.
- SoftTalker 4y agoAlso set your shell config so they don't save history persistently. Anything that you use history for should be set as an alias or shell function or script.
- tptacek 4y agoThe concern being raised here is that transcript-level SSH audit logs are the equivalent of permanent shell histories for everyone, and they are. But if you're giving team members a reason to ever type a password into an SSH session, you're got a bigger gap to close. We already have to do secrets management at scale, because it's a feature we provide to our customers, and so we already have a process for loading secrets into environments for host work.
- anarazel 4y agoI'd be more worried (but not terrified) that the session transcripts can teach an attacker a fair bit about how systems work, should an attacker get access to those. Of course only a small subset of attackers is going to care...
- sandGorgon 4y agothank you for writing this. this is one of the truest accounts of SOC2 ever!
- dvtrn 4y agoCareful, now. “Getting SOC2-certified” isn't the same as “doing the engineering work to get SOC2-certified”. *Do the engineering now. As early as you can. The work, and particularly its up-front costs, scale with the size of your team.* Emphasis mine. I've uttered this phrase or something like it so many times that I want to get this line fire etched onto a large, thick and sturdy block of wood and go back in time to several jobs where I and my cohorts got stuck with shoring up SOC2 tasks and smack my former execs with it. Half joking but the pain and trauma from past SOC2 audits due to exactly this is real.
- time0ut 4y agoCongrats to the team at Fly.io. I can only imagine how lucky they felt to have Thomas. I'm sure having that level of expertise and experience in house made the process relatively smooth. > You’re going to write a bunch of policies. It’s up to you how seriously you’re going to take this. I think this is one of the most important points in the article. SOC2 is going to hold you to these policies so they better make sense for your business, the nature of the product, and how your company works. Creating well written policies is an art though.
- AtlasBarfed 4y agoWas ssh really broken that you need teleport? No. Things were once capable of being automated across fleets of cattle-vms with ssh + keys now are not since teleport was shoved into my face. Oh, you type teleport to get somewhere, and then your SSO appears in a web page/browser! Because what all IT at scale needs is a human constantly manually authenticating with a web browser every 2-4 hours. I will say that at least teleport is a "solution" rather than an imposition by some powerpoint security group run by people graduating from Florida Gulf Coast College running through checklists. Meanwhile the core change-my-company-password webpage won't run in chrome because the SSL algorithms are so out of date. But somewhere... the company is in compliance. I like that the article is self-aware how shitty the security industry is, mostly a bunch of bullshit they sell the execs so that the execs can say "we did what was best practice". Meanwhile, Okta SSO got breached by some teenagers bribing people with bitcoin, and everyone just ate their public "it wasn't that bad" and moved on with no real accountability to what was pushed out in the press release. I do appreciate those evil little shits pulling down the pants of Okta. China must laugh at the security employed by US companies. I guess the real challenge isn't getting in, it's just not getting caught. The problem with security audit log software is that it implies you have centralized systems with universal access. Uhoh, your security audit logs are now a ransomware vector, and as someone else pointed out, a place you mistakenly write your password or worse. The problem with organizational behavior audits is that there's no way your org can resist being bribed, because your org is cheap, mistreats its workers, and abuses them. Teenagers with bitcoin will lulz you, background checks or no. These auditors are consultant scum. The good news is that with all things like Rational Rose, Xtreme agile, etc, they too shall pass. Especially with teenagers with bitcoins embarrassing them on the regular. Sorry for the rambling. There are no easy answers, and fuck all the compliance and "enterprisey solutions" for saying there is.
- rsync 4y ago"There are no easy answers, and fuck all the compliance and "enterprisey solutions" for saying there is." https://www.rsync.net/resources/regulatory/pci.html https://www.rsync.net/resources/regulatory/pci.html
- whatinthenote 4y ago
- rising-sky 4y ago> We held the line on refusing to do background checks, and ultimately got out of it after tracking down another company our auditors had worked with, finding out what they did instead of background checks, stealing their process, and telling our auditors “do what you did for them”. This worked fine. Wonder why the author is coy about the specifics of this when they weren't earlier on a different topic...
- tptacek 4y agohttps://news.ycombinator.com/item?id=32022022 https://news.ycombinator.com/item?id=32022022
- wdb 4y agoI was waiting for ISO 27001 :(
- collsni 4y agoSoc 2 is policy audit.
- whatinthenote 4y agoI love it when nothing is documented and every process exists in someone's head.
- pimartin 4y agoI think one of the things that you are missing out on big time with your existing SOC 2 report is the design of controls that are specific to your business and shows your customers the steps that you take to protect their data. It sounds like you are doing amazing things from a security perspective, but that those are not included in your report, which means that your customers are missing out on. As you mentioned, sure you can put it in a white paper, but who is to say that you are actually doing it? That's the point of the SOC 2 report. My hunch on this comes from the fact that you mentioned that SOC 2 comes from answering a giant spreadsheet from your auditors. It doesn't really, however that is how a lot of auditing firms get their clients to become compliant with SOC 2. They need something, and a spreadsheet is a good place to start. You're right, some businesses won't need AV, or Background checks, but in your case if it makes sense to have a control that mentions "The company only leverages memory-safe languages", you should be able to, and even explain it further in your Section 3. The key is to design controls that address the SOC 2 criteria but also are 100% reflective of how you are operating your business. It sounds to me like you got to SOC 2, but that you could get a lot more out of it. This is unfortunately super common. Source: I help SaaS companies with SOC 2 and I have been for 7 years now. Currently helping companies in unusual spaces such as crypto exchanges and making sure that all the amazing work they put in security is reflected in their report. I'd love to chat more in depth if you are interested, I am very passionate about this and I've worked with a lot of companies you probably use one way or another in your stack, or that are in a similar space as yours. Don't hesitate to reach out and good luck!
- pw 4y agoYou sound super interesting and I’m sure plenty of people would be interested in chatting (including me!), so please think about putting your email in your profile!
- TheCloudlessSky 4y agoI have personal experience working with pimartin. If you're looking for a reference, they really know what they're talking about for SOC 2. They helped me get SOC 2 at the company I co-founded, ProcedureFlow where I'm VP of Engineering. My main concern was this: we are a growing company and I didn't want to bolt on "some corporate SOC 2 thing" just to make us seem more secure. Honestly, my attitude was similar to the OP. Sales were getting blocked and delayed by lack of SOC 2 but also having to fill out security questionnaires for every customer. I found pimartin and they really showed us how SOC 2 is customizable and isn't black and white like most people think. SOC 2 is not prescriptive about how you do things. He also helped us find an auditor that understands our business and made the process very easy for us. When our prospective customers now do their IT/Security reviews, we pass with flying colors because of the changes that have been made to our organization and the big attitude shift we had about it. SOC 2 is not a burden in our company. Happy to talk more about our experience with pimartin and doing SOC 2 "right"!
- jimmygrapes 4y agoThe policy section and example policy had me cringing hard, not because of the "simplistic" tone (well, a little) but more because of the blithe ignorance of why policies are written with "whereas" and "designee" and such. It's all well and good to have a quirky fun simple summary of a policy, or a simple straightforward policy that is super vague to cover every base, but don't discount the legal jargon. I don't know if the example given is real or not, but something as simple as saying "Slack or E-mail" might make sense if those are literally the only two methods of electronic communication used, but when shit hits the fan that language won't cover it if someone sent a SMS. And let's face it: shit will hit the fan. This doesn't mean any given policy has to be unreadable (and in fact that might be detrimental) but neither can it be so jocular that it is ignored or unenforceable. If fly.io has received ISO certifications with those examples in actual usage, I'd be skeptical about who issued those certs; ISO doesn't certify directly, relying on the free market reputation of external companies/consultants to be truthful about compliance. Of course, ISO compliance isn't legally enforceable other than as a checkbox for some other procurement or investigative body, so maybe a dice roll on whether anyone checks is worth the cost. Just my two cents, take it for what it's worth in 2022 USD.
- tptacek 4y agoAs I am now an authority on the authoring of security policies I can reliably inform you: you are criticizing an information security policy for not being a data classification policy. The data classification policy spells out exactly what kinds of information are suitable for exactly which modes of transmission and storage. SOC2 demands both an information security policy and a data classification policy. And a retention policy. And an access review policy. And an incident response policy. And a BC/DR policy. And a change management policy. And a vulnerability management policy. And a vendor management policy. These are different policies. Some of them have broad audiences, like the data classification policy, which is incorporated by reference in the information security policy. Some of them have narrower audiences, like the vulnerability management policy. Hopefully that resolves your blitheness concern.
- jimmygrapes 4y ago
- funstuff007 4y ago> First, We needed a formal org chart posted where employees could find it. Isn't this bad infosec to publish this information? Also, probably bad from employee retention. Don't want the headhunters to get a hold of this.
- TheGeminon 4y agoIt’s where employees can find it, not necessarily having it public (which probably is a bad idea).
- srer 4y agoI work somewhere going through a similar process for much the same reason. And to be frank, I'm wondering if how it ends is in me looking for a new job. I know there's good reasons for the business to go through this process, and maybe it is really important for the business future but as an individual it sucks, my main motivator of achieving good outcomes for users, has been derailed by a large amount of procedure instigated under the belief that some auditor I've never met will be impressed. Now with all this new procedure piled on, all of which is very new and thus immature and untried, I feel my energy to innovate, build new things and just generally drive change sapping away. So the cost of your company isn't just the direct hours and the auditors fees! There's also a harder to quantify cost, as everything and everyone bends to change every team to work in the way some external entity thinks it should. This loss of autonomy and the loss of spirit is I think potentially far far more expensive than the direct costs. Of course, maybe we're just going about the whole process poorly. I like to think if you were to do it well maybe it's not so bad. But, to do that I think you'd need abnormally talented staff involved, who are well versed in the topic, enthusiastic and empathetic. Most people involved in these sorts of processes in my experience aren't like that, and I can't say I blame them.
- jkingsman 4y agoFor what it's worth, as someone who helped quarterback the SOC2 process, it's a place that's ripe for personal innovation. We had automated scripts do our quarterly screenshots, learned and made full use of AWS Config to check and enforce compliance, worked hard to automate patching, and started tracking all audit tasks using Jira Service Desk. It ended up we used about two engineer-days a quarter on audit tasks. There's no escaping the annual review, but if you spend some time to streamline yourself, it goes pretty smoothly we found.
- smt88 4y agoThere are tons of people automating it already (Vanta, Secureframe, etc.)
- nickjj 4y ago> For what it's worth, as someone who helped quarterback the SOC2 process, it's a place that's ripe for personal innovation I wonder if the parent comment was talking more about things like instead of committing code on their own, now it becomes a process that could be delayed only for the sake of compliance. Something that was purely self contained and blocked by nothing is now blocked by at least someone else reviewing what you've done. But it's not just reviewing the code. To get something reviewable also means writing a detailed description of the work, documenting any and all processes involved and then maybe answering follow up questions if they come up. If you happen to change something then it needs to be re-tested, etc.. This is how something goes from being done and self tested in 2 hours to taking 3+ days. It can be a motivation killer at a personal level and also delay goals, features and everything basically at the company level. I'm all for documenting work and following best practices like code reviews, well written tickets, etc.. but there are certain things where sometimes having all of that isn't an option because you don't have a team built around what you're doing. For example I'm in a platform / SRE / "devops engineer" type of role at a place and I end up shipping quite a lot of infrastructure code to production without review. I use my best judgment here. If it's something I can test in a test environment and I have high confidence I'll do it. If there's high stakes to the change I'll ask someone on the dev team to screenshare with me while I explain everything since most of them aren't deep in the woods with Terraform, Kubernetes, etc., but a 2nd set of eyes is still very helpful and valuable. The problem there is not every change needs that level of review. It becomes extremely wasteful. I mean, if I add a Kubernetes annotation to a deployment, that's something I can do on my own in 10 seconds and push it to production but if it needs a review then it needs a jira ticket, documenting why / etc., making a PR, doing the work, finding a reviewer, going over what a deployment is in Kubernetes to a developer without Kubernetes knowledge and then going over what an annotation is, then finish things up by explaining what Kustomize does and how it works along with Argo CD because the state of the pod after Argo CD deploys it is the verification step to make sure it works. That has to be done over a screenshare because an application developer won't have these tools installed or know how to use them. This ends up taking potentially longer than 1 day all-in. Maybe 15 minutes to do the ticket + PR but it could take 4-5 hours before someone is ready to review it, then another 30min for the screenshare, suddenly it's end of day. But for SOC 2 compliance I'm pretty sure you also need to deploy to a pre-prod environment as part of the workflow so that means the next day you spin up a temporary Kubernetes cluster in a proper test environment on the cloud, deploy a test application to it, applying your PR, verify it, kill your test cluster and then deploy it to prod which also involves someone else to approve the PR being merged to your infra's main branch. Now imagine this type of scenario coming up at least 5 or 10 times a week. You would be in a constant state of being blocked and delayed. Without SOC 2 compliance I would have just pushed the annotation to production (main branch) directly in 10 seconds and moved onto the next thing.
- joshgachnang 4y agoAbsolutely wonderful write up. And filed away for when I need to point my cofounders to something on why we should put it off as long as possible. It's crazy how different the approach to SOC2 here is, having someone in the company running it vs having a "security/compliance" consultant taking point. We wound up with basically everything in "What We Didn’t Let SOC2 Make Us Do", despite protests by most of the engineering team and lots of documentation. It was very much a case of badly aligned incentives. "Person who sells services suggests buying more services from him". One of the very annoying additional requirements was all the additional process around Jira (which is already annoying enough to use). Such as "Engineering cannot make this kind of card, it must be made by a product manager" and "Engineering cannot move the card into the final state, that must also be a product manager". Which might be fine for a larger company, but we had one PM and 3.5 devs.
- paulmendoza 4y agoGet Vanta. It really simplifies the process.
- tptacek 4y agoIn what way? When you used Vanta, did you install Vanta software on your fleet? What engineering tasks did Vanta get you to undertake?
- snowwrestler 4y ago> To get the merit badge, we also had to document an approval process that ensured changes that hit production were reviewed by another developer. > This isn’t something we were doing prior to SOC2. We have components that are effectively teams-of-one; getting reviews prior to merging changes for those components would be a drag. This mindset kind of blows my mind. As a customer, it makes me want to ask all my vendors for SOC2 now. As a comparison, we have a small in-house team that develops some nonprofit websites; no PR goes to production without being reviewed by another developer. I’m honestly flabbergasted at such a seemingly blasé attitude about insider threats. It doesn’t have to be a top-secret Chinese operation to subvert your company. It can be as simple as a developer you know well who is nursing some simmering resentment over a perceived slight.
- tptacek 4y agoNone of the companies you trust are secure from insider threats by dint of PR review --- a thing we do, but override for urgent changes in oddball services sometimes, like most companies I've worked with over the past 10 years.
- snowwrestler 4y agoI’m confused, in the post it seemed like you said you don’t do PR reviews, but here it seems like you’re saying you do, except for emergencies.
- tptacek 4y agoRead more carefully. Or, stay flabbergasted! Flabbergasting can be very satisfying.
- snowwrestler 4y agoI feel like you’re being unnecessarily confrontational. You wrote “changes that hit production were reviewed by another developer. This isn’t something we were doing…” I cited PR as an example of a change and you’re acting like I’m trying to argue that PR reviews alone stop insider threats. (I’m not.) There are lots of way to change production. To me it looks like you said effectively one person could do so without being checked by another person. To me that sounds like a hole for problems to drive through. And that SOC2 was the impetus for you to tighten it. If I’m wrong, you can take it as feedback on your writing, I guess. You know what you wanted to write; I’m telling you what I read. Or you can just treat me like I’m an idiot if you want. I don’t think I’m an idiot, and I’m not the only person you’re being combative with.
- paulmendoza 4y agoFirst, my company uses a tool called Vanta. Really simplifies the SOC2 process for a small to medium sized business. They have managed to automate large portions of the process. I think I created only 5 or so screenshots for the whole SOC2 audit. Second, the process is making your business more secure. There are so many things that people skip without the reminders of the SOC2 audit. Also, if someone has a SOC2 it tells me they are doing the Standard stuff I expect.
- tptacek 4y agoEveryone uses Vanta, at least among YC companies (Vanta is a YC company). There are things I like about Vanta, and you can't argue with the track record. I have one big concern with it, which is that in working with other companies that used Vanta, my experience was that it pulled those teams towards an expansive take on what SOC2 is, and induced them to do extra engineering work. Alarm bells should go off in your head when you do engineering work specifically for SOC2, because SOC2 has weird ideas of how serverside engineering should work, and you probably don't want to adopt them. I know Vanta (and the other tools like them) are customizable, and you don't have to do everything they suggest. If you know that going in, and you're careful about minimizing your Type I, they work great and you can't argue with the track record. The thing with me is: you really can't flunk a Type I. If you're serious about getting certified, you will get certified. So you should be much more worried about SOC2 dragging you into extra work or bad engineering decisions than you should be about whether you'll succeed and getting a Type I. So anything that creates the perception that "extra stuff" has to get done for SOC2 is something I'm automatically wary of.
- a13n 4y agoAgreed. We did a process on Drata vs Vanta and ended up choosing Drata, but they're similar in terms of product/pricing. Drata has saved us tons of time.
- fomine3 4y agoLooks reasonable compliance than I expected. Good work.
- 300 4y ago>SOC2 is the best-known infosec certification LOL. I've stop reading right there.
- dachryn 4y agobut it is? Which one is the best-known in your opinion then?
- londons_explore 4y ago> SOC2 reports: it's viral, like the GPL. The DRL strongly suggests you have a policy to “collect SOC2 reports from all your vendors”. This kind of thing should be frowned upon. Especially if it involves things one has to pay for. Requiring someone to get certified and then requiring everyone they speak to is also certified is just a racket for the certification provider. Even if elements of the certification are good, the viral nature is not. A certification should survive based on its merits, not based on who manages to use the most strongarm tactics to ensure an industry requires it of all participants.
- gorgoiler 4y agoTwo factor authentication is often implemented by SMS. For example, Microsoft’s single sign on is protected this way. Bring-your-own-device is often the only way employees can receive text messages. Obviously most people have phones but there are legitimate cases where employees don’t or can’t use their device. For example, they may have lost or permanently broken their phone and are saving up for a replacement. They might be an immigrant worker with a phone contract in their origin country that limits their ability to receive texts abroad. They might be a member of a union that would no longer represent them because they waived their employment rights by providing their own equipment. Is the point of SOC2 to show that you tried to get BYOD employees to do 2FA, or do you have to show that you implemented it without requiring personal devices? How do you do it without BYOD? Have all the SOC2 / ISO27001 certified businesses bitten the bullet and bought phones for all their employees? I would concede that phones and SMS are a crappy way of doing this. App-based 2FA is a better tool but still relies on your employees having two devices. I miss the days of RSA tokens. (This article is a good read, in general. I’ve lost count of the number of times someone has arbitrarily said “you can’t do that because we will fail our audit” without actually being able to give any details. This article is the first time I’m reading “you should do this” as opposed to “you shouldn’t do that”.)
- pid-1 4y agoMicrosoft (Azure AD to be more precise) single sign on can be configured in a few ways and SMS can be disabled. If you really care about users not needing BYOD, you can restrict 2FA to hardware keys. That said I think the overall sentiment of your post still stands, as most orgs just push the device issue to the user (either they need a phone of SMS, push notifications or OTP).
- q-big 4y ago> Obviously most people have phones but there are legitimate cases where employees don’t or can’t use their device. There do exist quite some privacy-conscious people who for this reason don't have a mobile phone and very actively don't want one.
- technion 4y ago
- scoot 4y ago> finding out what they did instead of background checks, stealing their process, and telling our auditors “do what you did for them”. And what might that be?
- cbare 4y agoEvery article about SOC2 should be illustrated with a Hieronymus Bosch-like hellscape.
- tptacek 4y agoI cheated and stole the art for our podcast as a placeholder. We have something better coming for this post. :)
- yencabulator 4y ago> try to find simple access control models Can I finally make a fly.io API key that isn't all-powerful? All I see is "Create" and "Revoke" buttons. There's not much access control visible, to this user. I'd love to delegate ability to scale a specific app up/down to some automation, but currently all it seems I can do is let that automation become me. (For the peanut gallery: https://fly.io/blog/api-tokens-a-tedious-survey/ https://fly.io/blog/api-tokens-a-tedious-survey/)
- tptacek 4y agoIt's coming, very soon! We did Fly Machines first. We're doing Macaroons. The backend code is mostly there, but the API integration is a work in progress. This is at the top of our punch list right now.