4 ms·

But there are no auditors required for HIPAA. Only the government (HHS OCR) itself can enforce the standards.

dgellow 5mo ago
Thanks for the clarification, in that case the text is indeed really weak. Does that system work in practice, or are companies just claiming they are HIPAA compliant with close to no actual auditing mechanism?
- tptacek 5mo ago
  You get that the technical controls in SOC2 are also extremely weak, right?
  - dgellow 5mo ago
    Sure, yes. The way I understand SOC2 relies on the auditors to set the effective standard. So it really depends who audited you
    - tptacek 5mo ago
      SOC2 auditors are accountants. A SOC2 auditor verifies only that you're doing what you say what you're doing.
      - dgellow 5mo ago
        Obviously, yes
        akerl_ 5mo ago
        A SOC auditor who tells you that you can’t use an nmap scan to meet SOC2 obligations is a bad SOC auditor, because they’re attempting to enforce a constraint on you that SOC2 does not. But the far more likely thing is that a medium SOC auditor, upon being told “we do our vulnerability scanning with nmap”, would say “I haven’t heard of nmap. You should use Tenable,” and if you’re letting SOC auditor drive your engineering you’d make a mistake and accidentally think that meant you needed to change your answer for SOC2 and go buy Tenable licenses.
        dgellow 5mo ago
        The whole thread drifted way too far from a very mild push back I had regarding the claim « any automated process that can plausibly be described as instrumental in finding some kind of vulnerability is a "vulnerability scan" ». My experience is that no, SOC2 auditors won’t consider literally any automated process of that sort as compliant. Which in no way implies the auditors are forcing you to use a licensed tool or driving your engineering. I will stop that thread here, I don’t think that exchange is productive
      - kevin_nisbet 5mo ago
        And the way they verify you are doing what you say you are doing is by asking you to provide evidence, which is usually pretty easy to demonstrate that a policy was followed once or twice, a lot harder for them to pick up consistency issues or exceptions.
        randerson 5mo ago
        I've had SOC2 auditors choose a random commit from our GitHub history, then ask to see the associated Jira ticket, logs from the build and deployment, etc. Hard to reliably pass an audit if you don't know which changes they'll drill down into. They also asked for proof of system-enforced processes (e.g. GitHub branch protection rules and the setting for enforcing peer review for each change) which were basically proof of consistency.
        tptacek 5mo ago
        They do that because in the DRL process you specified a change management process involving Github and Jira. If you specified a different process (for instance, Post-It notes applied to the bathroom wall), they would randomly ask for evidence about those Post-It notes. That's what we're talking about when we say virtually any tool you can come up with will satisfy "vulnerability scans". For Cloudflare, it was nmap. I think they're right about this.
        akerl_ 5mo ago
        An example of what the parent commenter meant is more like: 1. You write a DRL that says “we do a disaster recovery test annually to ensure that we can restore a production backup”. 2. It takes you 20 tries in 2026 to do a successful restore because your find out the first 19 times that your backup is broken and incomplete. 3. You never have to mention the first 19 tries to the auditors when you prove you did do a successful DR restore.
    - _hyn3 5mo ago
      If SOC2 relies on competent auditors (and you're right, it does), than it is an ineffective standard (and it mostly is).
      - latchkey 5mo ago
        It absolutely doesn't rely on competent auditors. The AICPA that fabricated SOC2, is the same AICPA that gives licenses to the auditors. At some point, they opened it up to getting it over the internet. Indian companies open up shell businesses in Wyoming and elsewhere, get "certified", and offer rubber stamp auditing services. Few ever check if you actually have SOC2, or what auditor you used (since, by definition, they need to be "legit"). By the way, the AICPA website was recently throwing https expired cert errors. Their solution after weeks of me pointing it out on twitter, was to take down the entire website.
- giaour 5mo ago
  It's been a few years since I worked in this space, but HIPAA doesn't really work under the same kind of legal framework. Oversimplifying here, but basically HIPAA defines what constitutes personal health information, how such information may be used, and establishes monetary penalties for improper use and unauthorized disclosure. The law doesn't have any certification standard, no more than the prohibition on stealing cars does. Maybe there's some kind of third party certification system to support signing information sharing agreements ("BAAs") with other health information systems. I worked at CMS on first-party stuff so I'm not really familiar with how it works in the private sector.

Thanks for the clarification, in that case the text is indeed really weak. Does that system work in practice, or are companies just claiming they are HIPAA compliant with close to no actual auditing mechanism?

tptacek 5mo ago
You get that the technical controls in SOC2 are also extremely weak, right?
- dgellow 5mo ago
  Sure, yes. The way I understand SOC2 relies on the auditors to set the effective standard. So it really depends who audited you
  - tptacek 5mo ago
    SOC2 auditors are accountants. A SOC2 auditor verifies only that you're doing what you say what you're doing.
    - dgellow 5mo ago
      Obviously, yes
      - akerl_ 5mo ago
        A SOC auditor who tells you that you can’t use an nmap scan to meet SOC2 obligations is a bad SOC auditor, because they’re attempting to enforce a constraint on you that SOC2 does not. But the far more likely thing is that a medium SOC auditor, upon being told “we do our vulnerability scanning with nmap”, would say “I haven’t heard of nmap. You should use Tenable,” and if you’re letting SOC auditor drive your engineering you’d make a mistake and accidentally think that meant you needed to change your answer for SOC2 and go buy Tenable licenses.
        dgellow 5mo ago
        The whole thread drifted way too far from a very mild push back I had regarding the claim « any automated process that can plausibly be described as instrumental in finding some kind of vulnerability is a "vulnerability scan" ». My experience is that no, SOC2 auditors won’t consider literally any automated process of that sort as compliant. Which in no way implies the auditors are forcing you to use a licensed tool or driving your engineering. I will stop that thread here, I don’t think that exchange is productive
    - kevin_nisbet 5mo ago
      And the way they verify you are doing what you say you are doing is by asking you to provide evidence, which is usually pretty easy to demonstrate that a policy was followed once or twice, a lot harder for them to pick up consistency issues or exceptions.
      - randerson 5mo ago
        I've had SOC2 auditors choose a random commit from our GitHub history, then ask to see the associated Jira ticket, logs from the build and deployment, etc. Hard to reliably pass an audit if you don't know which changes they'll drill down into. They also asked for proof of system-enforced processes (e.g. GitHub branch protection rules and the setting for enforcing peer review for each change) which were basically proof of consistency.
        tptacek 5mo ago
        They do that because in the DRL process you specified a change management process involving Github and Jira. If you specified a different process (for instance, Post-It notes applied to the bathroom wall), they would randomly ask for evidence about those Post-It notes. That's what we're talking about when we say virtually any tool you can come up with will satisfy "vulnerability scans". For Cloudflare, it was nmap. I think they're right about this.
        akerl_ 5mo ago
        An example of what the parent commenter meant is more like: 1. You write a DRL that says “we do a disaster recovery test annually to ensure that we can restore a production backup”. 2. It takes you 20 tries in 2026 to do a successful restore because your find out the first 19 times that your backup is broken and incomplete. 3. You never have to mention the first 19 tries to the auditors when you prove you did do a successful DR restore.
  - _hyn3 5mo ago
    If SOC2 relies on competent auditors (and you're right, it does), than it is an ineffective standard (and it mostly is).
    - latchkey 5mo ago
      It absolutely doesn't rely on competent auditors. The AICPA that fabricated SOC2, is the same AICPA that gives licenses to the auditors. At some point, they opened it up to getting it over the internet. Indian companies open up shell businesses in Wyoming and elsewhere, get "certified", and offer rubber stamp auditing services. Few ever check if you actually have SOC2, or what auditor you used (since, by definition, they need to be "legit"). By the way, the AICPA website was recently throwing https expired cert errors. Their solution after weeks of me pointing it out on twitter, was to take down the entire website.
giaour 5mo ago
It's been a few years since I worked in this space, but HIPAA doesn't really work under the same kind of legal framework. Oversimplifying here, but basically HIPAA defines what constitutes personal health information, how such information may be used, and establishes monetary penalties for improper use and unauthorized disclosure. The law doesn't have any certification standard, no more than the prohibition on stealing cars does. Maybe there's some kind of third party certification system to support signing information sharing agreements ("BAAs") with other health information systems. I worked at CMS on first-party stuff so I'm not really familiar with how it works in the private sector.