4 ms·

several levels of auditors were effectively asking us to downgrade to comply with their policy without even understanding the difference The auditors' policy?

several levels of auditors were effectively asking us to downgrade to comply with their policy without even understanding the difference The auditors' policy? Are you sure? An auditor's job is to check if you're doing what you say you should be doing. If you're arguing with an auditor then you're essentially arguing with your own organisation without any hope winning the argument.

ath0 6y ago
Not really a fair assessment. An auditor's job is often to check if you're doing what an external standard says you should be doing (SOC 2 => AICPA trust principles; FedRAMP => NIST 800-53, etc.). Unfortunately, these external standards may be written vaguely and while you may have policies that define X as Y, the auditor doesn't have to accept your answers. For example, when PCI requirement 5 says "Deploy anti-virus software on all systems commonly affected by malicious software (particularly personal computers and servers).", your policy may say "antivirus is not required inside containers that run on platforms like GKE, as these are not commonly affected by malicious software." It's very likely you'll have a discussion about your interpretation of that requirement.
- deleted 6y ago
  [deleted]
- edoloughlin 6y ago
  So every team gets to argue with the auditors? Seems like the people writing the policy should do that so there's one argument instead of n.
  - ath0 6y ago
    Sounds like you've mostly worked in perfectly-sized and structured corporations, where the auditors and policy-writers were perfectly connected to changing product, business and technical needs; well-staffed with policy writers and architectural governance committees who have the time, skill and background to have regular, even-handed tradeoff conversations when these issues occur, and where the engineering teams are prepped and able to engage in those conversations well. Are they hiring?
    - edoloughlin 6y ago
      Unfortunately, they have an audited policy of not hiring sarcastic people.
- viraptor 6y ago
  PCI also suggests a hardened system image, for example CIS and consistency checking like Aida. I'm getting tired of explaining that CIS (and other) "hardened" images just flip a few options and install lots of crap that can actually increase risk. E.g. You don't need cron? Haha, it's scored in the CIS benchmark, now you're running it. I don't mean to just single out CIS as bad, but recently I learned that Ubuntu CIS docker images contain Aida, cron, and sysctl configuration. Yes, you pay for that. I'll be making fun of that for a long time.
- hobs 6y ago
  No, they check if you're doing what an external standard that you purport to follow is being followed. You dont have to claim that you follow some standard that actually makes you less secure, you are just not going to be able to sell to clients who are also sheep in this manner (read: everyone.)
- dimitrios1 6y ago
  Actually, you are both half right half wrong. If GP was talking about internal audit, they are correct. If you are talking about external audit, then you are correct. If you are lumping both audit functions into one, you are both wrong. Big difference between the two.
- lazyant 6y ago
  Not responding to this, just using the example. It can be easily argued that adding AV software to (some) servers increase their surface attack and reduce their security (leaving alone performance and other AV issues).
the8472 6y ago
Multi-national corporation. The auditing department and the ones writing the policy aren't even on the same continent. So it is all internal and "our own policy" in a sense, but so far removed that I might as well be talking to an external bureaucracy. By coincidence we later learned about the contact person within our own sub-org who would be responsible for forwarding change requests on those policies. The auditors themselves didn't see that as their responsibility, even when you explicitly asked them to do so.
- PeterisP 6y ago
  It's definitely not the responsibility of auditors even if you explicitly ask them to do so. In essence, the auditors are there to evaluate whether certain assertions are true. E.g. the company is asserting that they are doing X because they have a policy that X must be done. If it turns out that the internal communication within the company is screwed up and some teams are following the policy and some are not, then the claim is false and the core job of the auditors is to detect and note this. Detecting such discrepancies the main reason for why such an audit (often external,, and if internal, then mostly independent, not reporting to the audited departments) is requested. It's not their job to fix this or decide who's in the wrong - the initial claims are found to be false, and that's not okay no matter if the policy should be changed or will be changed. It does not matter how (and if) the company negotiates changes to the policy, it does not excuse the misleading claims of "X is being done" if actually the company is doing Y instead of following the stated policy. When the management has fixed this (or claims to have fixed this) one way or another, then the auditors should re-evaluate whether the claims are true now.
  - the8472 6y ago
    Compliance serves a purpose and is not an end unto itself. So if you only enforce the rules even when they achieve the opposite of the underlying goal then you are goodhart's law personified. So even assuming for the moment that it's not the auditor's job to evaluate the rules themselves then at the very least they should escalate all the way back to those who write the rules if an inconsistency between rule and underlying goal is found. There's no point of having a separate channel for such communication because a particular issue for which it is relevant cannot be resolved (without missing the underlying goal) until the rule is clarified.

ath0 6y ago

Not really a fair assessment. An auditor's job is often to check if you're doing what an external standard says you should be doing (SOC 2 => AICPA trust principles; FedRAMP => NIST 800-53, etc.). Unfortunately, these external standards may be written vaguely and while you may have policies that define X as Y, the auditor doesn't have to accept your answers. For example, when PCI requirement 5 says "Deploy anti-virus software on all systems commonly affected by malicious software (particularly personal computers and servers).", your policy may say "antivirus is not required inside containers that run on platforms like GKE, as these are not commonly affected by malicious software." It's very likely you'll have a discussion about your interpretation of that requirement.

deleted 6y ago
[deleted]
edoloughlin 6y ago
So every team gets to argue with the auditors? Seems like the people writing the policy should do that so there's one argument instead of n.
- ath0 6y ago
  Sounds like you've mostly worked in perfectly-sized and structured corporations, where the auditors and policy-writers were perfectly connected to changing product, business and technical needs; well-staffed with policy writers and architectural governance committees who have the time, skill and background to have regular, even-handed tradeoff conversations when these issues occur, and where the engineering teams are prepped and able to engage in those conversations well. Are they hiring?
  - edoloughlin 6y ago
    Unfortunately, they have an audited policy of not hiring sarcastic people.
viraptor 6y ago
PCI also suggests a hardened system image, for example CIS and consistency checking like Aida. I'm getting tired of explaining that CIS (and other) "hardened" images just flip a few options and install lots of crap that can actually increase risk. E.g. You don't need cron? Haha, it's scored in the CIS benchmark, now you're running it. I don't mean to just single out CIS as bad, but recently I learned that Ubuntu CIS docker images contain Aida, cron, and sysctl configuration. Yes, you pay for that. I'll be making fun of that for a long time.
hobs 6y ago
No, they check if you're doing what an external standard that you purport to follow is being followed. You dont have to claim that you follow some standard that actually makes you less secure, you are just not going to be able to sell to clients who are also sheep in this manner (read: everyone.)
dimitrios1 6y ago
Actually, you are both half right half wrong. If GP was talking about internal audit, they are correct. If you are talking about external audit, then you are correct. If you are lumping both audit functions into one, you are both wrong. Big difference between the two.
lazyant 6y ago
Not responding to this, just using the example. It can be easily argued that adding AV software to (some) servers increase their surface attack and reduce their security (leaving alone performance and other AV issues).

the8472 6y ago

Multi-national corporation. The auditing department and the ones writing the policy aren't even on the same continent. So it is all internal and "our own policy" in a sense, but so far removed that I might as well be talking to an external bureaucracy. By coincidence we later learned about the contact person within our own sub-org who would be responsible for forwarding change requests on those policies. The auditors themselves didn't see that as their responsibility, even when you explicitly asked them to do so.

PeterisP 6y ago
It's definitely not the responsibility of auditors even if you explicitly ask them to do so. In essence, the auditors are there to evaluate whether certain assertions are true. E.g. the company is asserting that they are doing X because they have a policy that X must be done. If it turns out that the internal communication within the company is screwed up and some teams are following the policy and some are not, then the claim is false and the core job of the auditors is to detect and note this. Detecting such discrepancies the main reason for why such an audit (often external,, and if internal, then mostly independent, not reporting to the audited departments) is requested. It's not their job to fix this or decide who's in the wrong - the initial claims are found to be false, and that's not okay no matter if the policy should be changed or will be changed. It does not matter how (and if) the company negotiates changes to the policy, it does not excuse the misleading claims of "X is being done" if actually the company is doing Y instead of following the stated policy. When the management has fixed this (or claims to have fixed this) one way or another, then the auditors should re-evaluate whether the claims are true now.
- the8472 6y ago
  Compliance serves a purpose and is not an end unto itself. So if you only enforce the rules even when they achieve the opposite of the underlying goal then you are goodhart's law personified. So even assuming for the moment that it's not the auditor's job to evaluate the rules themselves then at the very least they should escalate all the way back to those who write the rules if an inconsistency between rule and underlying goal is found. There's no point of having a separate channel for such communication because a particular issue for which it is relevant cannot be resolved (without missing the underlying goal) until the rule is clarified.