5 ms·

Anyone else run into SOx problems where audit wont let DevOps staff make changes to meaningful source code. How do you get around that? Seems like an insurmount

by rb808 7y ago

Anyone else run into SOx problems where audit wont let DevOps staff make changes to meaningful source code. How do you get around that? Seems like an insurmountable barrier to proper devops.

aluminussoma 7y ago
Yes - the compliance teams needs to be on board with the goal. It's not an unsurmountable barrier unless they choose to make it one. In my current workplace, the compliance team spells out what the rules are and provides suggestions on how to achieve those rules.
chomp 7y ago
Sure. So for us IT folk, SOX boils down into 2 buckets (if you'll allow me to oversimplify a bit): 1. Separation of duties 2. Process (and well, documentation, but let's say that's a part of process.) Separation of duties boils down into needing separate people to develop, review, and deploy a piece of code. Okay, let's look at this from compliance's PoV. Compliance team just wants an easy way to show and enforce "yes, a separate individual performed each of these duties." Preferably in a way that non-technical people can understand. Easiest way to do this? RBAC. If you are a developer, you simply don't have permission to login to production. If you're ops, you don't have permission to contribute. Clean, easy enforcement, clear lines drawn in the sand, easy reporting to auditors. It also has a chilling effect on the devops mindset. Which seems to be the issue you are raising. How else can you satisfy your compliance team, follow regulations, and still practice devops? Well, instead of enforcing at the role level, enforce at the individual level. A lot of VCS software offer the ability to disallow branch merges without a secondary approver. This can be your separation between development and review. Then for implementation in production, you can automate this piece. Think of a git commit, enforced code review, and then k8s automatically deploying your change from the merge to master. This allows your dev and ops team to work on tasks together. We use Jira extensively (other tools work too), and certain important systems are tagged as SOX for a specific year. When an auditor asks "show me all changes to financially important systems", it's easy to pull up that year's changes, and then show off the 2 people signing off. We also have auto deploy (not k8s... yet) that is easy to show "hey look, this code was definitely not tampered with on its way to production." Getting your compliance team on board is difficult. It requires team work because if you are subject to SOX, you are possibly subject to other regulations that touch your processes. Like PCI-DSS. Or SOC. If your dev or ops teams can pitch a process change in a way that's easy to digest, document, and audit, then they will likely be on board.
ownagefool 7y ago
My sibling chomps reply gets it. You need to explain "peer review" or "merge requests" functionality from github, gitlab, what have you. Basically, you have: - An Author - Tests - Reviewer(s) - Audit - Robots - Protective Monitoring You should be able to explain that your average developer cannot bypass this, offering you greater assurance than the paper driven process before, and then point holes in the current process such as, people deploying manually can change the deployment after it's signed off. You can also mention that working in this way allows you to alert and triage on all assumed roles that aren't a robot account, meaning nothing that hasn't been reviewed should happen on your production cloud account, for example. Depending on the InfoSec team you might need to present your own risks, such as the CD environment, robot account setup, how they assume roles, the administrators, etc. I've been explaining this stuff to old school InfoSec teams for around 5 years now and have got pretty good at it. As long as they're actually interested in the risks, rather than just exerting control, you should be able to explain it to them.
ozim 7y ago
Compliance is mostly achieved by having ops people who have credentials to do stuff on prod server. They don't do dev, they do 2nd line support and they work with devs on new releases. Prod and acceptance changes are always 4 eyes. Though if no config changes then it is 0 eyes, because automated deploys, and ops guy just pushes button. So team is devops and those ops people are in the team having tasks on delivering releases. So whole devops is just working together not just throwing stuff over the fence. It works great with QA, security, ops. It is all about people willing to work with each other. No amount of tools will substitute that.
devonkim 7y ago
SOX section 404 does not mean ops and dev can’t write code or deploy to prod respectively. It means there should be clear roles and oversight and insight into authoritative sources. If there’s an incident in prod that requires code changes that ops knows how to fix, it doesn’t mean to wake up developers while waiting for a new build either. As long as it’s auditable, defensible to support the business, and there’s clear oversight it’s ok. For example, you may be able to temporarily assume a limited access role after recording that you have a business need to perform an action and the process can enforce another party that this taking place and provide temporary, time-sensitive credentials. Most shops that are bad or less mature at security processes don’t get this far and have code czars and prod czars.

by rb808 7y ago

Anyone else run into SOx problems where audit wont let DevOps staff make changes to meaningful source code. How do you get around that? Seems like an insurmountable barrier to proper devops.

aluminussoma 7y ago

Yes - the compliance teams needs to be on board with the goal. It's not an unsurmountable barrier unless they choose to make it one. In my current workplace, the compliance team spells out what the rules are and provides suggestions on how to achieve those rules.

chomp 7y ago

Sure. So for us IT folk, SOX boils down into 2 buckets (if you'll allow me to oversimplify a bit): 1. Separation of duties 2. Process (and well, documentation, but let's say that's a part of process.) Separation of duties boils down into needing separate people to develop, review, and deploy a piece of code. Okay, let's look at this from compliance's PoV. Compliance team just wants an easy way to show and enforce "yes, a separate individual performed each of these duties." Preferably in a way that non-technical people can understand. Easiest way to do this? RBAC. If you are a developer, you simply don't have permission to login to production. If you're ops, you don't have permission to contribute. Clean, easy enforcement, clear lines drawn in the sand, easy reporting to auditors. It also has a chilling effect on the devops mindset. Which seems to be the issue you are raising. How else can you satisfy your compliance team, follow regulations, and still practice devops? Well, instead of enforcing at the role level, enforce at the individual level. A lot of VCS software offer the ability to disallow branch merges without a secondary approver. This can be your separation between development and review. Then for implementation in production, you can automate this piece. Think of a git commit, enforced code review, and then k8s automatically deploying your change from the merge to master. This allows your dev and ops team to work on tasks together. We use Jira extensively (other tools work too), and certain important systems are tagged as SOX for a specific year. When an auditor asks "show me all changes to financially important systems", it's easy to pull up that year's changes, and then show off the 2 people signing off. We also have auto deploy (not k8s... yet) that is easy to show "hey look, this code was definitely not tampered with on its way to production." Getting your compliance team on board is difficult. It requires team work because if you are subject to SOX, you are possibly subject to other regulations that touch your processes. Like PCI-DSS. Or SOC. If your dev or ops teams can pitch a process change in a way that's easy to digest, document, and audit, then they will likely be on board.

ownagefool 7y ago

My sibling chomps reply gets it. You need to explain "peer review" or "merge requests" functionality from github, gitlab, what have you. Basically, you have: - An Author - Tests - Reviewer(s) - Audit - Robots - Protective Monitoring You should be able to explain that your average developer cannot bypass this, offering you greater assurance than the paper driven process before, and then point holes in the current process such as, people deploying manually can change the deployment after it's signed off. You can also mention that working in this way allows you to alert and triage on all assumed roles that aren't a robot account, meaning nothing that hasn't been reviewed should happen on your production cloud account, for example. Depending on the InfoSec team you might need to present your own risks, such as the CD environment, robot account setup, how they assume roles, the administrators, etc. I've been explaining this stuff to old school InfoSec teams for around 5 years now and have got pretty good at it. As long as they're actually interested in the risks, rather than just exerting control, you should be able to explain it to them.

ozim 7y ago

Compliance is mostly achieved by having ops people who have credentials to do stuff on prod server. They don't do dev, they do 2nd line support and they work with devs on new releases. Prod and acceptance changes are always 4 eyes. Though if no config changes then it is 0 eyes, because automated deploys, and ops guy just pushes button. So team is devops and those ops people are in the team having tasks on delivering releases. So whole devops is just working together not just throwing stuff over the fence. It works great with QA, security, ops. It is all about people willing to work with each other. No amount of tools will substitute that.

devonkim 7y ago

SOX section 404 does not mean ops and dev can’t write code or deploy to prod respectively. It means there should be clear roles and oversight and insight into authoritative sources. If there’s an incident in prod that requires code changes that ops knows how to fix, it doesn’t mean to wake up developers while waiting for a new build either. As long as it’s auditable, defensible to support the business, and there’s clear oversight it’s ok. For example, you may be able to temporarily assume a limited access role after recording that you have a business need to perform an action and the process can enforce another party that this taking place and provide temporary, time-sensitive credentials. Most shops that are bad or less mature at security processes don’t get this far and have code czars and prod czars.