4 ms·

I think you'll find that at larger companies, or companies with a need for taking their security posture seriously, that is the norm. Even before we grew, only

by _jal 8y ago

I think you'll find that at larger companies, or companies with a need for taking their security posture seriously, that is the norm. Even before we grew, only a couple engineers had prod access. You debug it via tooling, instrumentation and logs. I realize if you're accustomed to sudoing on prod when troubleshooting, this sounds crazy. Trust me, it works fine; better, in fact. Far fewer weird things happen in well-controlled environments.

justinclift 8y ago
Yeah, I've occasionally wondered how (if) the DevOps approach is implemented at places which have to be SOX compliant. https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act One of the technical things it does is segregate (on purpose) developers from any kind of production access. This is because (from historical experience) a "bad apple" developer can do amazingly fraudulent things, and have a more than reasonable chance of covering it up. o_O
- crankylinuxuser 8y ago
  I work in an org that has strong partitions between devs and sysads. Dev can make and test code. They have no access to prod. Sysads can configure and execute what devs make. They have read-only access to the source repos. Problems with this are as follows (real, not imagined) 1. AWS cloudformation scripts - who makes them? If dev does, sysads can't change. 2. Does dev have the security mindset to maintain configurations in IaaS things like Cloudformation? Who reviews things like NACLs, Security Groups, VPCs, and the like? 3. Scripts - how big or what impact does a script need to be written by sysad or dev? 4. Oncall - normally are sysads job, but when you implement strong gates between dev/sysad, you need oncall devs.
  - justinclift 8y ago
    Thanks, that all falls in line with roughly the kind of problems I'd expect. :)
  - deleted 8y ago
    [deleted]
- hessproject 8y ago
  I work on a SOX compliant project, it works pretty much as described. As a developer I have no access to prod, relying on other teams to make system changes and creating a lengthy mandated paper trail for the SOX audit team to look over. Not to say there aren't headaches with the approach, but thankfully at a big enough organization it's become a relatively smooth process.
- caspar 8y ago
  Engineering manager (& previously a developer) at a ~2k developer SOX-compliant company chiming in here. We have a platform as a service team which maintains our PaaS infrastructure (think an internal version of Heroku), and they are the only ones who can SSH into any production systems (<50 engineers, I'd guess). Engineers write the code (mandatory green build and peer review required before merge to protect against bad actors.. but that's just good engineering practice too!), build their own deployment pipelines on Bamboo or Bitbucket Pipelines to push up code & assets to a docker registry, and ultimately the deployment pipelines make some API calls to deploy docker images. Engineers are also responsible for keeping those services running; most products (such as Jira, Confluence, Bitbucket, etc) also have a dedicated SRE team who are focused on improving reliability of services which support that product. The vast majority (95%) of our production issues are troubleshooted by looking at Datadog metrics (from CloudWatch & services publish a great deal of metrics too) and Splunk (our services log a lot, we log all traffic, and the host systems also ship their logs off). Fixes are usually to do an automated rollback (part of the PaaS), turn off a feature flag to disable code, redeploy the existing code to fix a transient issue (knowing we'll identify a proper fix in the post incident review), or in rare cases, roll forward by merging a patch & deploying that (~30 mins turnaround time - but this happens <5% of the time). Good test coverage, fast builds (~5 mins on avg), fast deploys, and automated smoke tests before a new deploy goes live all help a lot in preventing issues in the first place. It's not perfect, but it works a lot better than you might expect.
  - justinclift 8y ago
    Interesting. With the deployed docker images, is that only for internal applications, or do you do external (public facing) applications as well? Asking because is one of the conceptual things I've been trying to figure out. Still currently using non Docker deployment of production stuff for our public services. Have looked at Docker a few times, but for deployment to the public internet where being accessible to clients on both IPv4 and IPv6 is mandatory, it just doesn't seem to suit. Docker (swarm) doesn't seem to do IPv6 at all, and the general networking approach in non-swarm docker seems insecure as hell for public services + it also seems to change arbitrarily between versions. For a (currently) 1 man setup, it seems like a bad use of time to have to keep on top of. ;) Maybe using Nginx on the public services, reverse proxying to not-publicly-accessible docker container hosts would be the right approach instead? Btw - asparck.com (as per your profile info) doesn't seem to be online?
deathanatos 8y ago
Tooling requires access. Are you saying no dev should ever `strace` a process? (This requires not only access, but presuming my UID != the service UID for the service, sudo, too.) Note that I'm not saying devs should have access to every production machine; I'm only saying that access should be granted to devs for what they are responsible for maintaining. Sure, one can write custom tooling to promote chosen pieces of information out of the system into other, managed systems. E.g., piping logs out to ELK. And we do this. But it often is not sufficient, and production incidents might end up involving information that I did not think to capture at the time I wrote the code. Certain queries might fail, only on particular data. That data may or may not be logged, and root-causing the failure will necessitate figuring out what that data is. And it may not be possible to add it to the code at the time of the incident; yes, later one might come back and capture that information in a more formal channel or tool now that one has the benefit of hindsight, but at the time of the outage, priority number one is always to restore the service to a functioning state. Deploying in the middle of that might be risky, or simply make the issue worse, particularly when you do not know what the issue is. (Which, since we're discussing introspecting the system, I think is almost always the part of the outage where you don't yet know what is wrong.)
- deleted 8y ago
  [deleted]

by _jal 8y ago

justinclift 8y ago

Yeah, I've occasionally wondered how (if) the DevOps approach is implemented at places which have to be SOX compliant. https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act One of the technical things it does is segregate (on purpose) developers from any kind of production access. This is because (from historical experience) a "bad apple" developer can do amazingly fraudulent things, and have a more than reasonable chance of covering it up. o_O

crankylinuxuser 8y ago
I work in an org that has strong partitions between devs and sysads. Dev can make and test code. They have no access to prod. Sysads can configure and execute what devs make. They have read-only access to the source repos. Problems with this are as follows (real, not imagined) 1. AWS cloudformation scripts - who makes them? If dev does, sysads can't change. 2. Does dev have the security mindset to maintain configurations in IaaS things like Cloudformation? Who reviews things like NACLs, Security Groups, VPCs, and the like? 3. Scripts - how big or what impact does a script need to be written by sysad or dev? 4. Oncall - normally are sysads job, but when you implement strong gates between dev/sysad, you need oncall devs.
- justinclift 8y ago
  Thanks, that all falls in line with roughly the kind of problems I'd expect. :)
- deleted 8y ago
  [deleted]
hessproject 8y ago
I work on a SOX compliant project, it works pretty much as described. As a developer I have no access to prod, relying on other teams to make system changes and creating a lengthy mandated paper trail for the SOX audit team to look over. Not to say there aren't headaches with the approach, but thankfully at a big enough organization it's become a relatively smooth process.
caspar 8y ago
Engineering manager (& previously a developer) at a ~2k developer SOX-compliant company chiming in here. We have a platform as a service team which maintains our PaaS infrastructure (think an internal version of Heroku), and they are the only ones who can SSH into any production systems (<50 engineers, I'd guess). Engineers write the code (mandatory green build and peer review required before merge to protect against bad actors.. but that's just good engineering practice too!), build their own deployment pipelines on Bamboo or Bitbucket Pipelines to push up code & assets to a docker registry, and ultimately the deployment pipelines make some API calls to deploy docker images. Engineers are also responsible for keeping those services running; most products (such as Jira, Confluence, Bitbucket, etc) also have a dedicated SRE team who are focused on improving reliability of services which support that product. The vast majority (95%) of our production issues are troubleshooted by looking at Datadog metrics (from CloudWatch & services publish a great deal of metrics too) and Splunk (our services log a lot, we log all traffic, and the host systems also ship their logs off). Fixes are usually to do an automated rollback (part of the PaaS), turn off a feature flag to disable code, redeploy the existing code to fix a transient issue (knowing we'll identify a proper fix in the post incident review), or in rare cases, roll forward by merging a patch & deploying that (~30 mins turnaround time - but this happens <5% of the time). Good test coverage, fast builds (~5 mins on avg), fast deploys, and automated smoke tests before a new deploy goes live all help a lot in preventing issues in the first place. It's not perfect, but it works a lot better than you might expect.
- justinclift 8y ago
  Interesting. With the deployed docker images, is that only for internal applications, or do you do external (public facing) applications as well? Asking because is one of the conceptual things I've been trying to figure out. Still currently using non Docker deployment of production stuff for our public services. Have looked at Docker a few times, but for deployment to the public internet where being accessible to clients on both IPv4 and IPv6 is mandatory, it just doesn't seem to suit. Docker (swarm) doesn't seem to do IPv6 at all, and the general networking approach in non-swarm docker seems insecure as hell for public services + it also seems to change arbitrarily between versions. For a (currently) 1 man setup, it seems like a bad use of time to have to keep on top of. ;) Maybe using Nginx on the public services, reverse proxying to not-publicly-accessible docker container hosts would be the right approach instead? Btw - asparck.com (as per your profile info) doesn't seem to be online?

deathanatos 8y ago

Tooling requires access. Are you saying no dev should ever `strace` a process? (This requires not only access, but presuming my UID != the service UID for the service, sudo, too.) Note that I'm not saying devs should have access to every production machine; I'm only saying that access should be granted to devs for what they are responsible for maintaining. Sure, one can write custom tooling to promote chosen pieces of information out of the system into other, managed systems. E.g., piping logs out to ELK. And we do this. But it often is not sufficient, and production incidents might end up involving information that I did not think to capture at the time I wrote the code. Certain queries might fail, only on particular data. That data may or may not be logged, and root-causing the failure will necessitate figuring out what that data is. And it may not be possible to add it to the code at the time of the incident; yes, later one might come back and capture that information in a more formal channel or tool now that one has the benefit of hindsight, but at the time of the outage, priority number one is always to restore the service to a functioning state. Deploying in the middle of that might be risky, or simply make the issue worse, particularly when you do not know what the issue is. (Which, since we're discussing introspecting the system, I think is almost always the part of the outage where you don't yet know what is wrong.)

deleted 8y ago
[deleted]