42 ms·

Engineering manager (& previously a developer) at a ~2k developer SOX-compliant company chiming in here. We have a platform as a service team which maintains o

by caspar 8y ago

Engineering manager (& previously a developer) at a ~2k developer SOX-compliant company chiming in here. We have a platform as a service team which maintains our PaaS infrastructure (think an internal version of Heroku), and they are the only ones who can SSH into any production systems (<50 engineers, I'd guess). Engineers write the code (mandatory green build and peer review required before merge to protect against bad actors.. but that's just good engineering practice too!), build their own deployment pipelines on Bamboo or Bitbucket Pipelines to push up code & assets to a docker registry, and ultimately the deployment pipelines make some API calls to deploy docker images. Engineers are also responsible for keeping those services running; most products (such as Jira, Confluence, Bitbucket, etc) also have a dedicated SRE team who are focused on improving reliability of services which support that product. The vast majority (95%) of our production issues are troubleshooted by looking at Datadog metrics (from CloudWatch & services publish a great deal of metrics too) and Splunk (our services log a lot, we log all traffic, and the host systems also ship their logs off). Fixes are usually to do an automated rollback (part of the PaaS), turn off a feature flag to disable code, redeploy the existing code to fix a transient issue (knowing we'll identify a proper fix in the post incident review), or in rare cases, roll forward by merging a patch & deploying that (~30 mins turnaround time - but this happens <5% of the time). Good test coverage, fast builds (~5 mins on avg), fast deploys, and automated smoke tests before a new deploy goes live all help a lot in preventing issues in the first place. It's not perfect, but it works a lot better than you might expect.

justinclift 8y ago
Interesting. With the deployed docker images, is that only for internal applications, or do you do external (public facing) applications as well? Asking because is one of the conceptual things I've been trying to figure out. Still currently using non Docker deployment of production stuff for our public services. Have looked at Docker a few times, but for deployment to the public internet where being accessible to clients on both IPv4 and IPv6 is mandatory, it just doesn't seem to suit. Docker (swarm) doesn't seem to do IPv6 at all, and the general networking approach in non-swarm docker seems insecure as hell for public services + it also seems to change arbitrarily between versions. For a (currently) 1 man setup, it seems like a bad use of time to have to keep on top of. ;) Maybe using Nginx on the public services, reverse proxying to not-publicly-accessible docker container hosts would be the right approach instead? Btw - asparck.com (as per your profile info) doesn't seem to be online?
- caspar 8y ago
  Internal and external apps are both deployed on the same PaaS. Only difference is that internal apps aren't reachable "outside the VPN"; when it comes to building your service, it's an extra line of yaml in your service config. There's a network engineering team who works with the PaaS team to make that happen - it's definitely a nice luxury of a big company that you don't need to worry about setting up VPCs etc yourself. The actual PaaS relies pretty heavily on AWS Cloud Formation - it predates swarm, mesosphere, kube, etc. So when we deploy a new version of a service, it's really "deploy an auto scaling group of EC2 instances across several AZs fronted by an ELB, then there's an automatic DNS change made which makes the new stack of EC2 instances live once the new stack is validated as working". The upside of the one service per EC2 image approach is no multi-tenancy interference - downsides are cost and it takes a bit longer to deploy. There's a project underway to switch compute to being Kube-based though, so that's promising. All this is apples and oranges though - solns for big companies don't make sense for a 1-person shop. I still have side projects being deployed via ansible and git pull instead of Docker, because it hasn't been worth the ROI to upgrade to the latter. Re asparck - yeah, it was my personal site but I struggled to find the time to work on it. In the end I decided it was better to have it offline than terribly out of date, but hopefully I'll resurrect it some day.

by caspar 8y ago

justinclift 8y ago

Interesting. With the deployed docker images, is that only for internal applications, or do you do external (public facing) applications as well? Asking because is one of the conceptual things I've been trying to figure out. Still currently using non Docker deployment of production stuff for our public services. Have looked at Docker a few times, but for deployment to the public internet where being accessible to clients on both IPv4 and IPv6 is mandatory, it just doesn't seem to suit. Docker (swarm) doesn't seem to do IPv6 at all, and the general networking approach in non-swarm docker seems insecure as hell for public services + it also seems to change arbitrarily between versions. For a (currently) 1 man setup, it seems like a bad use of time to have to keep on top of. ;) Maybe using Nginx on the public services, reverse proxying to not-publicly-accessible docker container hosts would be the right approach instead? Btw - asparck.com (as per your profile info) doesn't seem to be online?

caspar 8y ago
Internal and external apps are both deployed on the same PaaS. Only difference is that internal apps aren't reachable "outside the VPN"; when it comes to building your service, it's an extra line of yaml in your service config. There's a network engineering team who works with the PaaS team to make that happen - it's definitely a nice luxury of a big company that you don't need to worry about setting up VPCs etc yourself. The actual PaaS relies pretty heavily on AWS Cloud Formation - it predates swarm, mesosphere, kube, etc. So when we deploy a new version of a service, it's really "deploy an auto scaling group of EC2 instances across several AZs fronted by an ELB, then there's an automatic DNS change made which makes the new stack of EC2 instances live once the new stack is validated as working". The upside of the one service per EC2 image approach is no multi-tenancy interference - downsides are cost and it takes a bit longer to deploy. There's a project underway to switch compute to being Kube-based though, so that's promising. All this is apples and oranges though - solns for big companies don't make sense for a 1-person shop. I still have side projects being deployed via ansible and git pull instead of Docker, because it hasn't been worth the ROI to upgrade to the latter. Re asparck - yeah, it was my personal site but I struggled to find the time to work on it. In the end I decided it was better to have it offline than terribly out of date, but hopefully I'll resurrect it some day.