4 ms·

Immutable infrastructure is the future and it seems that even Netflix is planning to use containers for that: "Containers provide an interesting potential solut

by sytse 11y ago

Immutable infrastructure is the future and it seems that even Netflix is planning to use containers for that: "Containers provide an interesting potential solution to the last two challenges and we are exploring how containers can help improve our current build, bake, and deploy experience." I think that the future of deployments means that it is closely integrated with your source code management. Every new push builds a container that can go through the following steps: 1. Deployment (server with only test traffic) 2. Post-deploy test (smoke test) 3. Canary (part of the traffic) 4. Live / multiregion deploy 5. Manual overrides http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html http://techblog.netflix.com/2015/11/global-continuous-delive... "Spinnaker also provides cluster management capabilities and provides deep visibility into an application’s cloud footprint. Via Spinnaker’s application view, you can resize, delete, disable, and even manually deploy new server groups using strategies like Blue-Green (or Red-Black as we call it at Netflix). You can create, edit, and destroy load balancers as well as security groups." Also see https://gitlab.com/gitlab-org/gitlab-ce/issues/3286 https://gitlab.com/gitlab-org/gitlab-ce/issues/3286

lwander 11y ago
Google is actively adding Kubernetes support as well: https://github.com/spinnaker/spinnaker/issues/707 https://github.com/spinnaker/spinnaker/issues/707
chei0aiV 11y ago
> Immutable infrastructure is the future Are there any downsides to immutable infrastructure?
- moondev 11y ago
  It takes a while to bake the image, and storing thousands of images can add up. However containers can address both points.
- dminor 11y ago
  Build wait times invoke 90's nostalgia.
- phamilton 11y ago
  You need to commit to it. You just toe the line. If you currently depend on being able to SSH into boxes and tail logs you will feel a lot of friction. However, practices like those tend to be a crutch anyway.
- riskable 11y ago
  I'm on a team developing an immutable infrastructure solution with (Docker) containers at a large financial institution. The biggest downsides so far: * It can get complicated very quickly since it involves a lot of stages, systems, and loads of brand new tools where you're unlikely to be able to hire existing talent (you need people who can just read the docs on a complicated tool/architecture like Kubernetes and then start using it). * Speaking of talent, you need developers who are reasonably well-versed in Linux systems administration. Could your developers write a shell script that can configure every little aspect of a Linux host from packaging to authentication? Your developers also need to understand some hard core topics of TCP/IP (e.g. anycast), firewalls (e.g. port mapping), and DNS (TTLs, views, SRV records, and more). They also need to be well-versed in security (e.g. the impact of running as root inside a container) and especially authentication (e.g. HMAC, securely storing shared secrets) and encryption (SSL, managing certificates, knowing which hashing algorithm to use)! * Dealing with existing bureaucracy. Especially rules and policies in regards to security. For example, some people will think that containers and immutable VMs should be treated like regular hosts. That can mean waiting hours for an inventory system to acknowledge their existence before you can put them into production. Or having to wait 30 minutes after every container/VM is brought up so that a security tool can perform a scan even though each container/VM would be identical in every way and have already been scanned before production deployment. Sigh. Of course, a lot of that depends on where you work. At my work right now the biggest challenge is the bureaucracy for certain. If I can get containers classified differently than VMs then all my policy/BAU problems will (likely) go away. If I can't get that then, really, there's no point to our efforts since there would be no advantage to end users (the consumers of what we're building). I may be able to live with a pointless, half-hour-long security scan every time we bring up a new container but I absolutely cannot live with an 8-hour-long wait for a container to show up in our system-of-record (inventory system).
  - matt_wulfeck 11y ago
    Sounds like Devops.
  - jfindley 11y ago
    We are doing something very similar (mesos rather than k8s), also in a financial organisation. Our main issue is also security - as docker containers aren't very good at actually containing, from a security PoV, we're using SELinux MLS, assigning each container a unique SELinux level/category (fedora-atomic sets this up nicely for you if anyone wants to try). This is great for security, but breaks so many things - lots of the cutting edge work with docker and related tech isn't done by people that care deeply about security, or really understand this sort of usecase, so we run into a huge amount of edgecases and bugs. We spend a lot of time fighting this. It also makes having a really good build/deploy pipeline essential, as well as a good environment set up to reproduce and diagnose issues.
- sytse 11y ago
  I think immutable infrastructure still requires a lot of tools and moving parts today. We're spending a lot of time thinking on how to improve this. Two idea's we have: 1. Deploy directly from GitLab to Kubernetes https://gitlab.com/gitlab-org/gitlab-ce/issues/14040 https://gitlab.com/gitlab-org/gitlab-ce/issues/14040 2. Add a container registry to GitLab itself https://gitlab.com/gitlab-org/gitlab-ce/issues/3299 https://gitlab.com/gitlab-org/gitlab-ce/issues/3299 Any comments or suggestions are welcome.

by sytse 11y ago

lwander 11y ago

Google is actively adding Kubernetes support as well: https://github.com/spinnaker/spinnaker/issues/707 https://github.com/spinnaker/spinnaker/issues/707

chei0aiV 11y ago

> Immutable infrastructure is the future Are there any downsides to immutable infrastructure?

moondev 11y ago
It takes a while to bake the image, and storing thousands of images can add up. However containers can address both points.
dminor 11y ago
Build wait times invoke 90's nostalgia.
phamilton 11y ago
You need to commit to it. You just toe the line. If you currently depend on being able to SSH into boxes and tail logs you will feel a lot of friction. However, practices like those tend to be a crutch anyway.
riskable 11y ago
I'm on a team developing an immutable infrastructure solution with (Docker) containers at a large financial institution. The biggest downsides so far: * It can get complicated very quickly since it involves a lot of stages, systems, and loads of brand new tools where you're unlikely to be able to hire existing talent (you need people who can just read the docs on a complicated tool/architecture like Kubernetes and then start using it). * Speaking of talent, you need developers who are reasonably well-versed in Linux systems administration. Could your developers write a shell script that can configure every little aspect of a Linux host from packaging to authentication? Your developers also need to understand some hard core topics of TCP/IP (e.g. anycast), firewalls (e.g. port mapping), and DNS (TTLs, views, SRV records, and more). They also need to be well-versed in security (e.g. the impact of running as root inside a container) and especially authentication (e.g. HMAC, securely storing shared secrets) and encryption (SSL, managing certificates, knowing which hashing algorithm to use)! * Dealing with existing bureaucracy. Especially rules and policies in regards to security. For example, some people will think that containers and immutable VMs should be treated like regular hosts. That can mean waiting hours for an inventory system to acknowledge their existence before you can put them into production. Or having to wait 30 minutes after every container/VM is brought up so that a security tool can perform a scan even though each container/VM would be identical in every way and have already been scanned before production deployment. Sigh. Of course, a lot of that depends on where you work. At my work right now the biggest challenge is the bureaucracy for certain. If I can get containers classified differently than VMs then all my policy/BAU problems will (likely) go away. If I can't get that then, really, there's no point to our efforts since there would be no advantage to end users (the consumers of what we're building). I may be able to live with a pointless, half-hour-long security scan every time we bring up a new container but I absolutely cannot live with an 8-hour-long wait for a container to show up in our system-of-record (inventory system).
- matt_wulfeck 11y ago
  Sounds like Devops.
- jfindley 11y ago
  We are doing something very similar (mesos rather than k8s), also in a financial organisation. Our main issue is also security - as docker containers aren't very good at actually containing, from a security PoV, we're using SELinux MLS, assigning each container a unique SELinux level/category (fedora-atomic sets this up nicely for you if anyone wants to try). This is great for security, but breaks so many things - lots of the cutting edge work with docker and related tech isn't done by people that care deeply about security, or really understand this sort of usecase, so we run into a huge amount of edgecases and bugs. We spend a lot of time fighting this. It also makes having a really good build/deploy pipeline essential, as well as a good environment set up to reproduce and diagnose issues.
sytse 11y ago
I think immutable infrastructure still requires a lot of tools and moving parts today. We're spending a lot of time thinking on how to improve this. Two idea's we have: 1. Deploy directly from GitLab to Kubernetes https://gitlab.com/gitlab-org/gitlab-ce/issues/14040 https://gitlab.com/gitlab-org/gitlab-ce/issues/14040 2. Add a container registry to GitLab itself https://gitlab.com/gitlab-org/gitlab-ce/issues/3299 https://gitlab.com/gitlab-org/gitlab-ce/issues/3299 Any comments or suggestions are welcome.