9 ms·
Containers Will Not Fix a Broken Culture
- jefe78 9y agoAs a systems engineer, I struggle with this virtually every day. We're called 'DevOps' by most and anytime we encounter a new problem, everyone invariably screams for containers. Containers aren't a magic bullet. My favourite example is when our AWS TAMs offer a solution, knowing we have ZERO pipeline/infrastructure setup for supporting containers. They always push containers. We don't use containers, stop forcing them down our throat. We've tried, we've been burned, VMs work for us. Stop! When did containers become perceived as the end-all solution? I see their value and uses but they don't meet ours so why have we started ignoring the right solution for the job? I see this everywhere I go.
- jitl 9y agoYou need containers to run on Kubernetes my dude. And running on Kubernetes is critical.
- jefe78 9y agoReminds me of this: http://www.mongodb-is-web-scale.com/ http://www.mongodb-is-web-scale.com/ "Docker, docker, docker!" Edit: I'm not knocking either of these products - I actively use Mongodb in production. I like docker/containers/Kubernetes and have used them for various projects. I just take offence with how people have started ignoring common sense, like: we don't have the tooling in place to support this product, or: it doesn't meet our business needs.
- bdcravens 9y agoI think orchestration is critical. Kubernetes has obviously "won", but we're getting alot of wins from running on ECS that we'd have to rollback and reimplement on K8S, at least until EKS becomes available. (honestly couldn't tell if you were being sarcastic, so assumed you weren't)
- DougWebb 9y agoIf you aren't currently struggling to manage a large pool of servers / services, then you don't need Kubernetes. Not yet, at least. It's over-complicated overkill if you don't need it yet. So are containers, actually, but the bar is a lot lower for them.
- scarface74 9y agoYes I know you were being sarcastic. But you don't need k8s or containers for orchestration. I chose Hashicorp's Nomad (I'm the dev lead for our company) precisely because I didn't want to commit to Docker from day one but I did want to leave that option open. Nomad works with everything - Docker containers, jar files, shell scripts, raw executables, etc and is dead simple to set up - one < 20Mb self contained executable that works as a client, server and as a member of a cluster. Configuration is dead simple if you use Consul.
- rospaya 9y agoRants like this are commonplace at my "devops" team as well. Docker is being pushed down everyone's throats and next up is Kubernetes.
- xkcd-sucks 9y agoIt could be worse-- Your AWS TAMs could be pushing Lambda functions
- erikb 9y agoContainers are not there to solve a problem. They are there to produce buzz, then busy engineers, then bills, all packaged in a way that overloaded (and sometimes quite unskilled) manager can show something to their bosses. And in that regard Containers are successful as hell. That's why we have a religion around them now. You can hate it but you can't really ignore it if you need to pay rent and work in the industry.
- deleted 9y ago[deleted]
- bdcravens 9y agoThey really did solve a lot of problems for us, but I don't think we over-engineered our solution, which I think is an easy rabbit-hole to dive into.
- shitgoose 9y agoi am sure there are folks who solved a lot of their problems by writing monolith apps. they just don't walk around preaching their religion.
- bdcravens 9y agoYeah, I would agree there's quite a disconnect between "rubber hits the road" and the "Gospel of Docker". This was magnified when I attended Dockercon in Austin last year. As a very small team that has used Docker (and ECS) to solve some problems, I was excited to learn more. It became quickly apparent that I was at the wrong conference for that: it felt like an enterprise vendor party where everyone was passing around cups of kool-aid.
- erikb 9y agoYeah the kool-aid stuff is really annoying. Btw have you checked out the details of systemd slices? I somehow thought it's nearly the same as docker, just without the hype. Just practical technology on the level it's supposed to be.
- merb 9y agowell as somebody already pointed out. docker/containers are useless. combine them with a useful system like kubernetes, they can be really useful.
- mschuster91 9y ago> docker/containers are useless. combine them with a useful system like kubernetes, they can be really useful. Not exactly. For example, running your core infrastructure via Docker containers makes "ordinary" maintenance/disaster recovery really easy. We run e.g. JIRA and Confluence in Docker... and it's a breeze to make a consistent backup and restore: stop the application and DB container, rsync delta to Netapp, restart the containers and the Netapp automatically does a snapshot. Daily, fully consistent backups with only 5 minutes of downtime - and because a restoration is as easy as "rsync the data out of the Netapp/LTO and do a (documented) docker run command with appropriate mounts" it's easy to verify that the backup actually works. Prior to using Docker, testing the backup involved setting up a server, copying data around and manually importing it. Needless to say it was not done very often. But now we can apply this policy to all our services, plus we can test version upgrades with minimal effort compared to setting up the software from scratch. Oh, and upgrading the software versions itself is also easy: rsync delta to backup, stop container, quick rsync again to ensure consistency (e.g. lock files, not flushed DB writes), stop & remove old docker container, run new docker container w/ new version, check if upgrade went OK - which it always did so far, but in case it did not, it's a matter of minutes to do a full rollback. The downside, of course, is that we depend on the container author(s) to provide regular new builds which incorporate not only new app versions but also updates for the packages of the "OS" inside the container. For example if the thread vector is something like "PHP/Java does a RCE when passed a certain HTTP header", on an "old school" system I'd do an apt-get update && apt-get upgrade and that's it - with Docker I either depend on the container vendor or have to roll my own docker images with e.g. "FROM <vendorimage>; RUN apt-get update && apt-get dist-upgrade && apt-cache clean"...
- eikenberry 9y ago100% wrong. Containers normalize deployment making the underlying tech stack irrelevant. This is their first and still primary use and they work really well for this. There is nothing wrong with 1 container per system, just like you used to do with standard deployment.
- AlexCoventry 9y agoI'm not sure what the point of this piece is. It could have stopped at "Complex socio-technical systems are hard; film at 11" and been just as informative.
- jacques_chester 9y agoTechnology can't fix socio-economic problems, but it can shift the landscape of what is possible. Old-school software engineering is very much about reports, forms, documents. Each tries to gather to itself every instance of some kind of information. Here is the Customer Requirements Document. Here is the Software Requirements Document. Here is the Software Design Document. Here are the Software Test Plans. These days most places work out of an issue management system. JIRA tickets, Github issues, stories in Pivotal Tracker, whatever. The work is broken into small chunks with their own lifetime. We don't wait while all the requirements pile up before opening the dam and letting them flow downstream in a batch. Each goes when it's ready to go. This is not possible without the right tools. That's a lie. It's totally possible. With 1980s word processing and spreadsheets you could absolutely do everything Tracker or JIRA do. You could use stacks of 3x5 cards to track thousands of items across dozens of teams. But you probably don't. The tooling lowers the threshold of the possible, in a social and economic sense. No, containers aren't miraculous. Of themselves, they do nothing to fix other problems. But they make it possible to achieve improvements that are more expensive and difficult in other ways. They lower the barrier of possibility. The landscape of alternatives shifts, mountains become hills. I've been on both sides of that divide now. As a consulting engineer I saw projects rapidly iterating but not being able to deploy ("ops are too busy right now"), leading to dozens of handsomely-billed hours being squandered in meetings and workarounds and emails and chats and phonecalls trying to get the code into any kind of production. I've also seen projects where deployment took an hour or two to set up and that was that. People got on with the job. And a major difference was the platform. One more thing. > Development teams love the idea of shipping their dependencies bundled with their apps, imagining limitless portability. Someone in security is weeping for the unpatched CVEs, but feature velocity is so desirable that security's pleas go unheard. Platform operators are happy (well, less surly) knowing they can upgrade the underlying infrastructure without affecting the dependencies for any applications, until they realize the heavyweight app containers shipping a full operating system aren't being maintained at all. This is a problem buildpacks have solved for well over a decade on multiple independent PaaSes. Disclosure: I work for Pivotal, we do some stuff with containers, but we sorta focus on the parts on top of and before them.
- jt2190 9y agoGiven developer x and developer y, each with adequate technical skills but poor teamwork abilities, there's some business value in being able to employ both given that the cost of dealing with their poor teamwork doesn't exceed the business value they can each deliver. In the past we might have to decide to forego employing one or both. Now we kind of have the option of giving each dev their own "playground" via containers, and not actually expect either to improve their teamwork skills. Again, as long as the cost of supporting the container infrastructure is lower that the business value each dev can deliver, it's a net win. In practice I'm not sure if containers can really deliver on this promise, but it's a very seductive idea.
- cat199 9y agoSame is true for VM's, BSD jails, or even chroot() think the issues being pointed out here are more about the whole 'linux container ecosystem' which has a notion of statically built containers, automated system orchestration, etc, and primarily in an operations context..
- apeace 9y agoThis is really off-base, misses the point, and is another form of the "you don't need containers" criticism which has become very tired at this point. This is mostly a critique of microservice architectures, not containers. If that were the main point I'd have little disagreement. > Someone in security is weeping for the unpatched CVEs... > ...the heavyweight app containers shipping a full operating system aren't being maintained at all... This is just wrong, it's the opposite of that. Never have I had more up-to-date operating systems, programming languages, and frameworks than when I started using containers. It's just so damn easy, especially if you use `FROM python:3` instead of `FROM python:3.6.2`. It auto-updates every time you deploy. > There is no substitute for experimentation in your real production environment; containers are orthogonal to that... They're not orthogonal to it, they're a really useful way to get very, very close to production. The maxim isn't untrue, but again, I sleep better than I ever have in my life because I know that these problems are now rare for me. The difference between my local, staging, and production is tiny. I haven't encountered such an issue in over a year. All of the problems in the article are true no matter what tools you use to build and deploy. The author focuses a lot on developers' desire to go off in a corner and build their own little world. That's still a risk if you're using Ansible or Chef. Bottom line: writing a Dockerfile is the most powerful way I've ever found to define your OS's configuration in code. Stop discouraging people from trying it just so you can make grand arguments about the types of problems every engineering team faces.
- deleted 9y ago[deleted]
- DougWebb 9y agoHow does "It auto-updates every time you deploy" fit with "The difference between my local, staging, and production is tiny"? It sounds like you have little control over your dependencies, and the differences between your local, staging, and production environments are the newer versions of your dependencies which you haven't tested against. I like the idea of being able to precisely control both my code and all of my dependencies, so that I know for certain that I'm deploying exactly the same overall system that I tested. Containers are much better for that than the old way, because you could never be certain that your OS and system software were exactly the same in production as they were local and staging. But to achieve that precision, you need to use precise version numbers, and you need to install your dependencies from a local repository to be really sure.
- mailslot 9y agoSo many companies I've interviewed for are rushing toward microservices and containerization as the cure to all problems. The problem is that the champions often have no clue what any of this means. I recently spoke with a company that had no testing whatsoever for a large production app. When I asked about it, they proudly said "Oh, we do CI. We have Jenkins!" Any tests? "We're going to add them after we move to microservices. Moving away from our monolith is top priority because monoliths are difficult to debug." I see a ton of companies shitting all over best practices and then chanting buzzwords to pretend that they're all about it. That, or gross misunderstanding of any concept behind buzzwords. X company uses Docker. We should use Docker. "Um. This code runs on an FPGA." "Does it run Docker?"
- apeace 9y agoIf it weren't containers it would be a new programming language, framework, or another agile methodology. Your argument has very little to do with containers. Containers are just a better tool for writing OS configuration scripts. (If your team is full of Chef experts then it's not "better" for your team, but for a lot of teams it is). What you're saying applies a lot more to microservices, which are a fundamental architecture choice. Containers aren't, they're just better than a tangle of bash scripts which create stateful VMs. And the problems you're describing apply no matter which tools a team uses. Remember that you can use containers without complicated orchestration or microservices. I think a better argument would be to untangle these three things and describe how each one can solve certain problems or make the problem worse, and under which conditions.
- _ix 9y agoNot your point, but I guess it may Be little different for the Chef experts if container pipelines are defined with Packer.io using Chef provisioners.
- tty7 9y agoContainers are not OS configuration scripts.. they still need a host to run on. They still need networking managed.. im not sure how you figure that one out. Chef experts likely deploy your underlying hosts, setup other required services (eg load balancers, state) while you can provide a generic "container" that will run whatever you want and provide consistent ingress/egress points so the "chef experts" can run it without caring what it is they are running
- EngineerBetter 9y agoIf an enterprise adopts a technology that allows them to go faster, but doesn't change any of their processes to make them go faster, they have effectively thrown money down the drain. I'm sick of seeing tech companies eagerly switch out one tech for another without at least _trying_ to address the people and process problems. Stop trying to reduce costs. Work out your cost of delay, and focus on getting things done more quickly and more effectively, not more cheaply and more efficiently.
- scarface74 9y agoI'm the lead dev for our company and I chose as phase 1 for our implementation Nomad for orchestration, a bunch of "micro-apps" (single purpose executables), and Consul for configuration and service discovery. I chose that combination for ease of use and flexibility - Nomad works with everything raw executables and containers. Of course later on the consultants came along later and scoffed at the simplicity of our process - our deployment process is basically one step in our continuous delivery pipeline - copy the bin/release directory to the destination folder. I tried to get them to articulate a business case for us to use containers. They couldn't come up with one. Then my manager, someone I really respect for his technical acumen finally gave me one. If we go to containers, we don't have to provision servers on AWS. We can use AWS Fargate. Lambda isn't an option we have long running processes that make more sense as apps. I wanted to do Docker anyway eventually just to add a bullet point on my resume and I could have as the dev lead but it felt unethical to make a choice that wasn't best for the business. Now I think it's the right way to go.
- tty7 9y agoSounds like your consultants were not sold on docker/containers as you already supported them. They wanted to lock you into a managed service so that you keep using them. If they really cared about containers they would have helped you continue using nomad/consul as thats a fine combination
- scarface74 9y agoI'm not opposed to a managed service and yes Consul+Nomad is great. But I like the idea of not having to manage VMs. We will probably move to a hybrid approach.
- randyrand 9y agoisn't debugging a bunch of small executables annoying? How do you debug/trace the flow of execution between them easily? You can't even get a good stack trace inter-executable, can you?
- 9y ago