20 ms·
While the article goes into the more technical reasons for not using Docker in production, the practical reason "Why Docker Is Not Yet Succeeding Widely in Prod
by minimaxir 11y ago
While the article goes into the more technical reasons for not using Docker in production, the practical reason "Why Docker Is Not Yet Succeeding Widely in Production" is that if it ain't broke, don't fix it. The advantages of Docker do not necessarily outweigh the opportunity cost of rewriting the startup's entire infrastructure.
Docker will likely be more prevalent in a few years with startups who have built their infrastructure form the ground up.
- nailer 11y agoEven if you're building brand new infrastructure from scratch right now many issues (discussed elsewhere in these comments and the article) are still unsolved.
- eropple 11y agoYup. At my last gig we built out a Mesos cluster and were deploying Docker containers, but we couldn't answer "how do we practically secure this to the same level as independent virtual machines?" and, finding no good answer, we went back to auto-scaling groups and baked AMIs.
- toomuchtodo 11y agoYes, doing the same thing here as well. Still using docker to streamline deployments though, but one docker container/role per instance, no "orchestration" for containers (baked AMIs, ASGs).
- eropple 11y agoI did that at one place, but I wasn't super satisfied with the process--having to download container images on spin-up was annoyingly slow and I didn't feel like we were getting better dev/prod consistency versus Vagrant and Packer.
- toomuchtodo 11y agoWe bake the container into the AMI, so no fetch is necessary at spin up (there is no cost to generate AMIs, only storage fees, so cost is not an issue). Packer is used to build the AMIs with the containers built-in, and Docker is used both in Prod (single container to each AMI) and Dev (Docker Compose to bring up entire dev env locally). Both used a shared docker registry.
- eropple 11y agoAhh, gotcha. That's a neat approach, though the double hit of Docker builds + AMI builds feels a little weird to me. Thanks for the insight.
- Leon 11y agoI also do what the other poster does, but we take it a step further and make sure that the layers on the image have smaller and more variable layers near the last layer add. On instance startup we can do a docker pull and bring down only a few k of bytes for docker image updates. This way we can update the ami less often (which it takes longer anyway) and we don't worry too much about pushing updates to the container repo without having to batch ami builds for quicker turn around deployment.
- toomuchtodo 11y agoIf you're in AWS, I wouldn't worry about how many bytes docker image updates take. Our registry is using S3 as its backend, and I can pull images under 100MB in a few seconds.
- Leon 11y agoIt's not always an option to host and manage a registry, some parts it's easier for the customers to rely on a registry service like quay, in which case it can make a difference for some images to think about layering. But you are right, s3 is fast and that's one of the reasons I'm glad Deis moved to support s3 out of the box.
- dehora 11y agoI was wondering, what did you think of the production readiness of Mesos independent of Docker?
- eropple 11y agoSeems reasonable, if you want to be running lots of things on the same boxes without isolation (I'm not comfortable with that, but you might be). If you're sharing those resources for stuff like Spark, ElasticSearch, etc. I think it makes sense as a work scheduler, but there are a lot of other options to consider too.
- devonkim 11y agoIt's quite possible but the most straightforward answer is somewhat ugly - install endpoint security in every container. For example, each container would need to have intrusion detection, iptables, etc. Other options would include having containers route traffic with a virtual LAN setup and you have a container whose function is to replace your usual network security appliances. And the irony is that shared services like that can be put into both control and data planes which is easy with hypervisors and software defined networking combined with storage fabric security. When it comes to security, you honestly should be securing things at every layer anyway, but in a lot of places I see people not bothering with iptables and delegating 80%+ of the security responsibilities to operations while application teams focus upon application security.
- shawnee_ 11y agoDocker will likely be more prevalent in a few years with startups who have built their infrastructure form the ground up. The opposite seems likely ... Docker will fade and become deprecated as building infrastructure from the ground up locally to feed into the cloud becomes cheaper and cheaper still. AWS is not always so cost-effective when you truly dig in and crunch the numbers. My guess as to why Docker won't succeed widely in production is because it's a software-based solution trying to glue together slippery pieces that just don't want to be glued together. The core issue of security will never be solved by a Docker-like solution; that problem is best solved by integrated hardware. This very issue is being addressed in ClearLinux: http://sched.co/3YD5 http://sched.co/3YD5
- toomuchtodo 11y agoDevOps/infra guy here rolling out Docker startup-wide at the moment. You and minimaxir are both correct. With regards to docker/lxc/container security, you're right. Some of the biggest players haven't solved the lxc/docker/container security issues yet; its a really hard problem to solve. Breaking out of container will always be easier than breaking out of deeper levels of virtualization (Xen/KVM).
- mjn 11y ago> Breaking out of container will always be easier than breaking out of deeper levels of virtualization (Xen/KVM). I agree it's not easy to get right, but it doesn't seem necessary that containers will always be leaky. Solaris/Illumos Zones are an OS-level virtualization approach that's pretty airtight, for example.
- ssmoot 11y agoI agree. But that's my biggest problem with Docker. Who runs SmartOS and uses Zones? Why? When you have a local server, that supports KVM and Zones, you choose KVM as the cleaner abstraction. While surrounded by neat tech, Zones are actually a bit of a pain and not all that portable between systems IME. OTOH I can `zfs send/recv` over SSH, drop a short bit of JSON in, and have my KVM instance reliably moved to another SmartOS box 100% of the time, no worries. So unless you're really worried about that last 5% or whatever of overhead, what's the point of Docker? It's not actually very portable at all it seems (on my Mac I'd have to run it inside VirtualBox). I don't have much experience with it, but my guess is that similar to Zones, you're at the mercy of the host system as far as common dependencies like OpenSSL or gcc go. It seems like a solution to a problem I'm having trouble even imagining. A slightly lower overhead, less secure, less portable lightweight "VM" with slightly less overhead. I guess if you're a PaaS and you could increase margins by 5% overnight by switching to Docker that might make sense? As someone who's set up Solaris 10, OpenBSD, FreeBSD, SmartOS, Debian, Redhat, Ubuntu, KVM, Xen, etc etc etc, I just have a real hard time figuring out Docker's value proposition. It seems like the Solaris world went from Zones to KVM, and some people are attempting to do just the opposite. Which I just can't think of a good excuse for.
- inopinatus 11y agoDocker solves a problem that most people don't have. It's not a PaaS, rather, it's (some of) the building blocks to create your own PaaS. Most folks don't need that. Most folks want to put files on a server and start a process. For those folks, Docker in the raw ends up being a whole lot of confusing & unnecessary scaffolding.
- dasil003 11y agoThere was also a time when most people thought they didn't need version control. Back in the 80s and 90s it was a justifiable viewpoint because existing version control systems sucked. The problem with Docker is not that it doesn't solve (or attempt to solve) widespread problems. At its best, Docker gives you dev/production parity, and dependency isolation which is useful even for solo developers working part-time. The problem is that it's not a well-defined problem that can be solved by thinking really hard and coming up with an elegant model—like, for example, version control—it's messy and the effort to make it work isn't worth it most of the time right now. That's no reason to write off Docker though. Pushing files to manually configured servers or VPSes is messy and leads to all kinds of long-term pain. You can add Chef / Puppet, but it turns into its own hairy mess. There's no easy solution, but from where I stand, the abstraction that Docker/LXC provide is one that has the most unfulfilled promise in front of it.
- batou 11y agoAnsible, systemd and go is stealing my heart at the moment. Basically pick the tech that doesn't cause the problems to start with. I still reckon that the main reason VMware ESX is as successful as it is comes down to the lack of isolation and sheer deployment hell that windows has been for years. The same can be said for python or ruby on a Linux machine for example. Docker removes some of that pain like ESX does.
- tracker1 11y agoThat's a bit of my point... If you're building relatively small independent services with Docker you can deploy service A with node 0.10 as it's tested environment and service B with iojs 2.4 on the same server, without them conflicting... when you need to update/enhance/upgrade service A you then can update the runtime. The same can be said for ruby, python and any number of other language environments where you have multiple services that were written at different times with differing base targets. I've seen plenty of instances where updating a host server to a new runtime breaks some service that also runs on a given server. With docker, you can run them all... given, you can do the same with virtualization, but that has a lot more overhead. It's about maximum utilization with minimal overhead... For many systems, you only need 2-3 servers for redundancy, but a lot can run on a single server (or very small cluster/set) I have to agree on ansible, systemd and go... I haven't done much with go, but the single executible is a really nice artifact that's very portable... and ansible is just nice. I haven't had the chance to work with systemd, but it's at least interesting.
- KirinDave 11y ago> is that if it ain't broke, don't fix it. I hate being passive-aggressive so I'll be directly aggresive here: this mentality is a way to say, "I don't want to revisit the operational aspects of my system because I don't like to do that work. Find someone else." Like any aspect of your system, your ops and deploy components can rot. Pretending otherwise is outright ignoring a consistent lesson offered by those who came before and have failed over and over. Docker offers to take over as a project many aspects of the system subject to bit-rot and make an explicit and consistent container abstraction for software to compose. While it has many features we do need (I agree wholeheartedly that it'd be great to parallelize layer creation, less so about secret exposure since the environment & volume tooling already can do that), it has also replaced whole categories of software and devops tooling with simple and extensible metaphors.
- feld 11y agoI disagree. It's all about "the evil you know". And then there's the part where Weave is slow, so you might as well stick to VMs or hardware... http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/ http://www.generictestdomain.net/docker/weave/networking/stu...
- KirinDave 11y agoOn Weave, yeah. Not the best performance, and improving. Flannel is obviously better, and one of the reasons we're excited about rocket. That changes almost nothing for shops hoping to move to containerized architectures now. It's only for existing early adopters. The idea that docker introduces that much uncertainty is outright fear mongering. There is a huge amount of recalcitrance in the community to do anything meaningful in the space due to a proposed risk aversion. My personal opinion is that we're all pretending we didn't write incredibly delicate and brittle provisioning and monitoring code with very dated tools. Many people I know, and more than a few I respect, ultimately point to all their provisioning shell scripts as the ultimate reluctance to change things. "It will be really hard to migrate and test these! Generating them is a pain!" Of course, the elephant in the room is we all knew this going into it and we all know we SHOULDN'T have been doing things like generate shell script execution and using git to provision on production boxes and w/e other hacky shit we've done. Of course, what we have is not any one thing but all too often an amalgam of spare hours and quick fixes and patches laid over some existing provisioning system like salt, ansible (or just a whole shit ton of puppet work). Counter-intuitively, suddenly everyone has become a devops luddite when it comes to a genuinely novel approach even though container abstractions have already proven themselves at scale. People hem and haw and suggest that somehow it's not ready for production. Meanwhile major players in the space are already using them, even for core services, with excellent results. Lightweight containerization has been used to solve this for awhile now. Docker as a product and initiative is relatively new, but to suggest it was the first example of a container engine used in production ignores the actual history of lightweight containers.
- api 11y agoI have mixed feelings about Docker. I've found three major use cases so far: (1) Testing. (2) Build environments -- it's helpful to build distribution Linux binaries in older Linux versions like CentOS 6 so that they'll work on a wider range of production systems. (3) Installing and running "big ball of mud" applications that want to drag in forty libraries, three different databases, memcached, and require a custom Apache configuration (and only Apache, thank you very much). #3 is really the killer app. This has led me to conclude that Docker is a stopgap anesthetic solution to a deeper source of pain: the Rube Goldberg Machine development anti-pattern. More specifically, Docker is a far better solution than the abomination known as the "omnibus package," namely the gigantic RPM or DEB file that barfs thousands of libraries and other crap all over your system (that may conflict with what you have). Well written software that minimizes dependencies and sprawl and abides by good development and deployment practices doesn't need Docker the way big lumps of finely woven angel hair spaghetti do. Docker might still be nice for perfect reproducibility, ability to manage deployments like git repos, and other neat features, but it's less of a requirement. It becomes maybe a nice-to-have, not a must-have. But... if my software is not a sprawling mess that demands that I mangle and pollute the entire system to install it, why not just coordinate development and deployment with 'git'? Release: git tag. Deploy: git pull X, git checkout tag, restart. Finally, Docker has a bit of systemd disease. It tries to do too much in one package/binary. This made the rounds around HN a while back: https://github.com/p8952/bocker https://github.com/p8952/bocker It demonstrates that at least some of Docker's core functionality does not require a monster application but can be achieved by using modern filesystems and Linux features more directly. So honestly I am a bit "meh" about Docker right now. But hey it's the hype. Reading devops stuff these days makes me wonder if "Docker docker docker docker docker docker docker" is a grammatically correct sentence like "Buffalo buffalo buffalo buffalo buffalo buffalo."
- davexunit 11y ago>Docker might still be nice for perfect reproducibility Docker actually doesn't help reproducibility at all, because the underlying reproducibility problems present in the distro and build systems are used are still present. See GNU Guix, Nix, and Debian's Reproducible Builds project for efforts to make build truly reproducible. I had a good laugh when I read "the Rube Goldberg Machine development anti-pattern". This describes the situation of "modern" web development perfectly. I'll add that such software typically requires 3 or more different package managers in order to get all of the necessary software. And yes, Omnibus is an abomination and Docker is much better. I think Docker is papering over issues with another abstraction layer. It's like static linking an entire operating system for each application. Rather than solving the problem with traditional package management, Docker masks the problem by allowing you to make a disk image per application. That's great and all, but now you have an application that can only reasonably be run from within a Linux container managed by Docker. Solving this problem at the systems level, which tools like GNU Guix do, allows even complex, big ball of mud software to run in any environment, whether that is unvirtualized "bare metal", a virtual machine, or a container.
- brianwawok 11y agoNew startup. Did not use docker. Happy with scripts and ansible.
- joshmn 11y agoI have a client, pre-funding but has some revenue, who is hell-bent on using their Rackspace credit for their production environment. "But we have a sysadmin who works for yahoo" He tries to demo me what they have currently and the damn thing timed out during login. I laughed. The cost of these headaches is easily avoidable. Get off the ground and running first, pay the kind-of-premium Heroku bill, and when you're ready to really scale, make the switch. There are few exceptions to managing an infrastructure, such as RackSpace, a cluster of AWS nodes, your own metal, etc. versus something like Heroku.