9 ms·

Docker in Production: An Update

170 points by smackay 10y ago

gregmac 10y ago
> I am moderately confident that there is no-one on the planet using Docker seriously AND successfully AND without major hassle. Though the tone of this article is very negative, the conclusion is interesting to me as someone that's considered Docker but hasn't done much with it beyond experimenting locally. I'm assuming if there's any forum that has users that can speak to their use of Docker in production, this would be the place.
- danielbln 10y ago
  We've been running docker in production since late 2013 (data science SaaS). We had some growing pains in the beginning, but since then it has been smooth sailing and a tremendous help in getting our services deployed reliably. However, we have always been very conservative when it comes to upgrading docker and have our own custom glue in place. Still, the statement that "no-one on the planet using Docker seriously AND successfully AND without major hassle" seems majorly hyperbolic.
  - StreamBright 10y ago
    Size of your infra would he helpful.
    - danielbln 10y ago
      Pretty benign, 15 beastly bare metal machines in coloction and a bunch of VMs in AWS. A few database servers in both.
- n0us 10y ago
  Our experience is that it has made running servers locally a hassle because Docker for mac sucks beyond belief but it has been awesome in production. For us the real positive is Kubernetes and tbh docker would be useless without it. I really could care less about docker other than that the Dockerfile format is easy enough to use. We went through the gammut of swarm etc. and were never able to get any of them to work in a meaningful way.
- fnbr 10y ago
  We're technically using it in production for my site, www.bugdedupe.com, but we're still in beta, so we haven't experienced heavy usage. We've been using Kubernetes to deploy Docker containers on Google Container Engine, and while there have been a few issues due to Docker/Kubernetes (namely, getting the containers to expose localhost to each other, and to expose themselves to the world), the issues that we've had so far have been issues that we would have been bit by eventually. Namely, if we hadn't been forced to deal with the issues now, we would have been screwed later. There have been some weird bugs due to the internal environment that containers use (we had extremely slow DNS lookups that caused our request times to shoot up to 9s each). These issues have been transient though, so it's not clear that it's Dockers fault, or if we're making mistakes in our code. Docker has made our deployment much easier. You just build & push your container, and you instantly have a versioned deployable instance of your code. Kubernetes makes it extremely easy to rollout or rollback containers, so I have nothing but good things to say about containers.
  - StreamBright 10y ago
    Size of your infra would he helpful to know.
    - ownagefool 10y ago
      I don't know the above poster but we run over 4500 docker containers on AWS distributed using kubernetes. Docker has indeed caused us a problem or two in the past, but actually it's largely smooth sailing and most of our problems are with the app layers and both docker and kubernetes have largely made our lives much easier. A quick skim would suggests that sits on about 8TB of Ram and 1000vCPUs spread over 100 machines. It's not the biggest stack in the world, but it's doing a good job as our little internal PaaS. We have some non-aws versions too, but I don't have metrics on those right now.
  - gregmac 10y ago
    An interesting comment also from the article: > Google offers containers as a service, but more importantly, as confirmed by internal sources, their offering is 100% NOT Dockerized. > Google merely exposes a Docker interface, all the containers are run on internal google containerization technologies, that cannot possibly suffer from all the Docker implementation flaws.
callumjones 10y ago
> As confirmed by internal sources, they experienced massive troubles to get Docker working in any decent condition I call B.S. on this. Amazon wouldn't have spent so much effort on ECS if this was true.
- user5994461 10y ago
  Amazon employee confirming the article: https://news.ycombinator.com/item?id=12364522 https://news.ycombinator.com/item?id=12364522
  - StreamBright 10y ago
    Details matter at Amazon scale as they say. I have used Docker ever since it was beta, it never made it to our production systems though. LXD looks pretty promising, unprivileged containers, no container service and so on. When I have more time I will investigate how to roll it on CentOS.
- yebyen 10y ago
  Has Amazon spent a lot of effort on ECS? I am totally ignorant here, but the people who I consider more knowledgeable about AWS things have said in a nutshell that "in 2015 the things I heard about ECS were not good things, and I have basically not heard any new things since then." Basically stating that ECS was Amazon's attempt to plant a flag in the container-space and that it was half-hearted, not done with the rigor of solutions like k8s that had additional advantages of also being cloud-agnostic, and finally that ECS was not a solution that anyone they knew was recommending or would recommend.
  - coding123 10y ago
    Don't know about ECS but ECR (the registry) is a freaking godsend.
  - abbyaws 10y ago
    hey yebyen- i was one of the very first beta users of ECS, and I now work for AWS. first off, sounds like maybe we're not doing the best job we can be with educating users on ECS- that's on us, and we'll focus on improving. to dig into your actual question: when ECS launched, the goal was to do a smaller number of things really well, and then listen to the community on what _they_ wanted to see from a container management platform, and grow accordingly. I've seen a number of significant improvements over the last year or so. Off the top of my head: ECR (thanks @coding123), task placement policies and strategies to give developers more control over how they place tasks and use resources, IAM roles for tasks, event stream for cloudwatch events, service level autoscaling, ALB support, and a number of smaller configuration changes, like multiple network modes, and out-of-the-box support for and awslogs driver. also discovered while writing a workshop the other day that there is a pretty sweet first-run wizard for users just trying out ECS for the first time. in any case, a couple of main takeaways here: i'm seeing a focus on adding features and services that reduce the operational work for developers- let AWS worry about scaling and managing your cluster infrastructure, and you can focus on building cool stuff. beyond that, though, developers have asked for more control, flexibility, and extensibility, and i think ECS is working on delivering that: the cloudwatch event stream can be consumed by other services, the blox open source project (and the already opensourced ecs-agent) let you build custom schedulers/functionality on top of what AWS offers, placement policies let you customize how ECS consumes your cluster resources. would love to know where you get your news and why you haven't heard much about ECS, so we can make sure we fix it. if you want to talk more, you can also DM me
Artemis2 10y ago
> Google merely exposes a Docker interface, all the containers are run on internal google containerization technologies, that cannot possibly suffer from all the Docker implementation flaws. Google running their own containerization tech with a Docker interface seems a bit far-fetched given the level of integration of Kubernetes with Docker. That's totally possible though, I'd like to read more about it.
- raesene6 10y ago
  It is far-fetched 'cause it's not AFAIK true. My experience for this is having set up a GKE Kubernetes cluster last night, it was definitely running Docker.
- daenney 10y ago
  From one of the GKE folks: https://news.ycombinator.com/item?id=13715358 https://news.ycombinator.com/item?id=13715358 It's not true.
crb 10y ago
It's great that hftguy thinks Google Container Engine is stable (I work on it) but I'm sorry to say it's very easy to prove that is, in fact, running Docker on the nodes. You can just SSH into one and see for yourself. Kubernetes was built from the ground up to orchestrate Docker. CoreOS did a lot of work to make it possible to trade rkt in for Docker's engine, and the cri (Container Runtime Interface) is now generalising that so that there is a clear abstraction between the kubelet and the engines it orchestrates. Read about it here: http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html http://blog.kubernetes.io/2016/12/container-runtime-interfac... If you want to do things that are different to what we provide support for on GKE as a Managed Service (tm), you're able to run your own Kubernetes clusters on GCE. (We do let you run a Kubernetes alpha version, but only on non-supported clusters that self-destruct after 30 days.
- Draiken 10y ago
  We very recently moved from some bare-metal pet machines into Google GKE and couldn't be happier. Honestly the hardest thing is keeping up with how fast Kubernetes evolves and gets better and better. The same goes for all Google services (pubsub, bigquery, etc) We started the migration on Kubernetes 1.1 and are now live on 1.5.1 Even using it for things we probably shouldn't (old stateful applications) without a single problem. At least no Docker related problems. I don't know, this article seems to be very presumptuous. A lot of bold claims, little backing and, as you state, some pretty false claims. TBH I don't know why it's among the top of HN
- deleted 10y ago
  [deleted]
- SEJeff 10y ago
  Is it eventually going to run rkt/cri-o/lmctfy?
  - crb 10y ago
    If you're a customer and you're interested in either of the first two, please ping me an e-mail. GitHub's lmctfy README states "We are not actively developing lmctfy further and have moved our efforts to libcontainer."
mwpmaybe 10y ago
I am neither a Docker expert nor evangelist, and I have my own gripes and frustrations with it, but this article is full of misinformation and FUD. To wit: > CoreOS is an operating [system] that can only run Docker and is exclusively intended to run Docker. No, it isn't. (Maybe if you substitute "Docker" with "containers.") > First, the main benefit of Docker is to unify dev and production. Having a separate OS in production only for containers totally ruins this point. No, it doesn't. Completely the opposite, in fact: Docker makes it possible to not care about OS differences between dev, test, and prod. > Docker on Debian is major no-go > Docker is 100% guaranteed suicide on Debian 8 and it’s been since the inception of Docker a few years ago > Debian froze the kernel to a version that doesn’t support anything Docker needs and the few components that are present are rigged with bugs. rubs temples It's Linux. It's Debian. You can run any kernel you want. There are plenty of repositories out there with binary kernels, including Debian's own back-ported 4.9 kernel, or you can build one from source. Because, you know, open source. I've run many instances of Jessie on Linode with their 4.8 and 4.9 kernels and it's no problem. Hundreds of companies are running Docker in production on Debian and Debian-derived systems. > I am not aware of any serious companies than run on Ubuntu. Just because you are not aware does not mean they do not exist. How about Netflix, Snapchat, Dropbox, Uber, and Tesla for starters? > I cannot comment on the LTS 16 as I do not use it. It's been out since last April. If you're serious about this survey, this is a no-brainer. > I received quite a few comments and unfriendly emails of people saying to “just” use the latest Ubuntu beta What? No. Just use the latest LTS. > I am moderately confident that there is no-one on the planet using Docker seriously AND successfully AND without major hassle. LOL
- hmottestad 10y ago
  Currently using docker with a Norwegian government organisation. We are very happy with it. Been running production since December. And yes, they run on Ubuntu.
- deleted 10y ago
  [deleted]
- mdekkers 10y ago
  It's Linux. It's Debian. You can run any kernel you want. The guy works in finance, not an area where you can typically say "oh, no worries, we'll just push out this distro with a non-standard kernel." Enterprise IT, and specifically finance (and healthcare) have different requirements. Currently working with a client where not only the full technology stack has to be certified by accountable vendors, but all the internal documentation and customer facing content has to pass strict legal reviews. "Here is Debian with a custom kernel" won't fly. Nether will "Here is Debian" - Ubuntu LTS will barely pass the line, and that will be after a whole lot of timewasting and ass-covering from many people involved.
kordless 10y ago
> CoreOS is an operating that can only run Docker and is exclusively intended to run Docker. This is a patently false statement and needs to be revised, especially given: > I will not comment on it. P.S. I'm getting downvoted for speaking the truth, no matter how pedantic it may seem. CoreOS can run rkt containers and is not exclusively intended to run Docker. One possible reason is because Docker is the 800 lb. gorilla in the room and having options is always good. P.P.S. Gotta love the Streisand Effect. :)
- yebyen 10y ago
  You're not wrong, CoreOS can also run arbitrary go binaries (or anything else that can be statically linked) rigged up into systemd or fleet units, without any containers at all.
- wrs 10y ago
  It is an especially silly statement considering that CoreOS had a much-publicized falling-out with Docker and created rkt to replace it.
merb 10y ago
the problem is, that the docker (community) or rather the company behind it, focuses on too much, instead of just doing one thing and doing it great. no they need to reimplement the world and create their own kubernetes (or some sort of it). their new website is less accessible as it was in the old version, etc... and all the new projects and people. and the various ways to configure docker. why does a simple container engine needs to have so many parameters to configure it? the network experience is still not as good as it could be and docker is still way more complicated to setup than a simple aws and some AWS AMIs.
- holydude 10y ago
  It is simple imho. Docker turned into an enterprise. Attracted a lot of users (hype) and is trying to bank on it
- coding123 10y ago
  I'd like to see Rkt/Coreos folks start showing how their stuff IS accessible than keep spewing this flamewar crap on HN. I've had nothing but excellent comments from the Kube developers, but every time a Rkt dev or fan comes on here, this is the crap we get.
  - jzelinskie 10y ago
    I think you might be conflating pot stirrers with actual rkt devs. CoreOS (along with many other companies) are the Kubernetes developers. FWIW, there was drama with the initial rkt announcement, but we've actually seen docker adopt most of the original criticisms and that should be applauded for both CoreOS and Docker.
deleted 10y ago
[deleted]
holydude 10y ago
I still think that Solaris zones are / were a superior technology. Sadly it died with the demise of Solaris as an operating system. Still in use in some "must have enterprise" types of companies. I would not run anything serious on docker (definitely not yet) if I would not be heavily invested in understanding and contributing to the technology (docker codebase + related). Docker is a nice to have but absolutely abstracts too much and takes control from you. I am wondering what people like Brendand Gregg think about docker.
- marios 10y ago
  Solaris within Oracle is as good as dead from the latest reports. However, Illumos is a Solaris descendant that is very much alive. It has all the good systems stuff that came out of Sun (zones, ZFS, DTrace) and is backed by Joyent (now part of Samsung). There's also FreeBSD jails that can be used to contain applications. I'm not sure about the timeline, but I think jails came first. Sun engineers wanted to achieve the same so they ported the same technology over to Solaris.
  - StreamBright 10y ago
    Oracle is the company where great projects go and die.
    - adictator 10y ago
      Oracle is just death & evil incarnate.
  - baq 10y ago
    joyent has done some rather dark magic to run Linux containers on an illumos/solaris/smartos/whatever-they-call-it cluster; e.g. https://docs.joyent.com/public-cloud/api-access/docker https://docs.joyent.com/public-cloud/api-access/docker
- shykes 10y ago
  > I am wondering what people like Brendand Gregg think about docker. He will be speaking at Dockercon in April, so I suppose we'll find out soon enough.
mdekkers 10y ago
Every three months or so I tell myself "right,get your finger out, and do something production quality with containers" and come to the same conclusion this guy did.
- dastbe 10y ago
  production quality as in "in production" or as in "something that i will actually use in my day to day"?
  - mdekkers 10y ago
    production quality as in "I will be happy to run my clients' stacks on this, and won't worry about being woken up in the middle of the night by pagerduty" - I have this right now, and although there are more things I want, the pricetag is too high to justify. I have been doing VM (and container) work since a one of the fist versions of VMWare blew my mind, but given the fact that things work well for me right now, there is no pressing problem to solve. It's a cool way to do things, and I am fully on board with the ideas behind it, but right now it simply doesn't solve any problems I have, and only introduces a ton of new ones. There is no payback for the hassle...
- diminoten 10y ago
  My latest foray into bleeding edge ended when I found out AWS Lambda doesn't support python3. :(
- cdelsolar 10y ago
  Have you tried? Make some Dockerfiles, upload to ECR, deploy to an ECS cluster. Hooray, no more dealing with dependency hell and config management.
  - mdekkers 10y ago
    yeah, I tried....
  - StreamBright 10y ago
sandGorgon 10y ago
TLDR of the first 60% of the blog post - "I cannot comment on the LTS 16 as I do not use it. It’s the only distribution to have Overlay2 and ZFS available, that gives some more options to be tried and maybe find something working?" Second part TLDR - "If you are locked-in on Docker and running on AWS, your only salvation might be to let AWS handles it for you.". Use Docker-For-AWS that sets up Cloudformation + Swarm mode. Google offers containers as a service, but more importantly, as confirmed by internal sources, their offering is 100% NOT Dockerized. That is a huge label of quality: Containers without docker. Actually, this is deliberate. Containerd. For example, rkt support on GKE is "coming soon" . https://www.mail-archive.com/google-containers@googlegroups.com/msg00417.html https://www.mail-archive.com/google-containers@googlegroups....
apeace 10y ago
I have to agree when it comes to the business perspective. I can't imagine how Docker is going to survive financially. I use Docker at work, and I 1) don't have any loyalty to their brand and 2) try as hard as I can to abstract away their specific APIs. For example, we use Convox [0] to deploy containers to AWS. I could care less what Convox and/or AWS are doing under the hood. They could switch out Docker for rkt under my feet, and I probably wouldn't even notice. It is kind of like POSIX to me. My apps are designed to run in a POSIX environment, not specifically CentOS or Debian. And just like it's easy for a new Linux distro to come along, give me POSIX, and give me some other shiny features I like, it will be easy for any competitor to come along and replace the Docker interfaces I use. [0] https://convox.com/ https://convox.com/
gtaylor 10y ago
I get the sense that this person wrote a sarcastic, vaguely entertaining piece that drew lots of views. Hoping for lightning to strike twice, they did this again because who doesn't love traffic? That seems to be the entire motivation behind this post, from what I can tell. There's just not a ton of substantiated content here. It's mostly really lousy anecdata like: > Sadly, I am not aware of any serious companies than run on Ubuntu. I am troubled by the direction of Docker and there are serious issues, some of which were raised in this post. However, whatever signal there is in this post is lost in a sea of noisy ranting. My advice to anyone who doesn't already have strong opinions: * Complement any reading that you do with your own research. * Don't try to invent your own container orchestration system. * If you aren't sure how to best do something, ask someone! And for the love of everything holy: * Don't run stateful systems on Docker if you can't handle failure or data loss! Docker and orchestration systems like Kubernetes can be an excellent pairing. It's going to require research, a change in how you develop, build, test, and deploy systems, and a gradual building of operational experience. It will not be a quick process, and it's not for every org. But for some orgs and usage cases, it's an excellent way to go!
- awinder 10y ago
  I'm not sure I've personally been able to figure out if it was sarcasm or willful ignorance, but yeah, this article annoyed the pants off me just as much as the first in the series. The fact that it's now worked twice to hit the front page is also a little unsettling, because apparently one of the best ways to reach people on HN now is to just post a bunch of misinformed FUD (with !!Attitude!!) and jump on board the rocket ship!
bitwalker 10y ago
> CoreOS is an operating that can only run Docker and is exclusively intended to run Docker. This shows an astonishing level of ignorance for someone who claims to have done their research. > First, the main benefit of Docker is to unify dev and production. Having a separate OS in production only for containers totally ruins this point. What? This makes no sense. Your images will be the same between dev and prod, even if the host running the containers is different - which is really the whole point - if you build and run an image in dev, it should run identically in prod. > If you like playing with fire, it looks like that’s the OS of choice. We spent the last year+ running containers on Centos7 with no problems from the OS. Whatever issues we did encounter were either transient bugs with Docker or our own configuration. Perhaps we got super lucky, but we were running 120+ containers on 12 hosts, so I would've expected at least some evidence of significant problems within that timeframe if it were really such a risky setup. > It’s not possible to build a stable product on a broken core, yet both Pivotal and RedHat are trying. We've been running OpenShift Origin since March of last year, it's been very stable during that time - the few issues we did encounter were due to our own mistakes, and were usually fixed just by changing some configuration and restarting the host. While there are undoubtedly problems with Docker, and likely many of the issues you brought up are very real, there are many teams like mine that use it successfully, and painlessly. Docker isn't the tire fire you want to make it out to be.
- Veratyr 10y ago
  > What? This makes no sense. Your images will be the same between dev and prod, even if the host running the containers is different - which is really the whole point - if you build and run an image in dev, it should run identically in prod. I think the author is saying that while this is the point, it's not the reality. As it is right now, things can run absolutely fine in dev and then due to prod running a different operating system, things break.
- StreamBright 10y ago
  >> 120+ containers on 12 hosts it is virtually irrelevant what you use to run and operate this sort of cluster. it gets tricky when you pass 100 nodes and even trickier when you get to 1000+ nodes in a non-linear fashion. Docker certainly has issues that are pretty severe when you are in the financial space and not a big deal if you are running a popular blog like medium.com for example.
dekhn 10y ago
The claim that GKE doesn't use Docker internally is false. You can ssh to the master and see all the docker containers.
- crb 10y ago
  You can't ssh to the master with GKE, but you can check the nodes (as mentioned at https://news.ycombinator.com/item?id=13715358. https://news.ycombinator.com/item?id=13715358.)
  - dekhn 10y ago
    Sorry, yes. You used to be able to ssh to the master.
random3 10y ago
I think Docker was and still is a great way to popularize the concept of containers along with the smarter tooling around it, like resource schedulers. By the time we adopted Docker (because it was popular and realized it's a good vehicle to push some concepts across organizations) we knew we're going to use it as a package system in a distributed scheduling environment (based on Mesos, Zookeeper, HDFS) and that it may get replaced later on. However, it took longer for Docker Inc. and the community to depart from the original ideas (initially very developer task centric, including trying to figure out how to store data inside containers) and figure how to enable scheduling, service discovery, health checking etc. I think Kubernetes did a good job popularizing better practices. IMO two of the few systems that were sound were Mesos and Kubernetes (and their roots are somehow interleaved) and neither puts the Docker as the central piece, nor the container, which is just a building block to achieve the actual goals of running distributed, highly available jobs and services efficiently in a shared environment.
stevecalifornia 10y ago
This 'Hitler uses Docker' video seems to be a pretty similar viewpoint on Docker and the world. https://www.youtube.com/watch?v=PivpCKEiQOQ https://www.youtube.com/watch?v=PivpCKEiQOQ
- kordless 10y ago
  Don't cry, you can run bash on Windows 10 now!
  - jhasse 10y ago
    You could always run bash on Windows using Cygwin or MSYS before.
    - nunez 10y ago
      LXSS (Bash on Ubuntu on Windows) is an honest-to-goodness installation of Ubuntu running on top of Windows using the Linux subsystem. Cygwin/MSYS are more limited in comparison (or at least I have found them to be so).
      - jhasse 10y ago
        Have you tried MSYS2?
necklace 10y ago
Yikes, I could never work in aerospace.
dman 10y ago
Can anyone comment on whether its safe to run ceph inside docker?
_Codemonkeyism 10y ago
While other people just use Docker to serve hundreds of thousands of users (mid size startup) and make money.
- StreamBright 10y ago
  Yeah like running a CI job for customers. Absolutely great with Docker, nobody cares about long term reliability. Container crashes? Now worries we spin up 10 more and re-start the CI job. Perfect match for Docker. Running financial infra is quite the opposite. Long term reliable processes, not much space left for failures and mishaps, especially not in the lowest level of your infrastructure. What is the benefit for using Docker for this sort of services? Almost zero.
jhasse 10y ago
What about Fedora?
parzivalm 10y ago
The author says that the AWS ECS AMI they provide is based on ubuntu but it is an amazon linux AMI which is based on centos/rhel.
erikb 10y ago
Lot of memes, but a little more facts, statistics, links to bug reports etc would have been nice. It may all be true, but how can I decide if I believe it or not? It's not that I know you, dear author, and therefore could trust your personal judgement. Make your point a little more clearly. Please.

170 points by smackay 10y ago

> I am moderately confident that there is no-one on the planet using Docker seriously AND successfully AND without major hassle. Though the tone of this article is very negative, the conclusion is interesting to me as someone that's considered Docker but hasn't done much with it beyond experimenting locally. I'm assuming if there's any forum that has users that can speak to their use of Docker in production, this would be the place.

danielbln 10y ago
We've been running docker in production since late 2013 (data science SaaS). We had some growing pains in the beginning, but since then it has been smooth sailing and a tremendous help in getting our services deployed reliably. However, we have always been very conservative when it comes to upgrading docker and have our own custom glue in place. Still, the statement that "no-one on the planet using Docker seriously AND successfully AND without major hassle" seems majorly hyperbolic.
- StreamBright 10y ago
  Size of your infra would he helpful.
  - danielbln 10y ago
    Pretty benign, 15 beastly bare metal machines in coloction and a bunch of VMs in AWS. A few database servers in both.
n0us 10y ago
Our experience is that it has made running servers locally a hassle because Docker for mac sucks beyond belief but it has been awesome in production. For us the real positive is Kubernetes and tbh docker would be useless without it. I really could care less about docker other than that the Dockerfile format is easy enough to use. We went through the gammut of swarm etc. and were never able to get any of them to work in a meaningful way.
fnbr 10y ago
We're technically using it in production for my site, www.bugdedupe.com, but we're still in beta, so we haven't experienced heavy usage. We've been using Kubernetes to deploy Docker containers on Google Container Engine, and while there have been a few issues due to Docker/Kubernetes (namely, getting the containers to expose localhost to each other, and to expose themselves to the world), the issues that we've had so far have been issues that we would have been bit by eventually. Namely, if we hadn't been forced to deal with the issues now, we would have been screwed later. There have been some weird bugs due to the internal environment that containers use (we had extremely slow DNS lookups that caused our request times to shoot up to 9s each). These issues have been transient though, so it's not clear that it's Dockers fault, or if we're making mistakes in our code. Docker has made our deployment much easier. You just build & push your container, and you instantly have a versioned deployable instance of your code. Kubernetes makes it extremely easy to rollout or rollback containers, so I have nothing but good things to say about containers.
- StreamBright 10y ago
  Size of your infra would he helpful to know.
  - ownagefool 10y ago
    I don't know the above poster but we run over 4500 docker containers on AWS distributed using kubernetes. Docker has indeed caused us a problem or two in the past, but actually it's largely smooth sailing and most of our problems are with the app layers and both docker and kubernetes have largely made our lives much easier. A quick skim would suggests that sits on about 8TB of Ram and 1000vCPUs spread over 100 machines. It's not the biggest stack in the world, but it's doing a good job as our little internal PaaS. We have some non-aws versions too, but I don't have metrics on those right now.
- gregmac 10y ago
  An interesting comment also from the article: > Google offers containers as a service, but more importantly, as confirmed by internal sources, their offering is 100% NOT Dockerized. > Google merely exposes a Docker interface, all the containers are run on internal google containerization technologies, that cannot possibly suffer from all the Docker implementation flaws.

callumjones 10y ago

> As confirmed by internal sources, they experienced massive troubles to get Docker working in any decent condition I call B.S. on this. Amazon wouldn't have spent so much effort on ECS if this was true.

user5994461 10y ago
Amazon employee confirming the article: https://news.ycombinator.com/item?id=12364522 https://news.ycombinator.com/item?id=12364522
- StreamBright 10y ago
  Details matter at Amazon scale as they say. I have used Docker ever since it was beta, it never made it to our production systems though. LXD looks pretty promising, unprivileged containers, no container service and so on. When I have more time I will investigate how to roll it on CentOS.
yebyen 10y ago
Has Amazon spent a lot of effort on ECS? I am totally ignorant here, but the people who I consider more knowledgeable about AWS things have said in a nutshell that "in 2015 the things I heard about ECS were not good things, and I have basically not heard any new things since then." Basically stating that ECS was Amazon's attempt to plant a flag in the container-space and that it was half-hearted, not done with the rigor of solutions like k8s that had additional advantages of also being cloud-agnostic, and finally that ECS was not a solution that anyone they knew was recommending or would recommend.
- coding123 10y ago
  Don't know about ECS but ECR (the registry) is a freaking godsend.
- abbyaws 10y ago
  hey yebyen- i was one of the very first beta users of ECS, and I now work for AWS. first off, sounds like maybe we're not doing the best job we can be with educating users on ECS- that's on us, and we'll focus on improving. to dig into your actual question: when ECS launched, the goal was to do a smaller number of things really well, and then listen to the community on what _they_ wanted to see from a container management platform, and grow accordingly. I've seen a number of significant improvements over the last year or so. Off the top of my head: ECR (thanks @coding123), task placement policies and strategies to give developers more control over how they place tasks and use resources, IAM roles for tasks, event stream for cloudwatch events, service level autoscaling, ALB support, and a number of smaller configuration changes, like multiple network modes, and out-of-the-box support for and awslogs driver. also discovered while writing a workshop the other day that there is a pretty sweet first-run wizard for users just trying out ECS for the first time. in any case, a couple of main takeaways here: i'm seeing a focus on adding features and services that reduce the operational work for developers- let AWS worry about scaling and managing your cluster infrastructure, and you can focus on building cool stuff. beyond that, though, developers have asked for more control, flexibility, and extensibility, and i think ECS is working on delivering that: the cloudwatch event stream can be consumed by other services, the blox open source project (and the already opensourced ecs-agent) let you build custom schedulers/functionality on top of what AWS offers, placement policies let you customize how ECS consumes your cluster resources. would love to know where you get your news and why you haven't heard much about ECS, so we can make sure we fix it. if you want to talk more, you can also DM me

Artemis2 10y ago

> Google merely exposes a Docker interface, all the containers are run on internal google containerization technologies, that cannot possibly suffer from all the Docker implementation flaws. Google running their own containerization tech with a Docker interface seems a bit far-fetched given the level of integration of Kubernetes with Docker. That's totally possible though, I'd like to read more about it.

raesene6 10y ago
It is far-fetched 'cause it's not AFAIK true. My experience for this is having set up a GKE Kubernetes cluster last night, it was definitely running Docker.
daenney 10y ago
From one of the GKE folks: https://news.ycombinator.com/item?id=13715358 https://news.ycombinator.com/item?id=13715358 It's not true.

crb 10y ago

It's great that hftguy thinks Google Container Engine is stable (I work on it) but I'm sorry to say it's very easy to prove that is, in fact, running Docker on the nodes. You can just SSH into one and see for yourself. Kubernetes was built from the ground up to orchestrate Docker. CoreOS did a lot of work to make it possible to trade rkt in for Docker's engine, and the cri (Container Runtime Interface) is now generalising that so that there is a clear abstraction between the kubelet and the engines it orchestrates. Read about it here: http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html http://blog.kubernetes.io/2016/12/container-runtime-interfac... If you want to do things that are different to what we provide support for on GKE as a Managed Service (tm), you're able to run your own Kubernetes clusters on GCE. (We do let you run a Kubernetes alpha version, but only on non-supported clusters that self-destruct after 30 days.

Draiken 10y ago
We very recently moved from some bare-metal pet machines into Google GKE and couldn't be happier. Honestly the hardest thing is keeping up with how fast Kubernetes evolves and gets better and better. The same goes for all Google services (pubsub, bigquery, etc) We started the migration on Kubernetes 1.1 and are now live on 1.5.1 Even using it for things we probably shouldn't (old stateful applications) without a single problem. At least no Docker related problems. I don't know, this article seems to be very presumptuous. A lot of bold claims, little backing and, as you state, some pretty false claims. TBH I don't know why it's among the top of HN
deleted 10y ago
[deleted]
SEJeff 10y ago
Is it eventually going to run rkt/cri-o/lmctfy?
- crb 10y ago
  If you're a customer and you're interested in either of the first two, please ping me an e-mail. GitHub's lmctfy README states "We are not actively developing lmctfy further and have moved our efforts to libcontainer."

mwpmaybe 10y ago

I am neither a Docker expert nor evangelist, and I have my own gripes and frustrations with it, but this article is full of misinformation and FUD. To wit: > CoreOS is an operating [system] that can only run Docker and is exclusively intended to run Docker. No, it isn't. (Maybe if you substitute "Docker" with "containers.") > First, the main benefit of Docker is to unify dev and production. Having a separate OS in production only for containers totally ruins this point. No, it doesn't. Completely the opposite, in fact: Docker makes it possible to not care about OS differences between dev, test, and prod. > Docker on Debian is major no-go > Docker is 100% guaranteed suicide on Debian 8 and it’s been since the inception of Docker a few years ago > Debian froze the kernel to a version that doesn’t support anything Docker needs and the few components that are present are rigged with bugs. rubs temples It's Linux. It's Debian. You can run any kernel you want. There are plenty of repositories out there with binary kernels, including Debian's own back-ported 4.9 kernel, or you can build one from source. Because, you know, open source. I've run many instances of Jessie on Linode with their 4.8 and 4.9 kernels and it's no problem. Hundreds of companies are running Docker in production on Debian and Debian-derived systems. > I am not aware of any serious companies than run on Ubuntu. Just because you are not aware does not mean they do not exist. How about Netflix, Snapchat, Dropbox, Uber, and Tesla for starters? > I cannot comment on the LTS 16 as I do not use it. It's been out since last April. If you're serious about this survey, this is a no-brainer. > I received quite a few comments and unfriendly emails of people saying to “just” use the latest Ubuntu beta What? No. Just use the latest LTS. > I am moderately confident that there is no-one on the planet using Docker seriously AND successfully AND without major hassle. LOL

hmottestad 10y ago
Currently using docker with a Norwegian government organisation. We are very happy with it. Been running production since December. And yes, they run on Ubuntu.
deleted 10y ago
[deleted]
mdekkers 10y ago
It's Linux. It's Debian. You can run any kernel you want. The guy works in finance, not an area where you can typically say "oh, no worries, we'll just push out this distro with a non-standard kernel." Enterprise IT, and specifically finance (and healthcare) have different requirements. Currently working with a client where not only the full technology stack has to be certified by accountable vendors, but all the internal documentation and customer facing content has to pass strict legal reviews. "Here is Debian with a custom kernel" won't fly. Nether will "Here is Debian" - Ubuntu LTS will barely pass the line, and that will be after a whole lot of timewasting and ass-covering from many people involved.

kordless 10y ago

> CoreOS is an operating that can only run Docker and is exclusively intended to run Docker. This is a patently false statement and needs to be revised, especially given: > I will not comment on it. P.S. I'm getting downvoted for speaking the truth, no matter how pedantic it may seem. CoreOS can run rkt containers and is not exclusively intended to run Docker. One possible reason is because Docker is the 800 lb. gorilla in the room and having options is always good. P.P.S. Gotta love the Streisand Effect. :)

yebyen 10y ago
You're not wrong, CoreOS can also run arbitrary go binaries (or anything else that can be statically linked) rigged up into systemd or fleet units, without any containers at all.
wrs 10y ago
It is an especially silly statement considering that CoreOS had a much-publicized falling-out with Docker and created rkt to replace it.

merb 10y ago

the problem is, that the docker (community) or rather the company behind it, focuses on too much, instead of just doing one thing and doing it great. no they need to reimplement the world and create their own kubernetes (or some sort of it). their new website is less accessible as it was in the old version, etc... and all the new projects and people. and the various ways to configure docker. why does a simple container engine needs to have so many parameters to configure it? the network experience is still not as good as it could be and docker is still way more complicated to setup than a simple aws and some AWS AMIs.

holydude 10y ago
It is simple imho. Docker turned into an enterprise. Attracted a lot of users (hype) and is trying to bank on it
coding123 10y ago
I'd like to see Rkt/Coreos folks start showing how their stuff IS accessible than keep spewing this flamewar crap on HN. I've had nothing but excellent comments from the Kube developers, but every time a Rkt dev or fan comes on here, this is the crap we get.
- jzelinskie 10y ago
  I think you might be conflating pot stirrers with actual rkt devs. CoreOS (along with many other companies) are the Kubernetes developers. FWIW, there was drama with the initial rkt announcement, but we've actually seen docker adopt most of the original criticisms and that should be applauded for both CoreOS and Docker.

deleted 10y ago

[deleted]

holydude 10y ago

I still think that Solaris zones are / were a superior technology. Sadly it died with the demise of Solaris as an operating system. Still in use in some "must have enterprise" types of companies. I would not run anything serious on docker (definitely not yet) if I would not be heavily invested in understanding and contributing to the technology (docker codebase + related). Docker is a nice to have but absolutely abstracts too much and takes control from you. I am wondering what people like Brendand Gregg think about docker.

marios 10y ago
Solaris within Oracle is as good as dead from the latest reports. However, Illumos is a Solaris descendant that is very much alive. It has all the good systems stuff that came out of Sun (zones, ZFS, DTrace) and is backed by Joyent (now part of Samsung). There's also FreeBSD jails that can be used to contain applications. I'm not sure about the timeline, but I think jails came first. Sun engineers wanted to achieve the same so they ported the same technology over to Solaris.
- StreamBright 10y ago
  Oracle is the company where great projects go and die.
  - adictator 10y ago
    Oracle is just death & evil incarnate.
- baq 10y ago
  joyent has done some rather dark magic to run Linux containers on an illumos/solaris/smartos/whatever-they-call-it cluster; e.g. https://docs.joyent.com/public-cloud/api-access/docker https://docs.joyent.com/public-cloud/api-access/docker
shykes 10y ago
> I am wondering what people like Brendand Gregg think about docker. He will be speaking at Dockercon in April, so I suppose we'll find out soon enough.

mdekkers 10y ago

Every three months or so I tell myself "right,get your finger out, and do something production quality with containers" and come to the same conclusion this guy did.

dastbe 10y ago
production quality as in "in production" or as in "something that i will actually use in my day to day"?
- mdekkers 10y ago
  production quality as in "I will be happy to run my clients' stacks on this, and won't worry about being woken up in the middle of the night by pagerduty" - I have this right now, and although there are more things I want, the pricetag is too high to justify. I have been doing VM (and container) work since a one of the fist versions of VMWare blew my mind, but given the fact that things work well for me right now, there is no pressing problem to solve. It's a cool way to do things, and I am fully on board with the ideas behind it, but right now it simply doesn't solve any problems I have, and only introduces a ton of new ones. There is no payback for the hassle...
diminoten 10y ago
My latest foray into bleeding edge ended when I found out AWS Lambda doesn't support python3. :(
cdelsolar 10y ago
Have you tried? Make some Dockerfiles, upload to ECR, deploy to an ECS cluster. Hooray, no more dealing with dependency hell and config management.
- mdekkers 10y ago
  yeah, I tried....
- StreamBright 10y ago

sandGorgon 10y ago

TLDR of the first 60% of the blog post - "I cannot comment on the LTS 16 as I do not use it. It’s the only distribution to have Overlay2 and ZFS available, that gives some more options to be tried and maybe find something working?" Second part TLDR - "If you are locked-in on Docker and running on AWS, your only salvation might be to let AWS handles it for you.". Use Docker-For-AWS that sets up Cloudformation + Swarm mode. Google offers containers as a service, but more importantly, as confirmed by internal sources, their offering is 100% NOT Dockerized. That is a huge label of quality: Containers without docker. Actually, this is deliberate. Containerd. For example, rkt support on GKE is "coming soon" . https://www.mail-archive.com/google-containers@googlegroups.com/msg00417.html https://www.mail-archive.com/google-containers@googlegroups....

apeace 10y ago

I have to agree when it comes to the business perspective. I can't imagine how Docker is going to survive financially. I use Docker at work, and I 1) don't have any loyalty to their brand and 2) try as hard as I can to abstract away their specific APIs. For example, we use Convox [0] to deploy containers to AWS. I could care less what Convox and/or AWS are doing under the hood. They could switch out Docker for rkt under my feet, and I probably wouldn't even notice. It is kind of like POSIX to me. My apps are designed to run in a POSIX environment, not specifically CentOS or Debian. And just like it's easy for a new Linux distro to come along, give me POSIX, and give me some other shiny features I like, it will be easy for any competitor to come along and replace the Docker interfaces I use. [0] https://convox.com/ https://convox.com/

gtaylor 10y ago

I get the sense that this person wrote a sarcastic, vaguely entertaining piece that drew lots of views. Hoping for lightning to strike twice, they did this again because who doesn't love traffic? That seems to be the entire motivation behind this post, from what I can tell. There's just not a ton of substantiated content here. It's mostly really lousy anecdata like: > Sadly, I am not aware of any serious companies than run on Ubuntu. I am troubled by the direction of Docker and there are serious issues, some of which were raised in this post. However, whatever signal there is in this post is lost in a sea of noisy ranting. My advice to anyone who doesn't already have strong opinions: * Complement any reading that you do with your own research. * Don't try to invent your own container orchestration system. * If you aren't sure how to best do something, ask someone! And for the love of everything holy: * Don't run stateful systems on Docker if you can't handle failure or data loss! Docker and orchestration systems like Kubernetes can be an excellent pairing. It's going to require research, a change in how you develop, build, test, and deploy systems, and a gradual building of operational experience. It will not be a quick process, and it's not for every org. But for some orgs and usage cases, it's an excellent way to go!

awinder 10y ago
I'm not sure I've personally been able to figure out if it was sarcasm or willful ignorance, but yeah, this article annoyed the pants off me just as much as the first in the series. The fact that it's now worked twice to hit the front page is also a little unsettling, because apparently one of the best ways to reach people on HN now is to just post a bunch of misinformed FUD (with !!Attitude!!) and jump on board the rocket ship!

bitwalker 10y ago

> CoreOS is an operating that can only run Docker and is exclusively intended to run Docker. This shows an astonishing level of ignorance for someone who claims to have done their research. > First, the main benefit of Docker is to unify dev and production. Having a separate OS in production only for containers totally ruins this point. What? This makes no sense. Your images will be the same between dev and prod, even if the host running the containers is different - which is really the whole point - if you build and run an image in dev, it should run identically in prod. > If you like playing with fire, it looks like that’s the OS of choice. We spent the last year+ running containers on Centos7 with no problems from the OS. Whatever issues we did encounter were either transient bugs with Docker or our own configuration. Perhaps we got super lucky, but we were running 120+ containers on 12 hosts, so I would've expected at least some evidence of significant problems within that timeframe if it were really such a risky setup. > It’s not possible to build a stable product on a broken core, yet both Pivotal and RedHat are trying. We've been running OpenShift Origin since March of last year, it's been very stable during that time - the few issues we did encounter were due to our own mistakes, and were usually fixed just by changing some configuration and restarting the host. While there are undoubtedly problems with Docker, and likely many of the issues you brought up are very real, there are many teams like mine that use it successfully, and painlessly. Docker isn't the tire fire you want to make it out to be.

Veratyr 10y ago
> What? This makes no sense. Your images will be the same between dev and prod, even if the host running the containers is different - which is really the whole point - if you build and run an image in dev, it should run identically in prod. I think the author is saying that while this is the point, it's not the reality. As it is right now, things can run absolutely fine in dev and then due to prod running a different operating system, things break.
StreamBright 10y ago
>> 120+ containers on 12 hosts it is virtually irrelevant what you use to run and operate this sort of cluster. it gets tricky when you pass 100 nodes and even trickier when you get to 1000+ nodes in a non-linear fashion. Docker certainly has issues that are pretty severe when you are in the financial space and not a big deal if you are running a popular blog like medium.com for example.

dekhn 10y ago

The claim that GKE doesn't use Docker internally is false. You can ssh to the master and see all the docker containers.

crb 10y ago
You can't ssh to the master with GKE, but you can check the nodes (as mentioned at https://news.ycombinator.com/item?id=13715358. https://news.ycombinator.com/item?id=13715358.)
- dekhn 10y ago
  Sorry, yes. You used to be able to ssh to the master.

random3 10y ago

I think Docker was and still is a great way to popularize the concept of containers along with the smarter tooling around it, like resource schedulers. By the time we adopted Docker (because it was popular and realized it's a good vehicle to push some concepts across organizations) we knew we're going to use it as a package system in a distributed scheduling environment (based on Mesos, Zookeeper, HDFS) and that it may get replaced later on. However, it took longer for Docker Inc. and the community to depart from the original ideas (initially very developer task centric, including trying to figure out how to store data inside containers) and figure how to enable scheduling, service discovery, health checking etc. I think Kubernetes did a good job popularizing better practices. IMO two of the few systems that were sound were Mesos and Kubernetes (and their roots are somehow interleaved) and neither puts the Docker as the central piece, nor the container, which is just a building block to achieve the actual goals of running distributed, highly available jobs and services efficiently in a shared environment.

stevecalifornia 10y ago

This 'Hitler uses Docker' video seems to be a pretty similar viewpoint on Docker and the world. https://www.youtube.com/watch?v=PivpCKEiQOQ https://www.youtube.com/watch?v=PivpCKEiQOQ

kordless 10y ago
Don't cry, you can run bash on Windows 10 now!
- jhasse 10y ago
  You could always run bash on Windows using Cygwin or MSYS before.
  - nunez 10y ago
    LXSS (Bash on Ubuntu on Windows) is an honest-to-goodness installation of Ubuntu running on top of Windows using the Linux subsystem. Cygwin/MSYS are more limited in comparison (or at least I have found them to be so).
    - jhasse 10y ago
      Have you tried MSYS2?

necklace 10y ago

Yikes, I could never work in aerospace.

dman 10y ago

Can anyone comment on whether its safe to run ceph inside docker?

_Codemonkeyism 10y ago

While other people just use Docker to serve hundreds of thousands of users (mid size startup) and make money.

StreamBright 10y ago
Yeah like running a CI job for customers. Absolutely great with Docker, nobody cares about long term reliability. Container crashes? Now worries we spin up 10 more and re-start the CI job. Perfect match for Docker. Running financial infra is quite the opposite. Long term reliable processes, not much space left for failures and mishaps, especially not in the lowest level of your infrastructure. What is the benefit for using Docker for this sort of services? Almost zero.

jhasse 10y ago

What about Fedora?

parzivalm 10y ago

The author says that the AWS ECS AMI they provide is based on ubuntu but it is an amazon linux AMI which is based on centos/rhel.

erikb 10y ago

Lot of memes, but a little more facts, statistics, links to bug reports etc would have been nice. It may all be true, but how can I decide if I believe it or not? It's not that I know you, dear author, and therefore could trust your personal judgement. Make your point a little more clearly. Please.