8 ms·

Hi all. A few clarifications. - The meme that we are adding more and more features into the docker binary is unfounded. Please, please, I ask that before repea

by shykes 12y ago

Hi all. A few clarifications. - The meme that we are adding more and more features into the docker binary is unfounded. Please, please, I ask that before repeating it you do your homework and ask for actual examples. For example 1.4 is coming out next week: it has 500+ commits and basically no new features. It's all bugfixes, refactoring and a general focus on quality. That's going to be the trend from now on - Swarm and Machine are separate binaries. They are not part of the core docker runtime. You can use them in a completely orthogonal way. - Swarm follows the "batteries included but removable" principle. We are not doing all things to all people! There is a default scheduling backend but we want to make it swappable. In fact we had Mesosphere on stage today as part of a partnership to make Mesos a first-class backend. - there is an ongoing proposal to merge compose into the docker binary. I want to let the design discussion play out, but at the moment I'm leaning towards keeping it separate. Now's the time to comment if you care about this - that's how open design works :) Yes, our blog post is buzzwordy and enterprise-sounding. I am torn on this, on the one hand it helps make the project credible in IT departments which associates that kind of language with seriousness. We may find that strange but if it helps with the adoption of Docker, then it benefits every Docker user and that's ok with me. On the other hand, it is definitely not popular on HN and has the opposite connotation of dorky pencil holder suit douchiness. Being from that tribe I share that instinctive reaction. But I also realize it's mostly psychological. I care less about the specific choice of words than the substance. And the substance here is that we launched a lot of new stuff today, and put a lot of effort in keeping the focus on a small, reliable runtime, composable tools which do one thing well, pluggability, open APIs, and playing nice with the ecosystem. Basically everything the community has been worrying about recently.

potto 12y ago
While it is scary when a company takes its initial success in the technology scene and uses it to expand into its verticals, this is not quite the same case as the Microsoft analogy...Docker is open source, after all. I think the backlash that CoreOS started with its Rocket / AppContainer announcement ahead of DockerCon has caused some knee-jerk reactions. Does Docker's expansion into orchestration and clustering endanger CoreOS and maybe other players? Certainly. Is it the same approach? Definitely not. The CoreOS approach to containerization using Docker, and now Rocket, is novel. I haven't considered it for any of the architectures I'm presently building for companies - I prefer to let the operating system do what it does best, and build my services on top using containerization. I was worried about what Docker's plans for clustering entailed leading up to today. I have been using Mesos for the better part of a year, and see it answering many problems that reach beyond the container model. I think people should do as you suggest, and re-evaluate this on two fronts: 1) Swarm is separate from Docker, so we can continue to deploy Docker through Mesosphere Marathon onto Mesos as we've been doing. 2) Because scheduling is pluggable - as the source code validates for us - we can work in Mesos as we see fit. As for the proposal, I vote to keep Compose (and the fig codebase ;) ) separate. It will be much easier to add the hooks necessary to do cluster management between Compose and Swarm without risking regressions to Docker. Also - what has become of the Docker Governance Advisory Board? I think the ratification of a standard / specification would go a long way to solidify trust in the industry (much like CoreOS is trying to do with their App Container proposal). - Paul
- jbeda 12y ago
  The advisory board is just that -- advisory. It has no teeth. All approval of anything goes through Docker.com/Solomon. It is worth reading through the notes from the last/only meeting: https://docs.google.com/document/d/1JfWNzfwptsMgSx82QyWH_Aj0DRKyZKxYQ1aursxNorg/edit https://docs.google.com/document/d/1JfWNzfwptsMgSx82QyWH_Aj0...
  - potto 12y ago
    Have you participated in any other open source projects? All of them with any success in the community (that I can think of) have a monarchy or oligarchy at the head. Anyone may contribute, and a group of committers review and add if the contribution fits the project... who determines what the project and its roadmap? Usually a select few, driven by even fewer or just one. So I don't think it's necessarily unusual that Solomon heads it. As an aside - I've watched headless projects devolve into anarchy... here's a good read from years ago - http://ariya.blogspot.com/2008/03/common-characteristics-of-failed-open.html http://ariya.blogspot.com/2008/03/common-characteristics-of-...
    - jbeda 12y ago
      You are right -- perhaps I'm being unfair. It is Docker's right to run their project as they see fit. I'm glad to see the new projects (swarm, at least) layered on top of Docker. It is good to see those sink/swim on their own. I was uncomfortable when Docker was (last week) going down the path of building everything into their existing binary.
      - 23david 12y ago
        You're more than fair. They made the rules, and I guess they can break the rules. But really, the value of the project lies in the community's continued support and backing of the project. The good thing is that this is open-source Apache-licensed code. Easy enough to fork it if necessary, even if just temporarily while we wait for another project to mature and replace it. Hudson -> Jenkins Mysql -> MariaDB Docker -> ???
jacques_chester 12y ago
> Swarm and Machine are separate binaries. They appear to enjoy first-class support in the docker CLI. Is that right?
- shykes 12y ago
  No. There is a question on how to get machine to cleanly change which machine Docker talks to. The current answer is: standard dynamic config system in docker core, which machine plugs into to change values.
  - jacques_chester 12y ago
    I think I got swarm and machine mixed up. Swarm seems to be tied to the CLI, because the command is "docker run", with additional settings. I think I am skimming too skimmily. Pluggable scheduling and placement is a good move. It all looks like a straightforward operations research problem ... until you discover that you built a SPOF subject to the network fallacies.
    - thaJeztah 12y ago
      > because the command is "docker run", with additional settings The `-m` flag was already present in docker run. Swarm makes use of that flag to decide on which host to put a container. The `constraint:` setting is indeed new, but (afaik) the values behind it are not pre-defined, so if someone develops an alternative clustering-backend they should be able to make use of that for their needs. In any case, Swarm will be pluggable, so other back ends are possible and can be developed by other parties.
      - jacques_chester 12y ago
        Thanks.
doodpants 12y ago
I upvoted you just for using the word "meme" correctly.
- coldtea 12y ago
  The supposedly incorrect uses of the term are also correct -- even with the blessing of the original creator of the word: """In 2013 Dawkins characterized an Internet meme as being a meme deliberately altered by human creativity—distinguished from biological genes and Dawkins' pre-Internet concept of a meme which involved mutation by random change and spreading through accurate replication as in Darwinian selection. Dawkins explained that Internet memes are thus a "hijacking of the original idea," the very idea of a meme having mutated and evolved in this new direction."""
tlrobinson 12y ago
I'd also lean towards keeping compose a separate binary, particularly given the community's concern over Docker's feature-creep, and the fact that it uses a separate configuration file, and doesn't need to be the same binary for any particular reason. Also, I filed an issue on the Github repo for this, but "machine" is already used by a command line tool on OS X and other BSDs. Perhaps "docker-machine" and "docker-compose" are good compromises in both cases.
- hammerdr 12y ago
  Maybe docker could use subcommands a la git. So with docker-machine binary on the path, "docker machine" would just delegate to that. Similarly, if I had a docker-flynn binary, "docker flynn" would just magically work. This means that the main binary is composable from multiple parts and its easy to write plugins for other systems.
  - tlrobinson 12y ago
    I actually just suggested the git plumbing/porcelain model in another comment, which would fit nicely with this suggestion: https://news.ycombinator.com/item?id=8701085 https://news.ycombinator.com/item?id=8701085
  - pekk 12y ago
    Why not just "flynn"? Should I really be typing "docker" before every command for the sake of someone's branding?
    - pferde 12y ago
      Imagine if git, svn, cvs all installed their own "checkout", "clone", "commit", etc. executables. It would become chaos if you had more than one installed. I prefer when various software suites keep their executables in their own namespace, if only for sake of future-proofing, to prevent naming conflicts. There are always symlinks and/or shell aliases if you want to override that locally.
      - 12y ago
ridruejo 12y ago
I think the concern from third-parties is not so much about bloat or whether everything is bundled in the same binary but that by defining "the way" of doing things, you are taking the oxygen out of other ecosystem partners that have alternate approaches for doing so. You are making it pluggable but by having your official implementation it becomes the "de facto" standard. By defining a standard way of orchestrating you make life easier for users that can deploy with confidence the same environment across multiple clouds, etc. BUT the tradeoff is that you alienate another set of parties (providers are commoditized, ecosystem partners offering alternate orchestration tools feel marginalized). Secondarily, there are concerns whether your approach is better than others and that is too early to tell. I am curious to see how all this plays out, lots of things at stake here and a very thick fog of war :)
- nl 12y ago
  Anyone who has used Docker for more than a trivial app sees the need for orchestration. If the requirement is that obvious then it clearly needs to be part of the core.
  - 23david 12y ago
    Many many solutions have been meeting that need just fine and many new ones are still in the works. Adding a new solution will not help things.
    - nl 12y ago
      Yeah it will. At the moment I have to choose between multiple competing solutions. Competition is good when there are strong differentiating features. In the "Orchestrate Docker" field the requirements are so clear after using it for your first project that it isn't at all clear what benefit competition, or solutions outside of the core Docker product suite gives.
- smt88 12y ago
  As someone who has followed Docker since it was part of dotCloud, I can tell you that some standard practices will be great. I've seen dozens of different ways of doing similar things with Docker. By creating these orthogonal solutions, Docker isn't taking the air out of anything. Do these solutions hurt Amazon's proprietary use cases? Or Google's? Or [insert managed Docker platform here]? No. If anything, they just make it possible to use Docker without third parties. Maybe that's scary to third parties, but there's no reason a company shouldn't compete in its own ecosystem as long as it doesn't unfairly punish competition. So if Docker acts like modern-day Microsoft for enterprise, where solutions are sold from many vendors with many mixes of products, then great. If they act like Apple, where they crush and eliminate overlapping services, then I guess we'll all be switching to the CoreOS containers someday...
23david 12y ago
This is a completely new product and really should be differentiated as such via new packages: apt-get install docker.io-platform Just don't put any of this stuff into the docker.io or lxc-docker packages and mostly things will be well in the world.
AndrewHayes 12y ago
People who don't understand how much utility these improvements provide can complain all day. Personally you've delivered exactly what I have needed quickly and, from my use, bug free. My vote (not that it gets counted here) compose should definitely be merged. Great job on all the rest, ignore the haters, they just prove you've made it!
buster 12y ago
Hi shykes, Well, first of all, thanks for Docker, it is a great product, love using it! My only criticism is: I find it scary and kind of shortsighted that docker (and even more so rocket) seem to neglect basic networking features. From what i can tell everything is based on the approach that you write your webapp with docker in mind, but i for one, would like to use my legacy applications with docker and i would like to write applications with full networking in my control. I really, really don't want docker to choose some IP for me. And i really really don't want my apps to talk over some random ports. I want control. So far everything is still based on some random interfaces and ports that are passed around.
- MichaelGG 12y ago
  Bingo! We looked at Docker for VoIP a while ago and it didn't appear like it was a good fit for such applications. To be fair, Azure had the same odd-sighted flaw, although they have a weak workaround with public IPs now.
- teabee89 12y ago
  (disclaimer: I'm one of the Docker maintainers) We hear you and we already started working on addressing just that. The goal is to define a networking interface (a set of functions) that networking plugins would satisfy. Again: "batteries included but removable". There's a lot of work to be done and details to be figured out, but I can assure you that networking is not left out, and is going in the right direction as well. We will send out a proposal on Github as soon as we can. Stay tuned!
  - shykes 12y ago
    FYI today we had a design session on this topic with several members of the ecosystem (Weave, ClusterHQ, Socketplane, Tutum, VMWare) during Dockercon. As soon as there's something worthy of a proposal, we'll open it up for comment on the repo. I'm pretty excited about this.

by shykes 12y ago

potto 12y ago

While it is scary when a company takes its initial success in the technology scene and uses it to expand into its verticals, this is not quite the same case as the Microsoft analogy...Docker is open source, after all. I think the backlash that CoreOS started with its Rocket / AppContainer announcement ahead of DockerCon has caused some knee-jerk reactions. Does Docker's expansion into orchestration and clustering endanger CoreOS and maybe other players? Certainly. Is it the same approach? Definitely not. The CoreOS approach to containerization using Docker, and now Rocket, is novel. I haven't considered it for any of the architectures I'm presently building for companies - I prefer to let the operating system do what it does best, and build my services on top using containerization. I was worried about what Docker's plans for clustering entailed leading up to today. I have been using Mesos for the better part of a year, and see it answering many problems that reach beyond the container model. I think people should do as you suggest, and re-evaluate this on two fronts: 1) Swarm is separate from Docker, so we can continue to deploy Docker through Mesosphere Marathon onto Mesos as we've been doing. 2) Because scheduling is pluggable - as the source code validates for us - we can work in Mesos as we see fit. As for the proposal, I vote to keep Compose (and the fig codebase ;) ) separate. It will be much easier to add the hooks necessary to do cluster management between Compose and Swarm without risking regressions to Docker. Also - what has become of the Docker Governance Advisory Board? I think the ratification of a standard / specification would go a long way to solidify trust in the industry (much like CoreOS is trying to do with their App Container proposal). - Paul

jbeda 12y ago
The advisory board is just that -- advisory. It has no teeth. All approval of anything goes through Docker.com/Solomon. It is worth reading through the notes from the last/only meeting: https://docs.google.com/document/d/1JfWNzfwptsMgSx82QyWH_Aj0DRKyZKxYQ1aursxNorg/edit https://docs.google.com/document/d/1JfWNzfwptsMgSx82QyWH_Aj0...
- potto 12y ago
  Have you participated in any other open source projects? All of them with any success in the community (that I can think of) have a monarchy or oligarchy at the head. Anyone may contribute, and a group of committers review and add if the contribution fits the project... who determines what the project and its roadmap? Usually a select few, driven by even fewer or just one. So I don't think it's necessarily unusual that Solomon heads it. As an aside - I've watched headless projects devolve into anarchy... here's a good read from years ago - http://ariya.blogspot.com/2008/03/common-characteristics-of-failed-open.html http://ariya.blogspot.com/2008/03/common-characteristics-of-...
  - jbeda 12y ago
    You are right -- perhaps I'm being unfair. It is Docker's right to run their project as they see fit. I'm glad to see the new projects (swarm, at least) layered on top of Docker. It is good to see those sink/swim on their own. I was uncomfortable when Docker was (last week) going down the path of building everything into their existing binary.
    - 23david 12y ago
      You're more than fair. They made the rules, and I guess they can break the rules. But really, the value of the project lies in the community's continued support and backing of the project. The good thing is that this is open-source Apache-licensed code. Easy enough to fork it if necessary, even if just temporarily while we wait for another project to mature and replace it. Hudson -> Jenkins Mysql -> MariaDB Docker -> ???

jacques_chester 12y ago

> Swarm and Machine are separate binaries. They appear to enjoy first-class support in the docker CLI. Is that right?

shykes 12y ago
No. There is a question on how to get machine to cleanly change which machine Docker talks to. The current answer is: standard dynamic config system in docker core, which machine plugs into to change values.
- jacques_chester 12y ago
  I think I got swarm and machine mixed up. Swarm seems to be tied to the CLI, because the command is "docker run", with additional settings. I think I am skimming too skimmily. Pluggable scheduling and placement is a good move. It all looks like a straightforward operations research problem ... until you discover that you built a SPOF subject to the network fallacies.
  - thaJeztah 12y ago
    > because the command is "docker run", with additional settings The `-m` flag was already present in docker run. Swarm makes use of that flag to decide on which host to put a container. The `constraint:` setting is indeed new, but (afaik) the values behind it are not pre-defined, so if someone develops an alternative clustering-backend they should be able to make use of that for their needs. In any case, Swarm will be pluggable, so other back ends are possible and can be developed by other parties.
    - jacques_chester 12y ago
      Thanks.

doodpants 12y ago

I upvoted you just for using the word "meme" correctly.

coldtea 12y ago
The supposedly incorrect uses of the term are also correct -- even with the blessing of the original creator of the word: """In 2013 Dawkins characterized an Internet meme as being a meme deliberately altered by human creativity—distinguished from biological genes and Dawkins' pre-Internet concept of a meme which involved mutation by random change and spreading through accurate replication as in Darwinian selection. Dawkins explained that Internet memes are thus a "hijacking of the original idea," the very idea of a meme having mutated and evolved in this new direction."""

tlrobinson 12y ago

I'd also lean towards keeping compose a separate binary, particularly given the community's concern over Docker's feature-creep, and the fact that it uses a separate configuration file, and doesn't need to be the same binary for any particular reason. Also, I filed an issue on the Github repo for this, but "machine" is already used by a command line tool on OS X and other BSDs. Perhaps "docker-machine" and "docker-compose" are good compromises in both cases.

hammerdr 12y ago
Maybe docker could use subcommands a la git. So with docker-machine binary on the path, "docker machine" would just delegate to that. Similarly, if I had a docker-flynn binary, "docker flynn" would just magically work. This means that the main binary is composable from multiple parts and its easy to write plugins for other systems.
- tlrobinson 12y ago
  I actually just suggested the git plumbing/porcelain model in another comment, which would fit nicely with this suggestion: https://news.ycombinator.com/item?id=8701085 https://news.ycombinator.com/item?id=8701085
- pekk 12y ago
  Why not just "flynn"? Should I really be typing "docker" before every command for the sake of someone's branding?
  - pferde 12y ago
    Imagine if git, svn, cvs all installed their own "checkout", "clone", "commit", etc. executables. It would become chaos if you had more than one installed. I prefer when various software suites keep their executables in their own namespace, if only for sake of future-proofing, to prevent naming conflicts. There are always symlinks and/or shell aliases if you want to override that locally.
    - 12y ago

ridruejo 12y ago

I think the concern from third-parties is not so much about bloat or whether everything is bundled in the same binary but that by defining "the way" of doing things, you are taking the oxygen out of other ecosystem partners that have alternate approaches for doing so. You are making it pluggable but by having your official implementation it becomes the "de facto" standard. By defining a standard way of orchestrating you make life easier for users that can deploy with confidence the same environment across multiple clouds, etc. BUT the tradeoff is that you alienate another set of parties (providers are commoditized, ecosystem partners offering alternate orchestration tools feel marginalized). Secondarily, there are concerns whether your approach is better than others and that is too early to tell. I am curious to see how all this plays out, lots of things at stake here and a very thick fog of war :)

nl 12y ago
Anyone who has used Docker for more than a trivial app sees the need for orchestration. If the requirement is that obvious then it clearly needs to be part of the core.
- 23david 12y ago
  Many many solutions have been meeting that need just fine and many new ones are still in the works. Adding a new solution will not help things.
  - nl 12y ago
    Yeah it will. At the moment I have to choose between multiple competing solutions. Competition is good when there are strong differentiating features. In the "Orchestrate Docker" field the requirements are so clear after using it for your first project that it isn't at all clear what benefit competition, or solutions outside of the core Docker product suite gives.
smt88 12y ago
As someone who has followed Docker since it was part of dotCloud, I can tell you that some standard practices will be great. I've seen dozens of different ways of doing similar things with Docker. By creating these orthogonal solutions, Docker isn't taking the air out of anything. Do these solutions hurt Amazon's proprietary use cases? Or Google's? Or [insert managed Docker platform here]? No. If anything, they just make it possible to use Docker without third parties. Maybe that's scary to third parties, but there's no reason a company shouldn't compete in its own ecosystem as long as it doesn't unfairly punish competition. So if Docker acts like modern-day Microsoft for enterprise, where solutions are sold from many vendors with many mixes of products, then great. If they act like Apple, where they crush and eliminate overlapping services, then I guess we'll all be switching to the CoreOS containers someday...

23david 12y ago

This is a completely new product and really should be differentiated as such via new packages: apt-get install docker.io-platform Just don't put any of this stuff into the docker.io or lxc-docker packages and mostly things will be well in the world.

AndrewHayes 12y ago

People who don't understand how much utility these improvements provide can complain all day. Personally you've delivered exactly what I have needed quickly and, from my use, bug free. My vote (not that it gets counted here) compose should definitely be merged. Great job on all the rest, ignore the haters, they just prove you've made it!

buster 12y ago

Hi shykes, Well, first of all, thanks for Docker, it is a great product, love using it! My only criticism is: I find it scary and kind of shortsighted that docker (and even more so rocket) seem to neglect basic networking features. From what i can tell everything is based on the approach that you write your webapp with docker in mind, but i for one, would like to use my legacy applications with docker and i would like to write applications with full networking in my control. I really, really don't want docker to choose some IP for me. And i really really don't want my apps to talk over some random ports. I want control. So far everything is still based on some random interfaces and ports that are passed around.

MichaelGG 12y ago
Bingo! We looked at Docker for VoIP a while ago and it didn't appear like it was a good fit for such applications. To be fair, Azure had the same odd-sighted flaw, although they have a weak workaround with public IPs now.
teabee89 12y ago
(disclaimer: I'm one of the Docker maintainers) We hear you and we already started working on addressing just that. The goal is to define a networking interface (a set of functions) that networking plugins would satisfy. Again: "batteries included but removable". There's a lot of work to be done and details to be figured out, but I can assure you that networking is not left out, and is going in the right direction as well. We will send out a proposal on Github as soon as we can. Stay tuned!
- shykes 12y ago
  FYI today we had a design session on this topic with several members of the ecosystem (Weave, ClusterHQ, Socketplane, Tutum, VMWare) during Dockercon. As soon as there's something worthy of a proposal, we'll open it up for comment on the repo. I'm pretty excited about this.