4 ms·

Interesting benchmarks, ruined for me by a huge experimental no-no: when one of the tests (the Docker delete) didn't produce the result the author wanted, he in

Interesting benchmarks, ruined for me by a huge experimental no-no: when one of the tests (the Docker delete) didn't produce the result the author wanted, he investigated the underlying problem and fixed it. Sure enough, that became the headline 49x speed-up. Where the results fit the narrative, no optimization was done/reported. One thing I'm very interested by is Docker's idea of single-process containers; but whether this can be done with KVM isolation instead of (weaker but cheaper) LXC isolation. One thing I was surprised by from this presentation is that the memory cost of a KVM vs LXC instance was only 185MB vs 45MB. So KVM's stronger isolation might not be as expensive after all...

elliotf 12y ago
Docker does not necessarily have to be single-process containers. See https://phusion.github.io/baseimage-docker/ https://phusion.github.io/baseimage-docker/ as an example of a full-OS docker base image designed to be lightweight. These days, you can run the full OS inside docker[1][2], but you'll likely be running more than you need to. [1] https://github.com/dotcloud/docker/issues/1024 https://github.com/dotcloud/docker/issues/1024 [2] https://github.com/dotcloud/docker/issues/2276 https://github.com/dotcloud/docker/issues/2276
- nickstinemates 12y ago
  I personally think the phusion images are garbage and kind of miss the point. That being said, running init inside of a container is definitely a supported use case.
- justinsb 12y ago
  Absolutely agree, but (at least in my mind) Docker with a full OS really is just LXC. Running a single process inside the container is something that genuinely differentiates Docker from other approaches. It's a powerful idea which to date has been tied to LXC containers, though I think that's a false association: it's just as interesting with KVM or Xen containers.
nickstinemates 12y ago
I have successfully run KVM inside of a container, so there's no reason not to combine them if that makes sense. Or to run a container inside of KVM. The beauty (to me!) of Docker is that it still applies whether you're talking about any linux-based infrastructure. There are just trade-offs to be made in certain cases (like performance/density for very strict isolation)
deleted 12y ago
[deleted]
boden 12y ago
All of the data including both sets of delete / reboot tests are provided in the benchmark presentation. How others decide to create headlines or side news is out of my control. From my POV the results contain all the data points and don't seek to mislead the reader. Also keep in mind there are currently 23 pages of mysql images on docker index right now https://index.docker.io/search?q=mysql https://index.docker.io/search?q=mysql. Choosing an image which has "proper" bootstrapping is sometimes a trial and error process. This process is a bit more cut and dry with hypervisor / VM based images where you get a standard set of init processes out of the box. RE KVM -- I agree KVM is impressive and the low memory overhead is likely a result of KSM (memory de-duplication) which is enabled out of the box on Ubuntu 12.04 with KVM 1.0.
- justinsb 12y ago
  The problem is not that you didn't publish the data. It is that, when collecting it, you chose to fix/re-run some experiments, but not others. The ones you chose to re-run were (presumably) based on your pre-conceived notions of what things "should be". The slippery slope here is that, if you had different notions, perhaps you would have tried to fix different experiments. Perhaps KVM would have been faster if you believed it should be faster and so had used one of the boot-by-cloning-instance patches. Perhaps the memory usage would have been higher if you thought it should be and so turned off memory sharing (and then maybe the CPU load would have been much lower). I appreciate the transparency in admitting that you re-ran the test, and appreciate anyone that tries to provide data for OpenStack. But you shouldn't claim that you're just publishing numbers for people to interpret as they will, when interpretation entered into the production of those results. It's just not valid experimental practice.
  - boden 12y ago
    I understand your perspective and will take your comments under consideration for future efforts or subsequent iterations of the existing benchmark. However let me reiterate the intent of the existing work... The benchmark's purpose is to educate the community on the potential benefits of a LXC based technology (primarily from a Cloudy perspective) by providing a set of data points with full disclosure as to the means / methods used to perform tests and collect results. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. I believe this criteria is met in the current publication and therefore the results will stand on their own. As to test configuration variations such as those stated above (disable KSM, boot by clone, etc.) -- I believe additional data sets collected via variations or other benchmarks are key to obtaining a well rounded understanding of the factors / benefits involved in this space and I encourage the community to create / publish such results. As an example I would like to gauge the impact of the various KVM guest cache modes and docker LXC storage providers which I believe are fair game to test and add as a variation to the existing benchmark. Should you wish to iterate other configurations / requirements you believe are worth exploring I can make note of them for the next round of tests (time permitting). In addition I'm more than happy to document the instructions to setup and perform these tests so that others can run it themselves. I gladly welcome additional tests and data points from the broader community at large.

elliotf 12y ago

Docker does not necessarily have to be single-process containers. See https://phusion.github.io/baseimage-docker/ https://phusion.github.io/baseimage-docker/ as an example of a full-OS docker base image designed to be lightweight. These days, you can run the full OS inside docker[1][2], but you'll likely be running more than you need to. [1] https://github.com/dotcloud/docker/issues/1024 https://github.com/dotcloud/docker/issues/1024 [2] https://github.com/dotcloud/docker/issues/2276 https://github.com/dotcloud/docker/issues/2276

nickstinemates 12y ago
I personally think the phusion images are garbage and kind of miss the point. That being said, running init inside of a container is definitely a supported use case.
justinsb 12y ago
Absolutely agree, but (at least in my mind) Docker with a full OS really is just LXC. Running a single process inside the container is something that genuinely differentiates Docker from other approaches. It's a powerful idea which to date has been tied to LXC containers, though I think that's a false association: it's just as interesting with KVM or Xen containers.

nickstinemates 12y ago

I have successfully run KVM inside of a container, so there's no reason not to combine them if that makes sense. Or to run a container inside of KVM. The beauty (to me!) of Docker is that it still applies whether you're talking about any linux-based infrastructure. There are just trade-offs to be made in certain cases (like performance/density for very strict isolation)

deleted 12y ago

[deleted]

boden 12y ago

All of the data including both sets of delete / reboot tests are provided in the benchmark presentation. How others decide to create headlines or side news is out of my control. From my POV the results contain all the data points and don't seek to mislead the reader. Also keep in mind there are currently 23 pages of mysql images on docker index right now https://index.docker.io/search?q=mysql https://index.docker.io/search?q=mysql. Choosing an image which has "proper" bootstrapping is sometimes a trial and error process. This process is a bit more cut and dry with hypervisor / VM based images where you get a standard set of init processes out of the box. RE KVM -- I agree KVM is impressive and the low memory overhead is likely a result of KSM (memory de-duplication) which is enabled out of the box on Ubuntu 12.04 with KVM 1.0.

justinsb 12y ago
The problem is not that you didn't publish the data. It is that, when collecting it, you chose to fix/re-run some experiments, but not others. The ones you chose to re-run were (presumably) based on your pre-conceived notions of what things "should be". The slippery slope here is that, if you had different notions, perhaps you would have tried to fix different experiments. Perhaps KVM would have been faster if you believed it should be faster and so had used one of the boot-by-cloning-instance patches. Perhaps the memory usage would have been higher if you thought it should be and so turned off memory sharing (and then maybe the CPU load would have been much lower). I appreciate the transparency in admitting that you re-ran the test, and appreciate anyone that tries to provide data for OpenStack. But you shouldn't claim that you're just publishing numbers for people to interpret as they will, when interpretation entered into the production of those results. It's just not valid experimental practice.
- boden 12y ago
  I understand your perspective and will take your comments under consideration for future efforts or subsequent iterations of the existing benchmark. However let me reiterate the intent of the existing work... The benchmark's purpose is to educate the community on the potential benefits of a LXC based technology (primarily from a Cloudy perspective) by providing a set of data points with full disclosure as to the means / methods used to perform tests and collect results. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. I believe this criteria is met in the current publication and therefore the results will stand on their own. As to test configuration variations such as those stated above (disable KSM, boot by clone, etc.) -- I believe additional data sets collected via variations or other benchmarks are key to obtaining a well rounded understanding of the factors / benefits involved in this space and I encourage the community to create / publish such results. As an example I would like to gauge the impact of the various KVM guest cache modes and docker LXC storage providers which I believe are fair game to test and add as a variation to the existing benchmark. Should you wish to iterate other configurations / requirements you believe are worth exploring I can make note of them for the next round of tests (time permitting). In addition I'm more than happy to document the instructions to setup and perform these tests so that others can run it themselves. I gladly welcome additional tests and data points from the broader community at large.