17 ms·

Docker on Linode

187 points by ljoshua 13y ago

odonnellryan 13y ago
Hey! When doing this, do you run the latest unit tests on the source as well? Do they pass? Important: If you're targeting developers, and those devs use Docker, they might contribute to Docker. If they do, they'd need the tests to pass. Currently, on DO, the tests do _not_ pass. Thanks!
- akerl_ 13y ago
  The Linode kernel appears to only fail the AUFS portion, as expected: https://gist.github.com/akerl/8245192 https://gist.github.com/akerl/8245192 I'm not sure how the Docker devs handle non-AUFS-related contributions from contributors whose systems don't have AUFS (whether a failure on any tests means they refuse to consider patches, or whether they consider the tests relevant to the patch).
  - nickstinemates 13y ago
    > I'm not sure how the Docker devs handle non-AUFS-related contributions from contributors whose systems don't have AUFS We kindly point them to our use of libdevmapper, the other storage driver for Docker. And I should add, we should probably refactor those tests to detect storage capabilities like the Engine does.
    - odonnellryan 13y ago
      Don't the tests use the engine for the most part?
    - akerl_ 13y ago
      That's what I figured, and I'm glad to have it confirmed. I've seen enough projects that have a rigid "if your dev environment isn't perfect, your code's no good to us" mindset that I was wary. Also, just wanted to say that the Docker installation process and the rest of the Docker docs have been amazing. Originally, I was looking at writing a quick guide on getting up and running with Docker on Linode, but we decided linking the official docs made more sense considering how comprehensive they are.
      - nickstinemates 13y ago
        Thanks for the kind words, we're really happy to have Linode support!
      - jamtur01 13y ago
        That's awesome to hear! If you have any feedback on the documentation or would like to see other material please reach out to me: james.turnbull@docker.com. Or log a ticket and cc me on GitHub: jamtur01.
  - odonnellryan 13y ago
    Well, I think the answer to your question is complicated. But thanks for that! Are the integration tests not included? No insult, just wondering if they ran. It's tough. Docker integration tests currently fail on stock Ubuntu from what I can tell. Trying to figure that one out...
    - akerl_ 13y ago
      Can you clarify which integration tests you're referring to? I'd be happy to run them and provide the output. I ran through the whole process linked from the Github repo (http://docs.docker.io/en/latest/contributing/devenvironment/ http://docs.docker.io/en/latest/contributing/devenvironment/), though I only gist'd step 5
      - odonnellryan 13y ago
        http://github.com/dotcloud/docker/integration http://github.com/dotcloud/docker/integration should be the link, thank you!
        akerl_ 13y ago
        That 404s for me. I did some googling, and am not able to find any tests outside of the ones linked from the contributions guide EDIT: It looks like the integration tests reside here: https://github.com/dotcloud/docker/tree/master/integration https://github.com/dotcloud/docker/tree/master/integration They're called when you run make test, via this chain: https://github.com/dotcloud/docker/blob/master/Makefile https://github.com/dotcloud/docker/blob/master/Makefile https://github.com/dotcloud/docker/blob/master/hack/make.sh https://github.com/dotcloud/docker/blob/master/hack/make.sh https://github.com/dotcloud/docker/blob/master/hack/make/test-integration https://github.com/dotcloud/docker/blob/master/hack/make/tes... It looks like the integration tests may not be running because the AUFS test failed? I manually adjusted the Makefile to run the integration tests anyways, and here's the result: https://gist.github.com/akerl/8251863 https://gist.github.com/akerl/8251863 It appears to have a very unhappy time, primarily because the memory cgroup is not enabled. That was a deliberate choice because from what I can see there is a performance hit for enabling that (as well as the swap cgroup), and we decided to leave that disabled rather than give all users a performance hit in order to benefit docker users.
  - shykes 13y ago
    All storage drivers are unit-tested, but our integration tests (./integration/ in the repo) are currently tied to a specific host environment: namely, ubuntu 12.04 with an aufs-patched 3.8 kernel, version 0.8 of the lxc scripts, etc). We are working on upgrading the integration test harness to handle a wider matrix of target systems, including those that don't handle aufs. If you are interested in helping us out on this, come say hi on #docker-dev on freenode :) We are always looking for help and love to get new contributors and maintainers up and running.
nivla 13y ago
Does anyone run docker on a production server? Can it be used for better isolation and security? I am new to docker, so apologies if this turns out to be a naive question.
- justincormack 13y ago
  At present you should not view Docker as a significant security enhancement. It makes processes less visible to each other, and you can use it as part of a strategy to isolate network traffic, but the kernel attack surface is unchanged.
  - darkarmani 13y ago
    > At present you should not view Docker as a significant security enhancement. It makes processes less visible to each other, and you can use it as part of a strategy to isolate network traffic, but the kernel attack surface is unchanged How is process isolation not a significant security enhancement?
    - justincormack 13y ago
      Processes already should be running as different uids so they cant actually do anything to each other, so not being in the same namespace makes no real difference. There is slightly better information hiding, you cant see command lines of other processes and other info from /proc but no "significant" enhancement.
      - joevandyk 13y ago
        Processes already should be running as different uids Docker makes that way easier. If I want 3 instances of nginx running for different projects, I don't really want to setup 3 nginx users (nginx1, nginx2, nginx3). With Docker, I just start the container and it's isolated from everything else.
        icebraining 13y ago
seaghost 13y ago
I'm trying to understand why people pushing using of docker in the cloud instead of dedicated server. In the cloud it adds additional unnecessary level of abstraction.
- IgorPartola 13y ago
  Dedicated hardware typically has a large upfront cost. Also, it's a fixed resource, so for some workloads the "cloud" is more flexible (whether that be simple VPS's with quick provisioning a la DigitalOcean or AWS). From what I understand the overhead in performance is negligible, while giving you very fine grained control over your resources. Say you have a web application, a front-end reverse proxy, a database, a cache, a background worker process, and a message queue. Throw all these components into separate containers on a single server while the load is low. Then move the DB container into a separate VPS as your traffic grows. Or try doing an upgrade of the message queue in a brand new container, and if it fails, just don't use it. Sure, you can do the same thing with a VPS, but then you are paying the per-hour fee for it, while 99.99% of its resources will go unused.
- nickstinemates 13y ago
  I don't think anyone is saying use it instead of, at least not here at Docker. I think what we're saying is that in addition to using awesome IaaS services, you can use Docker to solve any number of issues like utilization, deployment, etc., whether they be with your existing providers, on bare metal, or on OpenStack (which can do both.) I personally have a Linode VPS where I continuously run about 30-40 containers powering the various services/demos I create. I'm looking forward to migrating to the new offering.
- fragmede 13y ago
  Would it help to think of docker as a VM with significantly lower overhead? (Yeah, yeah, technically, it's not...) You're already running VMs with 1-service per-VM for the sanity of dev-ops, right? Instead of 1 VM per-service, with docker, you can make 1 larger host, run 1-service per-docker container, and get more performance out of a single machine.
saosebastiao 13y ago
Sorry for being so uninformed, but is this running docker directly on the kernel, a la CoreOS? Or is this running in a VM?
- orthecreedence 13y ago
  The default Linode kernel (for all VMs) was updated/recompiled to support features that Docker needs to run. So it allows you to run Docker in an existing VM (provided you reboot the VM with the new kernel image).
jaequery 13y ago
shouldn't docker pretty much run on any host platform that supports modern linux kernels (lxc)?
- akerl_ 13y ago
  Docker requires some kernel options both for networking (VLANs/etc) and devicemapper (thin provisioning, in particular) that the Linode kernel didn't have enabled before. You could run Docker on a Linode by using your own kernel with PV-GRUB, but we've now flipped the right bits in our default kernel.
SEJeff 13y ago
As a longishtime Linode user, this makes me extremely happy. Keep it coming linode. Not that pv-grub is beyond my linux-foo by any means, but after working on Linux all day professionaly and dealing with $latest_version_of_fedora on my home pcs, I'd rather not play sysadmin too much more on my vps boxes.

187 points by ljoshua 13y ago

Hey! When doing this, do you run the latest unit tests on the source as well? Do they pass? Important: If you're targeting developers, and those devs use Docker, they might contribute to Docker. If they do, they'd need the tests to pass. Currently, on DO, the tests do _not_ pass. Thanks!

akerl_ 13y ago
The Linode kernel appears to only fail the AUFS portion, as expected: https://gist.github.com/akerl/8245192 https://gist.github.com/akerl/8245192 I'm not sure how the Docker devs handle non-AUFS-related contributions from contributors whose systems don't have AUFS (whether a failure on any tests means they refuse to consider patches, or whether they consider the tests relevant to the patch).
- nickstinemates 13y ago
  > I'm not sure how the Docker devs handle non-AUFS-related contributions from contributors whose systems don't have AUFS We kindly point them to our use of libdevmapper, the other storage driver for Docker. And I should add, we should probably refactor those tests to detect storage capabilities like the Engine does.
  - odonnellryan 13y ago
    Don't the tests use the engine for the most part?
  - akerl_ 13y ago
    That's what I figured, and I'm glad to have it confirmed. I've seen enough projects that have a rigid "if your dev environment isn't perfect, your code's no good to us" mindset that I was wary. Also, just wanted to say that the Docker installation process and the rest of the Docker docs have been amazing. Originally, I was looking at writing a quick guide on getting up and running with Docker on Linode, but we decided linking the official docs made more sense considering how comprehensive they are.
    - nickstinemates 13y ago
      Thanks for the kind words, we're really happy to have Linode support!
    - jamtur01 13y ago
      That's awesome to hear! If you have any feedback on the documentation or would like to see other material please reach out to me: james.turnbull@docker.com. Or log a ticket and cc me on GitHub: jamtur01.
- odonnellryan 13y ago
  Well, I think the answer to your question is complicated. But thanks for that! Are the integration tests not included? No insult, just wondering if they ran. It's tough. Docker integration tests currently fail on stock Ubuntu from what I can tell. Trying to figure that one out...
  - akerl_ 13y ago
    Can you clarify which integration tests you're referring to? I'd be happy to run them and provide the output. I ran through the whole process linked from the Github repo (http://docs.docker.io/en/latest/contributing/devenvironment/ http://docs.docker.io/en/latest/contributing/devenvironment/), though I only gist'd step 5
    - odonnellryan 13y ago
      http://github.com/dotcloud/docker/integration http://github.com/dotcloud/docker/integration should be the link, thank you!
      - akerl_ 13y ago
        That 404s for me. I did some googling, and am not able to find any tests outside of the ones linked from the contributions guide EDIT: It looks like the integration tests reside here: https://github.com/dotcloud/docker/tree/master/integration https://github.com/dotcloud/docker/tree/master/integration They're called when you run make test, via this chain: https://github.com/dotcloud/docker/blob/master/Makefile https://github.com/dotcloud/docker/blob/master/Makefile https://github.com/dotcloud/docker/blob/master/hack/make.sh https://github.com/dotcloud/docker/blob/master/hack/make.sh https://github.com/dotcloud/docker/blob/master/hack/make/test-integration https://github.com/dotcloud/docker/blob/master/hack/make/tes... It looks like the integration tests may not be running because the AUFS test failed? I manually adjusted the Makefile to run the integration tests anyways, and here's the result: https://gist.github.com/akerl/8251863 https://gist.github.com/akerl/8251863 It appears to have a very unhappy time, primarily because the memory cgroup is not enabled. That was a deliberate choice because from what I can see there is a performance hit for enabling that (as well as the swap cgroup), and we decided to leave that disabled rather than give all users a performance hit in order to benefit docker users.
- shykes 13y ago
  All storage drivers are unit-tested, but our integration tests (./integration/ in the repo) are currently tied to a specific host environment: namely, ubuntu 12.04 with an aufs-patched 3.8 kernel, version 0.8 of the lxc scripts, etc). We are working on upgrading the integration test harness to handle a wider matrix of target systems, including those that don't handle aufs. If you are interested in helping us out on this, come say hi on #docker-dev on freenode :) We are always looking for help and love to get new contributors and maintainers up and running.

nivla 13y ago

Does anyone run docker on a production server? Can it be used for better isolation and security? I am new to docker, so apologies if this turns out to be a naive question.

justincormack 13y ago
At present you should not view Docker as a significant security enhancement. It makes processes less visible to each other, and you can use it as part of a strategy to isolate network traffic, but the kernel attack surface is unchanged.
- darkarmani 13y ago
  > At present you should not view Docker as a significant security enhancement. It makes processes less visible to each other, and you can use it as part of a strategy to isolate network traffic, but the kernel attack surface is unchanged How is process isolation not a significant security enhancement?
  - justincormack 13y ago
    Processes already should be running as different uids so they cant actually do anything to each other, so not being in the same namespace makes no real difference. There is slightly better information hiding, you cant see command lines of other processes and other info from /proc but no "significant" enhancement.
    - joevandyk 13y ago
      Processes already should be running as different uids Docker makes that way easier. If I want 3 instances of nginx running for different projects, I don't really want to setup 3 nginx users (nginx1, nginx2, nginx3). With Docker, I just start the container and it's isolated from everything else.
      - icebraining 13y ago

seaghost 13y ago

I'm trying to understand why people pushing using of docker in the cloud instead of dedicated server. In the cloud it adds additional unnecessary level of abstraction.

IgorPartola 13y ago
Dedicated hardware typically has a large upfront cost. Also, it's a fixed resource, so for some workloads the "cloud" is more flexible (whether that be simple VPS's with quick provisioning a la DigitalOcean or AWS). From what I understand the overhead in performance is negligible, while giving you very fine grained control over your resources. Say you have a web application, a front-end reverse proxy, a database, a cache, a background worker process, and a message queue. Throw all these components into separate containers on a single server while the load is low. Then move the DB container into a separate VPS as your traffic grows. Or try doing an upgrade of the message queue in a brand new container, and if it fails, just don't use it. Sure, you can do the same thing with a VPS, but then you are paying the per-hour fee for it, while 99.99% of its resources will go unused.
nickstinemates 13y ago
I don't think anyone is saying use it instead of, at least not here at Docker. I think what we're saying is that in addition to using awesome IaaS services, you can use Docker to solve any number of issues like utilization, deployment, etc., whether they be with your existing providers, on bare metal, or on OpenStack (which can do both.) I personally have a Linode VPS where I continuously run about 30-40 containers powering the various services/demos I create. I'm looking forward to migrating to the new offering.
fragmede 13y ago
Would it help to think of docker as a VM with significantly lower overhead? (Yeah, yeah, technically, it's not...) You're already running VMs with 1-service per-VM for the sanity of dev-ops, right? Instead of 1 VM per-service, with docker, you can make 1 larger host, run 1-service per-docker container, and get more performance out of a single machine.

saosebastiao 13y ago

Sorry for being so uninformed, but is this running docker directly on the kernel, a la CoreOS? Or is this running in a VM?

orthecreedence 13y ago
The default Linode kernel (for all VMs) was updated/recompiled to support features that Docker needs to run. So it allows you to run Docker in an existing VM (provided you reboot the VM with the new kernel image).

jaequery 13y ago

shouldn't docker pretty much run on any host platform that supports modern linux kernels (lxc)?

akerl_ 13y ago
Docker requires some kernel options both for networking (VLANs/etc) and devicemapper (thin provisioning, in particular) that the Linode kernel didn't have enabled before. You could run Docker on a Linode by using your own kernel with PV-GRUB, but we've now flipped the right bits in our default kernel.

SEJeff 13y ago

As a longishtime Linode user, this makes me extremely happy. Keep it coming linode. Not that pv-grub is beyond my linux-foo by any means, but after working on Linux all day professionaly and dealing with $latest_version_of_fedora on my home pcs, I'd rather not play sysadmin too much more on my vps boxes.