5 ms·

For me, it is the ultimate in the idea in Continuous Delivery of "build once." I can be very confident that the docker image I build in the first stage of my pi

by grhmc 11y ago

For me, it is the ultimate in the idea in Continuous Delivery of "build once." I can be very confident that the docker image I build in the first stage of my pipeline will operate correctly in production. This is because that identical image was used for unit tests, to integration and functional testing, to the staging environment and finally production. There is no difference than configuration. This is the core that Docker solves, and in such a way that developers can do most of the dependency wrangling for me. I don't even mind Java anymore because the CLASSPATHs can be figured out once, documented in the Dockerfile in a repeatable programatic fashion, and then ignored. In my opinion the rest of it is gravy. Nice tasty gravy, but I don't care so much about the rest at the moment. Edit: As danesparz points out, nobody has mentioned immutable architecture. This is what we do at Clarify.io. See also: https://news.ycombinator.com/item?id=9845255 https://news.ycombinator.com/item?id=9845255

odiroot 11y ago
How do you handle different configurations then? Especially if you need to provide N values (or structured data). Also, how do you manage your containers in production?
- omarforgotpwd 11y ago
  I use environment variables, which can be passed in as arguments when you start the container. In cases where the configuration is complicated I'll use an environment variable to tell the container which redis key / db key to load config from.
- grhmc 11y ago
  Configuration: We use environment variables for anything small-ish. For more complicated configurations, similar to omarforgotpwd, we keep the values (files AND environment variables) in S3 and download them at deploy time. For stage/prod differences we can literally diff the different S3 buckets. Management: We create AMIs using Packer. Packer runs a provisioning tool which downloads tho container and the configuration and sets up the process monitoring. It then builds a machine image, and then we launch new servers.
- mark_round 11y ago
  As others have written, you can either download your config from an external host, or pass in environment variables to your container and generate a configuration based on them. Lots of people write their own scripts to do this; I wrote Tiller (http://github.com/markround/tiller http://github.com/markround/tiller) to standardise this usage pattern. While I have written a plugin to support storing values in a Zookeeper cluster, I tend to prefer to keep things in YAML files inside the container - at least, for small environments that don't change often. It just reduces a dependency on an external service, but I can see how useful it would be in larger, dynamic environments.
- falcolas 11y ago
  Configuration files, made available to containers as a read-only mount via the volume flag. No external network or service dependencies that way. I'm not terribly fond of using environment variables for configuration, personally. That method requires either a startup shim or development work to make your program aware of the variables, and your container manager has to have access to all configuration values for the services it starts up.
magicmu 11y ago
Like OP, I've been wondering exactly what problem Docker is meant to solve -- thank you for this explanation, it makes total sense.
crdoconnor 11y ago
I get this by using vagrant + ansible rather than docker. Easy to spin up or destroy the environment in the same way in a VM, staging server or live environment. I don't really see the point of lightweight virtualization. It provides an illusion of isolation which will likely come crashing down at some probably very inconvenient point (e.g. when you discover a bug caused by a different version of glibc or a different kernel).
- grhmc 11y ago
  The problem is Vagrant + Ansible violates the rule of "build once." I don't care about the isolation for isolation sake, I care about it for the artifact sake.
  - crdoconnor 11y ago
    I don't care so much for this rule. It sounds like a figleaf covering for broken build scripts. I care about tracking down issues before they reach production. Meaning that I want an environment that mirrors production as closely as possible. Meaning heavyweight not lightweight virtualization.
    - grhmc 11y ago
      Agreed about preventing issues getting to production, but that doesn't exclusively mean heavyweight virtualization. It also doesn't "cover up" broken build scripts. Our build scripts get tested a dozen times a day and cannot tolerate half-assed broken build scripts. Our deployment pipeline (after verifying the image is good enough to be deployed) packs the docker image into a machine image along with several other containers. The machine image is then deployed to staging. If the machine image passes staging, it goes to production. If there is an issue which has hit production exclusively (it has happened only a handful of times,) it is simply an issue of rolling back to the previous machine image.
      - crdoconnor 11y ago
        >Agreed about preventing issues getting to production, but that doesn't exclusively mean heavyweight virtualization. Well, it gets you a step closer to accurately mimicking production. >It also doesn't "cover up" broken build scripts. That seems to be what that 'build once' rule is for. If your building process isn't risky, why the need to prohibit running it twice?
0x0 11y ago
Sounds like a lot of work rebuilding and redeploying images every time a security update is available?
- grhmc 11y ago
  It isn't. Our deployment process is very simple, we download the configuration, and we download the container. This is extremely scriptable and repeatable. We have push-button deploys and don't mind rebuilding. That said, we do have a more mature pipeline, so it may be an issue for other people.
jebblue 11y ago
>> I don't even mind Java anymore because the CLASSPATHs Agree and add that Python's paths (forget what they're called) (as well as Java CLASSPATHS) have been a problem for me on occasion too which means Docker would probably help with all these types of path issues.
hueving 11y ago
This was already done at the development shop I last worked at with VM images. Docker didn't do anything in particular to help this except for reducing the image size.
platz 11y ago
of course, you loose the "identical image" as soon as you build again or run the dockerfile.
gaius 11y ago
Unix had this since day 1, you put your app in a file, and run it as a process. The modern day fashion of making an app out of a thousand separate files is just a silly fashion, and Docker is just a way to make this stupid model barely usable.

by grhmc 11y ago

odiroot 11y ago

How do you handle different configurations then? Especially if you need to provide N values (or structured data). Also, how do you manage your containers in production?

omarforgotpwd 11y ago
I use environment variables, which can be passed in as arguments when you start the container. In cases where the configuration is complicated I'll use an environment variable to tell the container which redis key / db key to load config from.
grhmc 11y ago
Configuration: We use environment variables for anything small-ish. For more complicated configurations, similar to omarforgotpwd, we keep the values (files AND environment variables) in S3 and download them at deploy time. For stage/prod differences we can literally diff the different S3 buckets. Management: We create AMIs using Packer. Packer runs a provisioning tool which downloads tho container and the configuration and sets up the process monitoring. It then builds a machine image, and then we launch new servers.
mark_round 11y ago
As others have written, you can either download your config from an external host, or pass in environment variables to your container and generate a configuration based on them. Lots of people write their own scripts to do this; I wrote Tiller (http://github.com/markround/tiller http://github.com/markround/tiller) to standardise this usage pattern. While I have written a plugin to support storing values in a Zookeeper cluster, I tend to prefer to keep things in YAML files inside the container - at least, for small environments that don't change often. It just reduces a dependency on an external service, but I can see how useful it would be in larger, dynamic environments.
falcolas 11y ago
Configuration files, made available to containers as a read-only mount via the volume flag. No external network or service dependencies that way. I'm not terribly fond of using environment variables for configuration, personally. That method requires either a startup shim or development work to make your program aware of the variables, and your container manager has to have access to all configuration values for the services it starts up.

magicmu 11y ago

Like OP, I've been wondering exactly what problem Docker is meant to solve -- thank you for this explanation, it makes total sense.

crdoconnor 11y ago

I get this by using vagrant + ansible rather than docker. Easy to spin up or destroy the environment in the same way in a VM, staging server or live environment. I don't really see the point of lightweight virtualization. It provides an illusion of isolation which will likely come crashing down at some probably very inconvenient point (e.g. when you discover a bug caused by a different version of glibc or a different kernel).

grhmc 11y ago
The problem is Vagrant + Ansible violates the rule of "build once." I don't care about the isolation for isolation sake, I care about it for the artifact sake.
- crdoconnor 11y ago
  I don't care so much for this rule. It sounds like a figleaf covering for broken build scripts. I care about tracking down issues before they reach production. Meaning that I want an environment that mirrors production as closely as possible. Meaning heavyweight not lightweight virtualization.
  - grhmc 11y ago
    Agreed about preventing issues getting to production, but that doesn't exclusively mean heavyweight virtualization. It also doesn't "cover up" broken build scripts. Our build scripts get tested a dozen times a day and cannot tolerate half-assed broken build scripts. Our deployment pipeline (after verifying the image is good enough to be deployed) packs the docker image into a machine image along with several other containers. The machine image is then deployed to staging. If the machine image passes staging, it goes to production. If there is an issue which has hit production exclusively (it has happened only a handful of times,) it is simply an issue of rolling back to the previous machine image.
    - crdoconnor 11y ago
      >Agreed about preventing issues getting to production, but that doesn't exclusively mean heavyweight virtualization. Well, it gets you a step closer to accurately mimicking production. >It also doesn't "cover up" broken build scripts. That seems to be what that 'build once' rule is for. If your building process isn't risky, why the need to prohibit running it twice?

0x0 11y ago

Sounds like a lot of work rebuilding and redeploying images every time a security update is available?

grhmc 11y ago
It isn't. Our deployment process is very simple, we download the configuration, and we download the container. This is extremely scriptable and repeatable. We have push-button deploys and don't mind rebuilding. That said, we do have a more mature pipeline, so it may be an issue for other people.

jebblue 11y ago

>> I don't even mind Java anymore because the CLASSPATHs Agree and add that Python's paths (forget what they're called) (as well as Java CLASSPATHS) have been a problem for me on occasion too which means Docker would probably help with all these types of path issues.

hueving 11y ago

This was already done at the development shop I last worked at with VM images. Docker didn't do anything in particular to help this except for reducing the image size.

platz 11y ago

of course, you loose the "identical image" as soon as you build again or run the dockerfile.

gaius 11y ago

Unix had this since day 1, you put your app in a file, and run it as a process. The modern day fashion of making an app out of a thousand separate files is just a silly fashion, and Docker is just a way to make this stupid model barely usable.