8 ms·

Documenting our migration to Docker

90 points by djelal 5y ago

DrBazza 5y ago
Worth reading just to "TIL" of a dockerfile linter: https://hadolint.github.io/hadolint/ https://hadolint.github.io/hadolint/
- willejs 5y ago
  +1 for using hadolint! I have been using it for ages, and I highly recommend it. I also sometimes write some Goss tests, and even run Trivy at the final stages of the container build and delete them once done in the same layer. It takes about 10 seconds or so longer to build the container, but its baked in and helps you catch issues before pushing the container.
  - DrBazza 5y ago
    And TIL goss+trivy as well. Not a devops guy, so also useful!
cle 5y ago
> How does Docker handle asynchronous intermittent processes? Not good, actually. It is well known in Docker 101 that thou shall not run cron in the same container as the main process. You can run cron just fine in Docker containers. Whether or not that's a good idea depends on what the cron is doing, what your application is doing, your runtime requirements, etc. Running cron in your container is definitely way simpler than the other options involving container scheduling.
- codeflo 5y ago
  This. The advice to move cron jobs outside has its place, of course. But unless you’re running at massive scale, Docker becomes easier once you stop worrying about what is and isn’t “proper container architecture”, and just remember that it’s Linux.
  - roenxi 5y ago
    A voice in support - it is more important to solve the problems that you do have than the problems that the Docker community thinks you might have. But it is important to at least show a little humility and recognising what problems they are solving even.
  - marcosdumay 5y ago
    > just remember that it’s Linux Hum... It's some Linux that appears out of nowhere and may go away at any time. This is the reason for most those container best practices. Anyway, this really doesn't translate to "don't", but be sure to keep the transiency in mind.
- folex 5y ago
  I think "1 container = 1 process" is a common misconception. There should be a separation of concerns, but there's no reason to go to extremes where it doesn't make sense. S6-overlay people lay it out pretty neat https://github.com/just-containers/s6-overlay#the-docker-way https://github.com/just-containers/s6-overlay#the-docker-way
  - a_conservative 5y ago
    When docker first arrived, people were confused about how to use it. I remember seeing lots of people putting an ssh daemon in their containers!?! I like the s6-overlay approach, it's pragmatic. I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea?
    - Quekid5 5y ago
      > I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea? The answer is as always: it depends. My rule of thumb here would be: would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db). This applies most of the time. Similarly, if the database itself is an integral part of the application which it doesn't make sense to swap out or have remote... then by all means just include it and tell the user to mount their data volume to /data or whatever. Example: A single-system file indexer.
      - Groxx 5y ago
        >would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db). This can probably be further simplified to "does it run over ports? then probably yes". E.g. you'd separate your application and database, but not your database and its filesystem.
- nonameiguess 5y ago
  The reason for 1 container, 1 process depends upon architecting your application in such a way that it works with it. Ideally, each application thread is stateless and independent of all others, most or all of them don't require graceful shutdown, and you can encapsulate state elsewhere in persistent volumes. If you're just using a single-node container runtime engine, it's not a huge benefit, but it does make fearless restarts a lot simpler when all you have to do is stop and start the container. Think of Erlang style crash-only programming where crashing is the only way to stop a process. When you get into multi-node orchestrators, though, then the benefit becomes much more apparent, especially at a scale where nodes are guaranteed to fail every now and again. Rescheduling to another node is then, again, as simple as launch a new container from the same image on a different node. That happens a lot faster if each container only needs to start one process. It also enables you to practice chaos engineering, which is the practice of intentionally crashing your own containers every few hours. This is a good practice for security, because if an attacker ever gains a foothold in one of your containers, they'll quickly lose it, but also to just force your infrastructure people and application architects to design in such a way that you're guaranteed to be treating servers like cattle instead of pets, because you're constantly killing them and have no choice if you want your application to still work. This ensures what you're deploying is robust and repeatable and doesn't secretly depend on some specific server state an admin achieved at 1 in the morning some random Sunday hopped up on Red Bull that he'll never remember and never be able to recreate.
  - cle 5y ago
    There's nothing intrinsic about crons in the same container that prevent you from rescheduling containers on other nodes, fearlessly restarting, or whacking random containers. What they're doing, how they're handling state, and their lifecycles might. Study your requirements. Whether you run your crons in the same container or a separate container depends on your system's behavior and failure/recovery scenarios. There are complexity tradeoffs--moving the cron scheduling outside of the container suddenly means you have moving parts outside of your container to test, instrument, monitor, and maintain. If you're already doing that, it's not too big of a deal, but if you aren't then I would think twice before adding all this complexity to your architecture, and make sure it's really worth it. (I can't tell from the article which approach works for their specific use cases...I'm pushing back against the general advice to "avoid cron in containers". I would default to using cron, and only move those processes to separate containers if you have a concrete reason to.)
rhacker 5y ago
Someone editorialized the title.
- enlyth 5y ago
  It seems the article was submitted by the author, so I don't see an issue here
tyingq 5y ago
I can see a PHP centric shop not feeling all the upside of containers. The typical PHP model is pretty simplistic, so it's not hard to replicate most of the docker benefits without using containers at all. That is, I can manage and deploy php apps with different php versions, database versions, repo versions, etc, without too much pain already. Because deploying a PHP app is mostly just dropping files into a directory and restarting a web server.
- marcosdumay 5y ago
  > different php versions How do you do that one? Last time I tried (granted, a long time ago), apache-mod-php was hardcoded into a single php version, and you couldn't have different versions of it on apache either. I always classified php as one of the most problematic languages that need containment. There's not only that hard dependency on the version, but the language also has a global configuration storage that, last time I used it, can not be overloaded.
  - tyingq 5y ago
    2 different running PHP-FPM instances. Then, any web server virtual host can choose which one to use. Via SetHandler for Apache.
  - beberlei 5y ago
    You use PHP FPM (Fast CGI) behind Nginx, so you run multiple PHP-FPMs each with their own PHP version. This falls out of stock Debian/Ubuntu already and can be easily extended to even most recent and older PHP versions through PHP Debian Maintainer Ondrejs PPA. Currently PHP 7.0 - 7.4 and 8.0 available via apt-get
  - knownjorbist 5y ago
    There's a number of ways to deal with it now. Having different versions of php installed is the easy part, having the webserver send requests to these is a little more tricky. Using PHP-FPM makes this considerably easier as you can have different versions listening on different unix sockets, for example.
  - 5y ago
chriswarbo 5y ago
> “Whatever is the underlying executing system, Docker can run it with the exact same code, byte by byte.” Having recently delved into containers, I don't like this quote. It's not wrong, but I think it's also misleading. In particular: > Whatever is the underlying execution system Docker can only execute containers on Linux. Indeed, Docker itself is essentially a UI and management tool for Linux kernel features (chroot, bind mounts, etc.). Whilst Docker provides software for other platforms, like macOS, those are just UIs bundled with a VM/hypervisor running Linux; and it's that Linux system which runs the containers. This may seem pedantic, but I think it's important to understand (a) what's actually happening when we run a container, and (b) what we can/can't do with such tools. For example, I was confused why my containers couldn't run macOS binaries: the whole point of containers is that they run inside the host OS, unlike VMs which have their own OS. That's certainly true, but I didn't realise that when running Docker on macOS, the host is still a VM running Linux! > Docker can run it Again, pedantic but important: Docker is a UI and management tool for containers, which are ultimately run by an underlying Linux system (usually via `runc`, or something compatible). I find this important since "Docker" provides much more than just running containers, e.g. it has Dockerfiles, images, registries, etc. If someone just wants to run a container, they don't need Docker; and I've found Docker to be a more complicated and convoluted way of running containers than, say, OCI tools. For example, I recently wrote an AWS Lambda function which needs to run from a custom container (it bundles some tools which are larger than Lambda's 50MB code limit; whilst containers have a 10GB limit). I originally did this with Docker, which required: - Building an 'image' containing the software (I actually did this with Nix, rather than `docker build`, for reasons I'll give below) - "Loading" that image into Docker - "Tagging" the image with the URL of an ECR 'image repository' (we could include this tag during building, but I find this way keeps more distinction between 'building' and 'deploying') - Running `aws ecr get-login | docker login` to "log in" to Docker. This seems ludicrous to me; I hear it's something to do with dockerhub compatibility or somesuch; which is still silly, since we're not using that. - Pushing to ECR using Docker In contrast, I've now switched that project to use OCI images, which only requires the following: - Building an 'image' containing the software. This is just a .tar.gz file, plus some .json files which specify the "EntryPoint", the SHA256 of the .tar.gz, etc. (all easily created with bash + tar + jq) - Uploading the image to ECR. This can be done with `aws ecr` shell commands, but they're quite low-level (e.g. files are uploaded 20MB at a time, which needs a loop) so I did this in Python using boto3. Note that the 'tag' is still needed, but it's just an argument to the 'ecr.put_image' function. > the exact same code, byte by byte This is my main problem with the quote. Whilst it might theoretically be possible to use Docker "properly", it seems to actively encourage awful practices. For example, to run "the exact same code, byte for byte" we would need to actually get those bytes in the first place. Whilst the underlying container is simply a directory (known as a "bundle", which `runc` actually executes), Docker abstracts over such bundles: first we tar them up into "layers", which we then list in a JSON file called an "image", which we then "push" to an "image repository". To run a container, we "pull" its image from a repository, which we reference by a "tag" (essentially a filename). The de facto tag is "latest", which appears in all sorts of documentation and tutorials. Tags may get replaced with new images at any time by another push, which makes the idea of "the exact same code, byte for byte" rather misleading. (Thankfully some repositories, like ECR, allow 'immutable tags', which can never be overwritten. Rather than 'latest', I use hashes of an image's content as its tag; so there's never a conflict when I push a new image.) Next, we can ask how those bytes come into existence in the first place. In the world of Docker, we don't put files into a .tar.gz; instead, we download an entire Linux distribution, then we run a bunch of non-deterministic package management commands like `apt-get update && apt-get install -y foo`, resulting in a massive, unreproducible binary blob. Whilst it can be argued that Docker will run 'the exact same code, byte by byte', it can also be argued that coin tosses are deterministic due to Newton's laws of motion (e.g. a reliable Docker system is illustrated in Figure 1 of http://statweb.stanford.edu/~susan/papers/headswithJ.pdf http://statweb.stanford.edu/~susan/papers/headswithJ.pdf )
denton-scratch 5y ago
...to Docker containers. I don't care to migrate the infrastructure on which I depend to a proprietary product, especially one with a record of sudden, surprising changes. It's perfectly possible to containerize without placing your faith in Docker.
- yjftsjthsd-h 5y ago
  What about Docker is proprietary or subject to surprising changes?
  - nickstinemates 5y ago
    Nothing but FUD here.
j45 5y ago
Docker can be as much work as having to reinstall your servers. Do your infrastructure work up front or later can both be a valid decision that can be different for each project depending on a lot of factors.

90 points by djelal 5y ago

Worth reading just to "TIL" of a dockerfile linter: https://hadolint.github.io/hadolint/ https://hadolint.github.io/hadolint/

willejs 5y ago
+1 for using hadolint! I have been using it for ages, and I highly recommend it. I also sometimes write some Goss tests, and even run Trivy at the final stages of the container build and delete them once done in the same layer. It takes about 10 seconds or so longer to build the container, but its baked in and helps you catch issues before pushing the container.
- DrBazza 5y ago
  And TIL goss+trivy as well. Not a devops guy, so also useful!

cle 5y ago

> How does Docker handle asynchronous intermittent processes? Not good, actually. It is well known in Docker 101 that thou shall not run cron in the same container as the main process. You can run cron just fine in Docker containers. Whether or not that's a good idea depends on what the cron is doing, what your application is doing, your runtime requirements, etc. Running cron in your container is definitely way simpler than the other options involving container scheduling.

codeflo 5y ago
This. The advice to move cron jobs outside has its place, of course. But unless you’re running at massive scale, Docker becomes easier once you stop worrying about what is and isn’t “proper container architecture”, and just remember that it’s Linux.
- roenxi 5y ago
  A voice in support - it is more important to solve the problems that you do have than the problems that the Docker community thinks you might have. But it is important to at least show a little humility and recognising what problems they are solving even.
- marcosdumay 5y ago
  > just remember that it’s Linux Hum... It's some Linux that appears out of nowhere and may go away at any time. This is the reason for most those container best practices. Anyway, this really doesn't translate to "don't", but be sure to keep the transiency in mind.
folex 5y ago
I think "1 container = 1 process" is a common misconception. There should be a separation of concerns, but there's no reason to go to extremes where it doesn't make sense. S6-overlay people lay it out pretty neat https://github.com/just-containers/s6-overlay#the-docker-way https://github.com/just-containers/s6-overlay#the-docker-way
- a_conservative 5y ago
  When docker first arrived, people were confused about how to use it. I remember seeing lots of people putting an ssh daemon in their containers!?! I like the s6-overlay approach, it's pragmatic. I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea?
  - Quekid5 5y ago
    > I haven't been using containers as much lately, but I wonder if s6-overlay's approach could be used to justify including a database into the container with an application. Is that a good idea? The answer is as always: it depends. My rule of thumb here would be: would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db). This applies most of the time. Similarly, if the database itself is an integral part of the application which it doesn't make sense to swap out or have remote... then by all means just include it and tell the user to mount their data volume to /data or whatever. Example: A single-system file indexer.
    - Groxx 5y ago
      >would it ever make sense to configure the application to use a different (or even just remote) database? If so, then the database should have a separate container (when using a local db). This can probably be further simplified to "does it run over ports? then probably yes". E.g. you'd separate your application and database, but not your database and its filesystem.
nonameiguess 5y ago
The reason for 1 container, 1 process depends upon architecting your application in such a way that it works with it. Ideally, each application thread is stateless and independent of all others, most or all of them don't require graceful shutdown, and you can encapsulate state elsewhere in persistent volumes. If you're just using a single-node container runtime engine, it's not a huge benefit, but it does make fearless restarts a lot simpler when all you have to do is stop and start the container. Think of Erlang style crash-only programming where crashing is the only way to stop a process. When you get into multi-node orchestrators, though, then the benefit becomes much more apparent, especially at a scale where nodes are guaranteed to fail every now and again. Rescheduling to another node is then, again, as simple as launch a new container from the same image on a different node. That happens a lot faster if each container only needs to start one process. It also enables you to practice chaos engineering, which is the practice of intentionally crashing your own containers every few hours. This is a good practice for security, because if an attacker ever gains a foothold in one of your containers, they'll quickly lose it, but also to just force your infrastructure people and application architects to design in such a way that you're guaranteed to be treating servers like cattle instead of pets, because you're constantly killing them and have no choice if you want your application to still work. This ensures what you're deploying is robust and repeatable and doesn't secretly depend on some specific server state an admin achieved at 1 in the morning some random Sunday hopped up on Red Bull that he'll never remember and never be able to recreate.
- cle 5y ago
  There's nothing intrinsic about crons in the same container that prevent you from rescheduling containers on other nodes, fearlessly restarting, or whacking random containers. What they're doing, how they're handling state, and their lifecycles might. Study your requirements. Whether you run your crons in the same container or a separate container depends on your system's behavior and failure/recovery scenarios. There are complexity tradeoffs--moving the cron scheduling outside of the container suddenly means you have moving parts outside of your container to test, instrument, monitor, and maintain. If you're already doing that, it's not too big of a deal, but if you aren't then I would think twice before adding all this complexity to your architecture, and make sure it's really worth it. (I can't tell from the article which approach works for their specific use cases...I'm pushing back against the general advice to "avoid cron in containers". I would default to using cron, and only move those processes to separate containers if you have a concrete reason to.)

rhacker 5y ago

Someone editorialized the title.

enlyth 5y ago
It seems the article was submitted by the author, so I don't see an issue here

tyingq 5y ago

I can see a PHP centric shop not feeling all the upside of containers. The typical PHP model is pretty simplistic, so it's not hard to replicate most of the docker benefits without using containers at all. That is, I can manage and deploy php apps with different php versions, database versions, repo versions, etc, without too much pain already. Because deploying a PHP app is mostly just dropping files into a directory and restarting a web server.

marcosdumay 5y ago
> different php versions How do you do that one? Last time I tried (granted, a long time ago), apache-mod-php was hardcoded into a single php version, and you couldn't have different versions of it on apache either. I always classified php as one of the most problematic languages that need containment. There's not only that hard dependency on the version, but the language also has a global configuration storage that, last time I used it, can not be overloaded.
- tyingq 5y ago
  2 different running PHP-FPM instances. Then, any web server virtual host can choose which one to use. Via SetHandler for Apache.
- beberlei 5y ago
  You use PHP FPM (Fast CGI) behind Nginx, so you run multiple PHP-FPMs each with their own PHP version. This falls out of stock Debian/Ubuntu already and can be easily extended to even most recent and older PHP versions through PHP Debian Maintainer Ondrejs PPA. Currently PHP 7.0 - 7.4 and 8.0 available via apt-get
- knownjorbist 5y ago
  There's a number of ways to deal with it now. Having different versions of php installed is the easy part, having the webserver send requests to these is a little more tricky. Using PHP-FPM makes this considerably easier as you can have different versions listening on different unix sockets, for example.
- 5y ago

chriswarbo 5y ago

> “Whatever is the underlying executing system, Docker can run it with the exact same code, byte by byte.” Having recently delved into containers, I don't like this quote. It's not wrong, but I think it's also misleading. In particular: > Whatever is the underlying execution system Docker can only execute containers on Linux. Indeed, Docker itself is essentially a UI and management tool for Linux kernel features (chroot, bind mounts, etc.). Whilst Docker provides software for other platforms, like macOS, those are just UIs bundled with a VM/hypervisor running Linux; and it's that Linux system which runs the containers. This may seem pedantic, but I think it's important to understand (a) what's actually happening when we run a container, and (b) what we can/can't do with such tools. For example, I was confused why my containers couldn't run macOS binaries: the whole point of containers is that they run inside the host OS, unlike VMs which have their own OS. That's certainly true, but I didn't realise that when running Docker on macOS, the host is still a VM running Linux! > Docker can run it Again, pedantic but important: Docker is a UI and management tool for containers, which are ultimately run by an underlying Linux system (usually via `runc`, or something compatible). I find this important since "Docker" provides much more than just running containers, e.g. it has Dockerfiles, images, registries, etc. If someone just wants to run a container, they don't need Docker; and I've found Docker to be a more complicated and convoluted way of running containers than, say, OCI tools. For example, I recently wrote an AWS Lambda function which needs to run from a custom container (it bundles some tools which are larger than Lambda's 50MB code limit; whilst containers have a 10GB limit). I originally did this with Docker, which required: - Building an 'image' containing the software (I actually did this with Nix, rather than `docker build`, for reasons I'll give below) - "Loading" that image into Docker - "Tagging" the image with the URL of an ECR 'image repository' (we could include this tag during building, but I find this way keeps more distinction between 'building' and 'deploying') - Running `aws ecr get-login | docker login` to "log in" to Docker. This seems ludicrous to me; I hear it's something to do with dockerhub compatibility or somesuch; which is still silly, since we're not using that. - Pushing to ECR using Docker In contrast, I've now switched that project to use OCI images, which only requires the following: - Building an 'image' containing the software. This is just a .tar.gz file, plus some .json files which specify the "EntryPoint", the SHA256 of the .tar.gz, etc. (all easily created with bash + tar + jq) - Uploading the image to ECR. This can be done with `aws ecr` shell commands, but they're quite low-level (e.g. files are uploaded 20MB at a time, which needs a loop) so I did this in Python using boto3. Note that the 'tag' is still needed, but it's just an argument to the 'ecr.put_image' function. > the exact same code, byte by byte This is my main problem with the quote. Whilst it might theoretically be possible to use Docker "properly", it seems to actively encourage awful practices. For example, to run "the exact same code, byte for byte" we would need to actually get those bytes in the first place. Whilst the underlying container is simply a directory (known as a "bundle", which `runc` actually executes), Docker abstracts over such bundles: first we tar them up into "layers", which we then list in a JSON file called an "image", which we then "push" to an "image repository". To run a container, we "pull" its image from a repository, which we reference by a "tag" (essentially a filename). The de facto tag is "latest", which appears in all sorts of documentation and tutorials. Tags may get replaced with new images at any time by another push, which makes the idea of "the exact same code, byte for byte" rather misleading. (Thankfully some repositories, like ECR, allow 'immutable tags', which can never be overwritten. Rather than 'latest', I use hashes of an image's content as its tag; so there's never a conflict when I push a new image.) Next, we can ask how those bytes come into existence in the first place. In the world of Docker, we don't put files into a .tar.gz; instead, we download an entire Linux distribution, then we run a bunch of non-deterministic package management commands like `apt-get update && apt-get install -y foo`, resulting in a massive, unreproducible binary blob. Whilst it can be argued that Docker will run 'the exact same code, byte by byte', it can also be argued that coin tosses are deterministic due to Newton's laws of motion (e.g. a reliable Docker system is illustrated in Figure 1 of http://statweb.stanford.edu/~susan/papers/headswithJ.pdf http://statweb.stanford.edu/~susan/papers/headswithJ.pdf )

denton-scratch 5y ago

...to Docker containers. I don't care to migrate the infrastructure on which I depend to a proprietary product, especially one with a record of sudden, surprising changes. It's perfectly possible to containerize without placing your faith in Docker.

yjftsjthsd-h 5y ago
What about Docker is proprietary or subject to surprising changes?
- nickstinemates 5y ago
  Nothing but FUD here.

j45 5y ago

Docker can be as much work as having to reinstall your servers. Do your infrastructure work up front or later can both be a valid decision that can be different for each project depending on a lot of factors.