10 ms·

What’s New in Docker 1.13?

149 points by hackerpt 10y ago

chrisfosterelli 10y ago
I'm very excited about the clean-up commands! Docker desperately needs this, it's so frustrating constantly having full disk space due to untagged containers and unbounded volumes.
- geerlingguy 10y ago
  If you're on Mac, it's still a bit of a problem; hopefully the SSD-gobbling Docker.qcow2 file problem[1] is fixed soon! [1] https://github.com/docker/for-mac/issues/371 https://github.com/docker/for-mac/issues/371
  - selvakn 10y ago
    Looks like it is fixed in https://twitter.com/kelseyhightower/status/822237099495550976 https://twitter.com/kelseyhightower/status/82223709949555097.... Im looking for official changelog for confirmation though.
    - iends 10y ago
      Judging by the github issue list there are a lot more issues with Docker.qcow2 now than before. Mostly related to Docker.qcow2 preventing docker from starting. Looks like a bad release. Hopefully 1.13.1 is soon.
      - djs55 10y ago
        (I work on Docker for Mac.) Apologies for the inconvenience this has caused. There was a race condition in a previous release which could allow multiple hypervisor instances to open the Docker.qcow2 simultaneously. Unfortunately this can corrupt the file by allocating the same physical block (cluster) twice, resulting in bad things happening. When this file-locking bug was fixed we also added an integrity check which checks the structure of the Docker.qcow2 on every application launch. For safety the app refuses to start if corruption is detected. I believe that in these cases, the corruption happened in the past and is now being detected since the upgrade. Unfortunately if the app refuses to start it makes it difficult to reach the "Reset to Factory defaults" menu option. The workaround described here https://github.com/docker/for-mac/issues/1159#issuecomment-273899215 https://github.com/docker/for-mac/issues/1159#issuecomment-2... is to remove the qcow2 and restart the app. Unfortunately containers and images will need to be rebuilt. For what it's worth after the integrity check and the locking fix went in, I've not seen any recurrence of this error. Please open an issue if you see any other problems!
  - justincormack 10y ago
    This is now fixed on the stable release, but space is only released when you restart docker at present. Online freeing is being worked on too.
- dijit 10y ago
  I have a little one-liner I keep around on all my docker hosts that does this for me. I don't think you needed to wait for them to add anything. #!/usr/bin/env bash docker rmi $(docker images --filter "dangling=true" -q --no-trunc)
  - alexellisuk 10y ago
    Yes and isn't it fun to type that in over and over - especially on a new machine where you don't have your .dotfiles yet :-/ The new command is going to really save hassle here.
    - nickjj 10y ago
      The new command is definitely a nice addition, but I can't say that I've manually ran the other clean up commands in a long time. They were sitting in a daily cron for the last 9 months or so.
- ryan_lane 10y ago
  The downside is the prune command is a giant hammer. In a lot of cases you don't want to clean everything, just things you know you won't be using. For instance, in a CI environment there's going to be some images that will always be there, but if you run a prune in-between jobs you've now made it necessary to re-pull all images.
dchuk 10y ago
The "Use compose-files to deploy swarm mode services" is really intriguing to me, especially when they quote this one liner: > docker stack deploy --compose-file=docker-compose.yml my_stack But there's no links to further documentation or anything. Can this be used to deploy easily to a cluster of Droplets for instance? I feel like the low end/longtail deployment of Docker is really underserved. I want to use Docker for its devops merits, but I have yet to find a clear, concise guide for deploying a simple web app to one or a cluster of VPS instances for a modest traffic project.
- andrewstuart2 10y ago
  Have you looked at Kubernetes? It's pretty dead simple to get a cluster up these days and the general abstractions they chose are really great. Service discovery, config and secret storage, control loops, autoscaling, even stateful containers these days. 100% worth giving it a shot, even just to say you've tried it. I've been using Docker for over three years now, and Kubernetes really is the realization of what I imagined containers would be like when I was just starting to use them in development.
  - dchuk 10y ago
    I haven't yet. Any good getting started guides/tutorials you could recommend?
    - alpb 10y ago
      I suggest jumping right into the documentation on kuberenetes.io. There is got decent documentation with a set of tutorials and you can check out this playground with a walkthrough: https://www.katacoda.com/courses/kubernetes/playground https://www.katacoda.com/courses/kubernetes/playground
  - sandGorgon 10y ago
    service discovery is not yet in kubernetes. the proposal is here - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service-discovery.md https://github.com/kubernetes/community/blob/master/contribu... and https://github.com/kubernetes/community/blob/a1d8453e184dfd8b1c5656e1e42528125d2eb906/contributors/design-proposals/aggregated-api-servers.md https://github.com/kubernetes/community/blob/a1d8453e184dfd8... secret storage is not done - this is the bug tracking the Hashicorp Vault proposal https://github.com/kubernetes/kubernetes/issues/10439 https://github.com/kubernetes/kubernetes/issues/10439 The configuration scheme is not fixed yet - https://github.com/kubernetes/kubernetes/issues/10439 https://github.com/kubernetes/kubernetes/issues/10439 The game is still up in the air. While I'm bullish on Kubernetes, its still not the superior choice.
iends 10y ago
Docker for Mac is a pretty terrible experience. https://github.com/docker/for-mac/issues/77 https://github.com/docker/for-mac/issues/77 has eaten a huge chunk of time for my team and the issue has been open and known about since the beta. Yes there are hacks to make things better, but as of a month or two ago the Docker team seems to have gone radio silent after providing periodic updates.
- duncan-donuts 10y ago
  I have given up on using docker on macOS. I now use Ubuntu at work, and it's interesting to watch a lot of my co-workers give up on macOS too. I totally agree it is a horrible experience.
  - nallerooth 10y ago
    This is the case where I work too. I'm guessing that at least 75% of our engineering team now uses Linux, even though everyone were given a MBP when hired.
  - dandermotj 10y ago
    Use Docker on Ubuntu on my own machine, but forced to use a Docker on Windows at work. Using Docker on Linux is an absolute joy. I totally advocate it. Dockerhub is my first port of call when I'm installing something new on my machine because I know if there's an official repo it will work. Docker outside of Linux is another story. I've run into config issues, VM memory problems, proxy issues, ect. Painful.
    - duncan-donuts 10y ago
      Same here. I was struggling using docker at work on my Mac, but when I was working at home on my Linux box it was amazing. Docker is awesome on Linux and I have never been more productive.
  - mmgutz 10y ago
    Same with our team. Linux, for the most part, is a much better development experience. The only con is our front-end guy can't use Photoshop. I don't even miss iterm2 anymore. i3 tiling WM is iterm2 for the whole desktop on steroids.
HorizonXP 10y ago
This release broke our Azure Container Service Kubernetes deployment. Had to downgrade back to 1.12 for now, but not without suffering a day of downtime trying to trace the issue.
- deleted 10y ago
  [deleted]
- tristanz 10y ago
  What was the issue?
  - HorizonXP 10y ago
    https://github.com/Azure/ACS/issues/3 https://github.com/Azure/ACS/issues/3
- kim0 10y ago
  Guess I'm not upgrading soon! Thanks for mentioning
jzelinskie 10y ago
Hopefully native squashing will allow some people to avoid writing Dockerfiles that look like RUN this && that && thistoo && thattoo && rm this && rm that in order to avoid generating extraneous layers. There have been some third party tools and registries that implement this feature, but it's nice to finally have it upstream.
- deleted 10y ago
  [deleted]
- alexlarsson 10y ago
  Finally they added this! I submitted a PR for this over two years ago (https://github.com/docker/docker/pull/4232 https://github.com/docker/docker/pull/4232).
- donw 10y ago
  Honestly, I see this as an antipattern, not because it generates extra layers, but because a Dockerfile isn't the right place to write a provisioning script. When building apps, I keep a small number of shell scripts that live in a `bin` directory, that perform essential operational tasks. One script installs all the dependencies required to either run or develop the app; another runs all tests and exits with either success or failure; and the final script just runs the app. Docker then just runs those scripts, which are written deliberately to be easily read, and thus function as living documentation for the project. These same scripts get used, both by developers on a daily basis, as well as by our CI/CD system when prepping containers. This also makes onboarding a snap: you run `bin/setup`, and your Mac or Linux box is good to go. And, because that script gets used every time the CI system spins up a build, it will never go out of sync with reality.
  - yladiz 10y ago
    The problem is that if you're building images a lot, Docker doesn't cache COPY calls because it doesn't know if the file has changed or not when calling the COPY command. So if you constantly build images and there are a lot of computationally intensive tasks within those scripts, but you've only changed one small item near the end of the script, your build time may be significantly longer by using the script than by putting it directly in the Dockerfile using the RUN command. I build a lot of modules within one of my Docker images for work, and while it doesn't change often, I would really not like to wait the 20 minutes if I really needed to push to production when I only change some certificate population segment at the end of the Dockerfile or something after a particularly intensive module build RUN step.
LeonM 10y ago
> docker system df will show used space, similar to the unix tool df I thought 'df' is for free space, 'du' is for used space. Can anyone explain why the went for 'df', not 'du'?
- legooolas 10y ago
  df shows both used and available space on a whole disk/device. du is for space used by particular files/directories.
jstoja 10y ago
Huge news, I really hope this is gonna help the workflow and give another alternative for scheduling. I have been waiting for the update since last November (iirc it was scheduled for end of 2016). BTW - Am I the only one finding the quality of the linked video very bad?
deleted 10y ago
[deleted]
jstoja 10y ago
I can't find anything about secrets in compose files... Any ideas about this?
- jstoja 10y ago
  Ok, it still seem to be in development: https://github.com/docker/docker/pull/30144 https://github.com/docker/docker/pull/30144
LeanderK 10y ago
why does the squash flag squash everything? Wouldn't it possible to only squash the layers introduces in the localhost build? The you could share the other layers again.
- cpuguy83 10y ago
  It only squashes the layers produced by the build.
unixhero 10y ago
I moved on to LXC Never looked back Congrats to the docker team in the new release anyhow.
- theamk 10y ago
  What do you use instead of "docker build" and "docker pull"?
  - jryan49 10y ago
    Could use the distro's package manager?
    - theamk 10y ago
      What do you mean? My Dockerfiles already use distro's package manager to install packages. Do you you mean "create a long-lived LXC container and use distro's package manager to install packages as needed?" The problem with that approach is you have no idea how to duplicate the server, nor how to roll back the changes reliably. For example: - Your distro upgraded a library, and the upgrade introduced a bug. For extra fun, this was not security upgrade so unattended-upgrade process did not install it, so only half of your servers have the bug. - Your package list is incorrect. Somehow, your old server ended up with an extra package (previous software version? installed while trying to troubleshoot problems), and your new server does not have it. - You have leftover files -- either from the previous version of your software, or from the package you have had installed before. These are definitely fixable in LXC with enough effort -- after all, Docker is not magic, and you can achieve a lot with LXC + shell scripts, and even more with LXC + shell scripts + chef. However, Docker is just so much easier and reliable than writing these scripts by hand.
      - unixhero 10y ago
        LXC is sane to me. Upon first glance it is understandable and works everytime. I went all in with Docker, it was all fun and games up until I tried it in production, and I started relying heavily on it. Docker introduces container linking voodoo which have burnt me badly. Sure enough voodoo comes with its upsides, we now have Kubernetes which obviously I think is very cool. Just far outside of my use case. Serenity now!
colemorrison 10y ago
In addition to crappy mounted volumes, Docker for Mac eats up CPU like crazy: https://forums.docker.com/t/com-docker-hyperkit-up-cpu-to-340/13057/2 https://forums.docker.com/t/com-docker-hyperkit-up-cpu-to-34... And, also like the mounted voluems issue, despite this being on of the top things on the forum, it's still been a problem for like 8 months with no attention.
tvaughan 10y ago
Use Terraform to provision some virtual machines, load balancers, bastion host, vpc, etc. https://news.ycombinator.com/item?id=13436415 https://news.ycombinator.com/item?id=13436415 Use Docker to setup development and production environments for a sample Flask application with CI/CD. https://news.ycombinator.com/item?id=13436452 https://news.ycombinator.com/item?id=13436452

149 points by hackerpt 10y ago

I'm very excited about the clean-up commands! Docker desperately needs this, it's so frustrating constantly having full disk space due to untagged containers and unbounded volumes.

geerlingguy 10y ago
If you're on Mac, it's still a bit of a problem; hopefully the SSD-gobbling Docker.qcow2 file problem[1] is fixed soon! [1] https://github.com/docker/for-mac/issues/371 https://github.com/docker/for-mac/issues/371
- selvakn 10y ago
  Looks like it is fixed in https://twitter.com/kelseyhightower/status/822237099495550976 https://twitter.com/kelseyhightower/status/82223709949555097.... Im looking for official changelog for confirmation though.
  - iends 10y ago
    Judging by the github issue list there are a lot more issues with Docker.qcow2 now than before. Mostly related to Docker.qcow2 preventing docker from starting. Looks like a bad release. Hopefully 1.13.1 is soon.
    - djs55 10y ago
      (I work on Docker for Mac.) Apologies for the inconvenience this has caused. There was a race condition in a previous release which could allow multiple hypervisor instances to open the Docker.qcow2 simultaneously. Unfortunately this can corrupt the file by allocating the same physical block (cluster) twice, resulting in bad things happening. When this file-locking bug was fixed we also added an integrity check which checks the structure of the Docker.qcow2 on every application launch. For safety the app refuses to start if corruption is detected. I believe that in these cases, the corruption happened in the past and is now being detected since the upgrade. Unfortunately if the app refuses to start it makes it difficult to reach the "Reset to Factory defaults" menu option. The workaround described here https://github.com/docker/for-mac/issues/1159#issuecomment-273899215 https://github.com/docker/for-mac/issues/1159#issuecomment-2... is to remove the qcow2 and restart the app. Unfortunately containers and images will need to be rebuilt. For what it's worth after the integrity check and the locking fix went in, I've not seen any recurrence of this error. Please open an issue if you see any other problems!
- justincormack 10y ago
  This is now fixed on the stable release, but space is only released when you restart docker at present. Online freeing is being worked on too.
dijit 10y ago
I have a little one-liner I keep around on all my docker hosts that does this for me. I don't think you needed to wait for them to add anything. #!/usr/bin/env bash docker rmi $(docker images --filter "dangling=true" -q --no-trunc)
- alexellisuk 10y ago
  Yes and isn't it fun to type that in over and over - especially on a new machine where you don't have your .dotfiles yet :-/ The new command is going to really save hassle here.
  - nickjj 10y ago
    The new command is definitely a nice addition, but I can't say that I've manually ran the other clean up commands in a long time. They were sitting in a daily cron for the last 9 months or so.
ryan_lane 10y ago
The downside is the prune command is a giant hammer. In a lot of cases you don't want to clean everything, just things you know you won't be using. For instance, in a CI environment there's going to be some images that will always be there, but if you run a prune in-between jobs you've now made it necessary to re-pull all images.

dchuk 10y ago

The "Use compose-files to deploy swarm mode services" is really intriguing to me, especially when they quote this one liner: > docker stack deploy --compose-file=docker-compose.yml my_stack But there's no links to further documentation or anything. Can this be used to deploy easily to a cluster of Droplets for instance? I feel like the low end/longtail deployment of Docker is really underserved. I want to use Docker for its devops merits, but I have yet to find a clear, concise guide for deploying a simple web app to one or a cluster of VPS instances for a modest traffic project.

andrewstuart2 10y ago
Have you looked at Kubernetes? It's pretty dead simple to get a cluster up these days and the general abstractions they chose are really great. Service discovery, config and secret storage, control loops, autoscaling, even stateful containers these days. 100% worth giving it a shot, even just to say you've tried it. I've been using Docker for over three years now, and Kubernetes really is the realization of what I imagined containers would be like when I was just starting to use them in development.
- dchuk 10y ago
  I haven't yet. Any good getting started guides/tutorials you could recommend?
  - alpb 10y ago
    I suggest jumping right into the documentation on kuberenetes.io. There is got decent documentation with a set of tutorials and you can check out this playground with a walkthrough: https://www.katacoda.com/courses/kubernetes/playground https://www.katacoda.com/courses/kubernetes/playground
- sandGorgon 10y ago
  service discovery is not yet in kubernetes. the proposal is here - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service-discovery.md https://github.com/kubernetes/community/blob/master/contribu... and https://github.com/kubernetes/community/blob/a1d8453e184dfd8b1c5656e1e42528125d2eb906/contributors/design-proposals/aggregated-api-servers.md https://github.com/kubernetes/community/blob/a1d8453e184dfd8... secret storage is not done - this is the bug tracking the Hashicorp Vault proposal https://github.com/kubernetes/kubernetes/issues/10439 https://github.com/kubernetes/kubernetes/issues/10439 The configuration scheme is not fixed yet - https://github.com/kubernetes/kubernetes/issues/10439 https://github.com/kubernetes/kubernetes/issues/10439 The game is still up in the air. While I'm bullish on Kubernetes, its still not the superior choice.

iends 10y ago

Docker for Mac is a pretty terrible experience. https://github.com/docker/for-mac/issues/77 https://github.com/docker/for-mac/issues/77 has eaten a huge chunk of time for my team and the issue has been open and known about since the beta. Yes there are hacks to make things better, but as of a month or two ago the Docker team seems to have gone radio silent after providing periodic updates.

duncan-donuts 10y ago
I have given up on using docker on macOS. I now use Ubuntu at work, and it's interesting to watch a lot of my co-workers give up on macOS too. I totally agree it is a horrible experience.
- nallerooth 10y ago
  This is the case where I work too. I'm guessing that at least 75% of our engineering team now uses Linux, even though everyone were given a MBP when hired.
- dandermotj 10y ago
  Use Docker on Ubuntu on my own machine, but forced to use a Docker on Windows at work. Using Docker on Linux is an absolute joy. I totally advocate it. Dockerhub is my first port of call when I'm installing something new on my machine because I know if there's an official repo it will work. Docker outside of Linux is another story. I've run into config issues, VM memory problems, proxy issues, ect. Painful.
  - duncan-donuts 10y ago
    Same here. I was struggling using docker at work on my Mac, but when I was working at home on my Linux box it was amazing. Docker is awesome on Linux and I have never been more productive.
- mmgutz 10y ago
  Same with our team. Linux, for the most part, is a much better development experience. The only con is our front-end guy can't use Photoshop. I don't even miss iterm2 anymore. i3 tiling WM is iterm2 for the whole desktop on steroids.

HorizonXP 10y ago

This release broke our Azure Container Service Kubernetes deployment. Had to downgrade back to 1.12 for now, but not without suffering a day of downtime trying to trace the issue.

deleted 10y ago
[deleted]
tristanz 10y ago
What was the issue?
- HorizonXP 10y ago
  https://github.com/Azure/ACS/issues/3 https://github.com/Azure/ACS/issues/3
kim0 10y ago
Guess I'm not upgrading soon! Thanks for mentioning

jzelinskie 10y ago

Hopefully native squashing will allow some people to avoid writing Dockerfiles that look like RUN this && that && thistoo && thattoo && rm this && rm that in order to avoid generating extraneous layers. There have been some third party tools and registries that implement this feature, but it's nice to finally have it upstream.

deleted 10y ago
[deleted]
alexlarsson 10y ago
Finally they added this! I submitted a PR for this over two years ago (https://github.com/docker/docker/pull/4232 https://github.com/docker/docker/pull/4232).
donw 10y ago
Honestly, I see this as an antipattern, not because it generates extra layers, but because a Dockerfile isn't the right place to write a provisioning script. When building apps, I keep a small number of shell scripts that live in a `bin` directory, that perform essential operational tasks. One script installs all the dependencies required to either run or develop the app; another runs all tests and exits with either success or failure; and the final script just runs the app. Docker then just runs those scripts, which are written deliberately to be easily read, and thus function as living documentation for the project. These same scripts get used, both by developers on a daily basis, as well as by our CI/CD system when prepping containers. This also makes onboarding a snap: you run `bin/setup`, and your Mac or Linux box is good to go. And, because that script gets used every time the CI system spins up a build, it will never go out of sync with reality.
- yladiz 10y ago
  The problem is that if you're building images a lot, Docker doesn't cache COPY calls because it doesn't know if the file has changed or not when calling the COPY command. So if you constantly build images and there are a lot of computationally intensive tasks within those scripts, but you've only changed one small item near the end of the script, your build time may be significantly longer by using the script than by putting it directly in the Dockerfile using the RUN command. I build a lot of modules within one of my Docker images for work, and while it doesn't change often, I would really not like to wait the 20 minutes if I really needed to push to production when I only change some certificate population segment at the end of the Dockerfile or something after a particularly intensive module build RUN step.

LeonM 10y ago

> docker system df will show used space, similar to the unix tool df I thought 'df' is for free space, 'du' is for used space. Can anyone explain why the went for 'df', not 'du'?

legooolas 10y ago
df shows both used and available space on a whole disk/device. du is for space used by particular files/directories.

jstoja 10y ago

Huge news, I really hope this is gonna help the workflow and give another alternative for scheduling. I have been waiting for the update since last November (iirc it was scheduled for end of 2016). BTW - Am I the only one finding the quality of the linked video very bad?

deleted 10y ago

[deleted]

jstoja 10y ago

I can't find anything about secrets in compose files... Any ideas about this?

jstoja 10y ago
Ok, it still seem to be in development: https://github.com/docker/docker/pull/30144 https://github.com/docker/docker/pull/30144

LeanderK 10y ago

why does the squash flag squash everything? Wouldn't it possible to only squash the layers introduces in the localhost build? The you could share the other layers again.

cpuguy83 10y ago
It only squashes the layers produced by the build.

unixhero 10y ago

I moved on to LXC Never looked back Congrats to the docker team in the new release anyhow.

theamk 10y ago
What do you use instead of "docker build" and "docker pull"?
- jryan49 10y ago
  Could use the distro's package manager?
  - theamk 10y ago
    What do you mean? My Dockerfiles already use distro's package manager to install packages. Do you you mean "create a long-lived LXC container and use distro's package manager to install packages as needed?" The problem with that approach is you have no idea how to duplicate the server, nor how to roll back the changes reliably. For example: - Your distro upgraded a library, and the upgrade introduced a bug. For extra fun, this was not security upgrade so unattended-upgrade process did not install it, so only half of your servers have the bug. - Your package list is incorrect. Somehow, your old server ended up with an extra package (previous software version? installed while trying to troubleshoot problems), and your new server does not have it. - You have leftover files -- either from the previous version of your software, or from the package you have had installed before. These are definitely fixable in LXC with enough effort -- after all, Docker is not magic, and you can achieve a lot with LXC + shell scripts, and even more with LXC + shell scripts + chef. However, Docker is just so much easier and reliable than writing these scripts by hand.
    - unixhero 10y ago
      LXC is sane to me. Upon first glance it is understandable and works everytime. I went all in with Docker, it was all fun and games up until I tried it in production, and I started relying heavily on it. Docker introduces container linking voodoo which have burnt me badly. Sure enough voodoo comes with its upsides, we now have Kubernetes which obviously I think is very cool. Just far outside of my use case. Serenity now!

colemorrison 10y ago

In addition to crappy mounted volumes, Docker for Mac eats up CPU like crazy: https://forums.docker.com/t/com-docker-hyperkit-up-cpu-to-340/13057/2 https://forums.docker.com/t/com-docker-hyperkit-up-cpu-to-34... And, also like the mounted voluems issue, despite this being on of the top things on the forum, it's still been a problem for like 8 months with no attention.

tvaughan 10y ago

Use Terraform to provision some virtual machines, load balancers, bastion host, vpc, etc. https://news.ycombinator.com/item?id=13436415 https://news.ycombinator.com/item?id=13436415 Use Docker to setup development and production environments for a sample Flask application with CI/CD. https://news.ycombinator.com/item?id=13436452 https://news.ycombinator.com/item?id=13436452