9 ms·

The first thing I wanted to do with docker was cleanup the mess I made walking through the examples. Turns out there is no built-in command to do that, and it l

by j_s 13y ago

The first thing I wanted to do with docker was cleanup the mess I made walking through the examples. Turns out there is no built-in command to do that, and it looks like it is going to stay that way. https://github.com/dotcloud/docker/issues/928 https://github.com/dotcloud/docker/issues/928 I was surprised to see that nothing had been done about this; a comment there mentioned that someone was able to clean up 19GB of space once he figured out what was going on. The solution is easily google-able since so many have run into it, but cleanup of unused resources should be a first-class operation in every virtualization solution. http://stackoverflow.com/questions/17236796/how-to-remove-old-docker-io-containers http://stackoverflow.com/questions/17236796/how-to-remove-ol...

danielbln 13y ago
I actually have a cron job running that cleans up old images once in a while. Not pretty, but yeah, otherwise disk usage grows and grows and grows.
nickstinemates 13y ago
What would you recommend be done? Good solutions are always welcome.
- j_s 13y ago
  I'm still learning my way around, but based on my understanding of containers as instantiations of images, there should be a command to delete unreferenced images. I understand the need to manually specify containers to delete but not the difficulty in cleaning up unreferenced dependencies.
  - danudey 13y ago
    The problem with that design is that in a lot of use cases, images don't necessarily need to be 'referenced'. For example, I'm using docker for a build/test system; we spin up a container from an image, run some stuff in it for a few seconds, and then shut it down. 90% of the time, the images are 'unreferenced'. The other suggestion, put forward by a docker dev in another comment, is to have a clean command for unnamed/untagged images. Our build system has 'buildbot' (python package compilation) and 'software' (test environment) images, so this sort of automated cleaning would get rid of all of that extra crap that we have lying around. The big issue for me is stacks of changes. Like git, anything you commit, remove something from, then commit again still takes up that space forever until you consolidate down. Which you can't do automatically.
shykes 13y ago
Hey, I'm a Docker maintainer. We don't plan on "keeping it that way". Disk clutter after lots of builds is a problem which we're very much addressing. It's specifically a top-level docker clean command that I'm not a fan of, because it doesn't capture a clear intent - what are you cleaning, and who determines what should be kept and what shouldn't? In general I like to avoid Docker having to guess what you want - the proverbial "magic". A few notes: * The reason Docker can clutter your disk is because we made an explicit tradeoff: it's better to keep too much data than to remove too much. That is why we're able to say "yes, you can run a database on Docker. Neither the container nor its data will be removed, at any point, unless you explicitly request it". Everytime I hear someone complain "there's too much stuff on my disk", I rejoice that I am not instead hearing "Docker ate my data". * We have already introduced a few improvements. For example Docker used to keep the intermediary container for each build step (for later inspection of the build output, transient state etc). Now, by default, we don't. * The most elegant fix, I believe, is to use naming and tagging as an indication of interest. If you are interested in keeping an image, give it a name (for example with 'docker build -t'). When an image no longer has a name referencing it, instead of keeping it around as an anonymous image like it does today, Docker can remove garbage collect it. That should take care of the problem elegantly and without requiring a new, clunky top-level command. TLDR: the "disk clutter" problem is the result of a conscious tradeoff which we are glad we made. We have started fixing it, and there are more fixes to come.
- j_s 13y ago
  > We don't plan on "keeping it that way". Cool! Can you point me to where the discussion on this is occurring? I was confused because the issue I referenced above was closed without a reference to any issue open on how this problem will be addressed.
  - shykes 13y ago
    Most of the discussion has been on irc: #docker and #docker-dev on Freenode. There are several contributions underway under the general "theme" I described, but if it helps I could write a summary of how the experience will change for the user (normally there should be almost no UI change).
darkarmani 13y ago
Thank you for your feedback! I'm not a Docker contributor but I'm a big fan, and everyone that responded to you, forgot to thank you for your feedback. Your feedback is immensely valuable to making it better. Obviously if you think nothing has been done about it and they are saying there are fixes, it needs to be communicated better.
- shykes 13y ago
  +1 on that, anyone who takes the time to tell us about their issues with Docker is automatically making a valuable contribution to the project. And it's appreciated, so yes, thank you :)
CSDude 13y ago
There is another problem of other than space. I grade programming assigments with Docker and let the students grade too, and when containers get too much, docker tries to read each of the metadata and it hangs, and I could not find the issue and could not remote reboot my server, so I walked to server room at 4 am. For a 8 core system, load avg was 33. So, be careful with your containers, remove them when not needed.
koffiezet 13y ago
There is a difference between "messing around", and actually running docker in production. I do NOT want a 'cleanup' command to be available on my servers. Cleaning up on commandline is rather easy with a few docker commands and xargs. I wrote small shellscripts to do this for me when building images, but these will never ever be installed on a production server.

by j_s 13y ago

danielbln 13y ago

I actually have a cron job running that cleans up old images once in a while. Not pretty, but yeah, otherwise disk usage grows and grows and grows.

nickstinemates 13y ago

What would you recommend be done? Good solutions are always welcome.

j_s 13y ago
I'm still learning my way around, but based on my understanding of containers as instantiations of images, there should be a command to delete unreferenced images. I understand the need to manually specify containers to delete but not the difficulty in cleaning up unreferenced dependencies.
- danudey 13y ago
  The problem with that design is that in a lot of use cases, images don't necessarily need to be 'referenced'. For example, I'm using docker for a build/test system; we spin up a container from an image, run some stuff in it for a few seconds, and then shut it down. 90% of the time, the images are 'unreferenced'. The other suggestion, put forward by a docker dev in another comment, is to have a clean command for unnamed/untagged images. Our build system has 'buildbot' (python package compilation) and 'software' (test environment) images, so this sort of automated cleaning would get rid of all of that extra crap that we have lying around. The big issue for me is stacks of changes. Like git, anything you commit, remove something from, then commit again still takes up that space forever until you consolidate down. Which you can't do automatically.

shykes 13y ago

Hey, I'm a Docker maintainer. We don't plan on "keeping it that way". Disk clutter after lots of builds is a problem which we're very much addressing. It's specifically a top-level docker clean command that I'm not a fan of, because it doesn't capture a clear intent - what are you cleaning, and who determines what should be kept and what shouldn't? In general I like to avoid Docker having to guess what you want - the proverbial "magic". A few notes: * The reason Docker can clutter your disk is because we made an explicit tradeoff: it's better to keep too much data than to remove too much. That is why we're able to say "yes, you can run a database on Docker. Neither the container nor its data will be removed, at any point, unless you explicitly request it". Everytime I hear someone complain "there's too much stuff on my disk", I rejoice that I am not instead hearing "Docker ate my data". * We have already introduced a few improvements. For example Docker used to keep the intermediary container for each build step (for later inspection of the build output, transient state etc). Now, by default, we don't. * The most elegant fix, I believe, is to use naming and tagging as an indication of interest. If you are interested in keeping an image, give it a name (for example with 'docker build -t'). When an image no longer has a name referencing it, instead of keeping it around as an anonymous image like it does today, Docker can remove garbage collect it. That should take care of the problem elegantly and without requiring a new, clunky top-level command. TLDR: the "disk clutter" problem is the result of a conscious tradeoff which we are glad we made. We have started fixing it, and there are more fixes to come.

j_s 13y ago
> We don't plan on "keeping it that way". Cool! Can you point me to where the discussion on this is occurring? I was confused because the issue I referenced above was closed without a reference to any issue open on how this problem will be addressed.
- shykes 13y ago
  Most of the discussion has been on irc: #docker and #docker-dev on Freenode. There are several contributions underway under the general "theme" I described, but if it helps I could write a summary of how the experience will change for the user (normally there should be almost no UI change).

darkarmani 13y ago

Thank you for your feedback! I'm not a Docker contributor but I'm a big fan, and everyone that responded to you, forgot to thank you for your feedback. Your feedback is immensely valuable to making it better. Obviously if you think nothing has been done about it and they are saying there are fixes, it needs to be communicated better.

shykes 13y ago
+1 on that, anyone who takes the time to tell us about their issues with Docker is automatically making a valuable contribution to the project. And it's appreciated, so yes, thank you :)

CSDude 13y ago

There is another problem of other than space. I grade programming assigments with Docker and let the students grade too, and when containers get too much, docker tries to read each of the metadata and it hangs, and I could not find the issue and could not remote reboot my server, so I walked to server room at 4 am. For a 8 core system, load avg was 33. So, be careful with your containers, remove them when not needed.

koffiezet 13y ago

There is a difference between "messing around", and actually running docker in production. I do NOT want a 'cleanup' command to be available on my servers. Cleaning up on commandline is rather easy with a few docker commands and xargs. I wrote small shellscripts to do this for me when building images, but these will never ever be installed on a production server.