3 ms·

Shrink your Conda Docker images with conda-pack

5 points by itamarst 6y ago

phuicy 6y ago
Could it be smaller if you used alpine instead of debian?
- itamarst 6y ago
  Yes. But the Alpine C library is musl, not glibc, and musl can sometimes be incompatible with code compiled against glibc, so I don't like using it.
sterlinm 6y ago
I've been doing a lot of research on this issue and I can't remember what it was but I remember getting a few weird error messages with conda-pack. I've found your articles at pythonspeed.com really useful, so I'm glad to see this article. I've been exploring a few other options that I'd be interested to hear your thoughts on. I'll explain them separately in replies to this comment.
- sterlinm 6y ago
  * conda constructor * https://github.com/conda/constructor https://github.com/conda/constructor Conda installers like miniconda3 and miniforge3 can be built using the conda constructor package. It offers some more customization than is available with conda-pack but you can build an image of a similar size as the one you describe in your article. If you want to use vanilla docker, I think you need three stages in your build. Stage 1: Build the installer. Stage 2: Run the installer. Stage 3: Copy /venv from stage 2. I think if you use buildkit you might be able to use a mount command in your RUN statement and get it down to 2 stages but I haven't tried that yet.
- sterlinm 6y ago
  * conda-docker * https://github.com/conda-incubator/conda-docker https://github.com/conda-incubator/conda-docker I haven't done as much with this one yet, but it's interesting in that it claims to be able to do much faster builds because it doesn't actually use the docker engine it just manipulates the image files. I think it's pretty neat but I don't have much experience with it yet and I think it might not fit as well into normal workflows since it's not using the docker engine. I could be wrong though.
- sterlinm 6y ago
  * conda-store * https://github.com/Quansight/conda-store/ https://github.com/Quansight/conda-store/ This one is really interesting, particularly for data science workloads. I think it's not a good solution for serving applications, but really interesting for things like JupyterHub, Dask, and Fargate. The general idea is that you store your conda environments (the equivalent of the /venv folder in your example) on an NFS drive (or EFS if you're on AWS). Your containers then mount the network drive. This can help you a lot if you want to build containerized development environments for data scientists. For an application you can strip down your conda environment to the bare minimum package set required to run your app, but if you want to containerize a jupyter (or VS Code) development environment your users will likely want to have 10-20 packages at the minimum. The package sets I've worked with often lead to 3GB+ images if you're really aggressive about trying to keep it down. The official jupyter/datascience-notebook image is 1.8GB compressed and 3.8GB after the pull is finished. conda-store is a component of the QHub project from Quansight. The architecture diagram from that project might be of interest. https://github.com/Quansight/qhub/tree/master/architecture https://github.com/Quansight/qhub/tree/master/architecture

5 points by itamarst 6y ago

phuicy 6y ago

Could it be smaller if you used alpine instead of debian?

itamarst 6y ago
Yes. But the Alpine C library is musl, not glibc, and musl can sometimes be incompatible with code compiled against glibc, so I don't like using it.

sterlinm 6y ago

I've been doing a lot of research on this issue and I can't remember what it was but I remember getting a few weird error messages with conda-pack. I've found your articles at pythonspeed.com really useful, so I'm glad to see this article. I've been exploring a few other options that I'd be interested to hear your thoughts on. I'll explain them separately in replies to this comment.

sterlinm 6y ago
* conda constructor * https://github.com/conda/constructor https://github.com/conda/constructor Conda installers like miniconda3 and miniforge3 can be built using the conda constructor package. It offers some more customization than is available with conda-pack but you can build an image of a similar size as the one you describe in your article. If you want to use vanilla docker, I think you need three stages in your build. Stage 1: Build the installer. Stage 2: Run the installer. Stage 3: Copy /venv from stage 2. I think if you use buildkit you might be able to use a mount command in your RUN statement and get it down to 2 stages but I haven't tried that yet.
sterlinm 6y ago
* conda-docker * https://github.com/conda-incubator/conda-docker https://github.com/conda-incubator/conda-docker I haven't done as much with this one yet, but it's interesting in that it claims to be able to do much faster builds because it doesn't actually use the docker engine it just manipulates the image files. I think it's pretty neat but I don't have much experience with it yet and I think it might not fit as well into normal workflows since it's not using the docker engine. I could be wrong though.
sterlinm 6y ago
* conda-store * https://github.com/Quansight/conda-store/ https://github.com/Quansight/conda-store/ This one is really interesting, particularly for data science workloads. I think it's not a good solution for serving applications, but really interesting for things like JupyterHub, Dask, and Fargate. The general idea is that you store your conda environments (the equivalent of the /venv folder in your example) on an NFS drive (or EFS if you're on AWS). Your containers then mount the network drive. This can help you a lot if you want to build containerized development environments for data scientists. For an application you can strip down your conda environment to the bare minimum package set required to run your app, but if you want to containerize a jupyter (or VS Code) development environment your users will likely want to have 10-20 packages at the minimum. The package sets I've worked with often lead to 3GB+ images if you're really aggressive about trying to keep it down. The official jupyter/datascience-notebook image is 1.8GB compressed and 3.8GB after the pull is finished. conda-store is a component of the QHub project from Quansight. The architecture diagram from that project might be of interest. https://github.com/Quansight/qhub/tree/master/architecture https://github.com/Quansight/qhub/tree/master/architecture