3 ms·
* conda-store * https://github.com/Quansight/conda-store/ https://github.com/Quansight/conda-store/ This one is really interesting, particularly for data scien
by sterlinm 6y ago
* conda-store *
https://github.com/Quansight/conda-store/ https://github.com/Quansight/conda-store/
This one is really interesting, particularly for data science workloads. I think it's not a good solution for serving applications, but really interesting for things like JupyterHub, Dask, and Fargate.
The general idea is that you store your conda environments (the equivalent of the /venv folder in your example) on an NFS drive (or EFS if you're on AWS). Your containers then mount the network drive. This can help you a lot if you want to build containerized development environments for data scientists. For an application you can strip down your conda environment to the bare minimum package set required to run your app, but if you want to containerize a jupyter (or VS Code) development environment your users will likely want to have 10-20 packages at the minimum. The package sets I've worked with often lead to 3GB+ images if you're really aggressive about trying to keep it down. The official jupyter/datascience-notebook image is 1.8GB compressed and 3.8GB after the pull is finished.
conda-store is a component of the QHub project from Quansight. The architecture diagram from that project might be of interest. https://github.com/Quansight/qhub/tree/master/architecture https://github.com/Quansight/qhub/tree/master/architecture