8 ms·

A deep dive into the official Docker image for Python

189 points by itamarst 6y ago

bransonf 6y ago
> Debian names all their releases after characters from Toy Story. I can’t believe I’m just now learning this, but that’s good to know next time someone asks why the names seem random.
- lub 6y ago
  And sid (unstable) is the neighbor who breaks the toys, which is why the name never changes :)
  - beagle3 6y ago
    Story I heard is that it started with “sid” being an acronym for “still in development”, but once that caught, it blazed the trail for the rest of the Toy Story characters....
    - fanf2 6y ago
      I heard it was because when Bruce Perens was Debian Project Leader, he was working for Pixar around the time of the Toy Story release, and chose the naming scheme. The descriptive name for unstable has always been unstable, not "still in development".
  - Igelau 6y ago
    Looking back, Sid was an artist being smothered by mundane suburbia, while Andy was a milquetoast who couldn't look after his stuff.
    - mikepurvis 6y ago
      Others have noticed this too— some examples: https://www.reddit.com/r/FanTheories/comments/1qmtoi/sid_from_toy_story_wasnt_actually_a_bad_kid_he/ https://www.reddit.com/r/FanTheories/comments/1qmtoi/sid_fro... https://matthewdicks.com/film/2018-5-23-sid-from-toy-story-was-a-completely-normal-person/ https://matthewdicks.com/film/2018-5-23-sid-from-toy-story-w...
      - emmelaich 6y ago
        and https://puzzling.org/entertainment/movies-entertainment/2013/04/team-sid-why-the-antagonist-became-my-favourite/ https://puzzling.org/entertainment/movies-entertainment/2013...
- umvi 6y ago
  Holy cow ...! I guess the reason I never noticed was because I started using Debian around version 7, and the majority of the iconic Toy Story names (Woody, Buzz, Slink, Potato, etc.) are all pre-7.
  - R0b0t1 6y ago
    Same exact thing for me. I rue the day they run out of names :(.
    - LeoPanthera 6y ago
      Plenty more characters in the sequels.
- pyuser583 6y ago
  Yeah - wow.
- dazzawazza 6y ago
  It's cute but really frustrating. I wish they'd just use the version number or force everyone to use "<version #>-<stupidname>". but I'm a FreeBSD user so what do I know?
  - kemotep 6y ago
    To be fair, you can just point to the `stable` repositories in the `/etc/apt/sources.list` configuration file and simply just refer to your install as "Debian Stable" ('s/stable/testing/g'). However every few years there might be a not so "stable" update around the new release.
    - dazzawazza 6y ago
      It's not for me, it's for when I get bug reports and I need to decipher what flavour of linux, what release of the kernel, what libc etc. "I'm running decrepit duck" doesn't really narrow it down too much. Especially when multiple distributions use these jovial monikers.
      - kemotep 6y ago
        I can understand that frustration if you are used to the BSD's. Windows 10 and macOS keep it pretty simple too.
        black3r 6y ago
        Ubuntu is the most simplest.., 20.04 means it was released in april 2020. Windows 2004 means it was finalized in april 2020, but released some times later (may? june? who knows).., plus they both try to release every 6 months and in case of ubuntu you know that even aprils are LTS, so it's always predictable to know what's the latest version. mac 10.15 means what? and the next release is gonna be 11.0. someone who doesn't follow or use macs has no idea to figure out if the latest version is 10.15 or 10.14, or 11.0...
        6y ago
- Hamuko 6y ago
  I think even the Deb + Ian naming is still a fairly unknown piece of knowledge.
monkpit 6y ago
> A common mistake for people using this base image is to install Python again, by using Debian’s version of Python Is that really a common mistake?
- JulianWasTaken 6y ago
  16,328 results just on GitHub, though it fluctuates on refresh, and some are commented out: https://github.com/search?q=%22from+python%3A%22+apt-get+install+python3-dev&type=Code https://github.com/search?q=%22from+python%3A%22+apt-get+ins... (And there's likely what to clean up in the search query to make it filter more irrelevant results). Seems like though "yes" is the answer.
- digitallogic 6y ago
  Whoa boy, is it ever, but maybe not for the reason you're thinking. ie it isn't caused by people typing `apt-get install python`. There are many packages that have Python as a dependency these days. For example, on my Ubuntu system: > ~$ apt-cache rdepends python|wc -l > 4649 I think the best illustration of how this can happen is installing postgres libraries needed to build the psycopg2 PG client. If you know to install `libpq-dev` then you're great. But if you do something that on the surface feels totally reasonable, like installing the `postgresql-client` package... guess what? You just installed another Python interpreter. edit: formatting
- itamarst 6y ago
  Beyond the other two answers, I added this bit because I keep seeing people make this mistake in their StackOverflow questions about why their Dockerfile isn't working.
claytonjy 6y ago
> The packages—gcc and so on—needed to compile Python are removed once they are no longer needed. Is there a reason to prefer this method, where installation, usage, and removal all happen in one RUN, vs. using a multi-stage build? I tend to prefer the latter but am not aware of tradeoffs beyond the readability of the Dockerfile.
- vbernat 6y ago
  The reason is to minimize the number of layers, mostly the layers that are not used in the final image (in this example, a layer would include gcc while another would remove it, but you would still need to download the layer with gcc).
  - claytonjy 6y ago
    I thought non-final stages in a multi-stage build are left out of the final artifact; is that incorrect? I'm thinking the "build step" would be done in an earlier stage; it could be the exact same RUN statement, or it could be split into multiple for readability, and wouldn't bother removing any installed packages, since they won't carry over to the final stage. Then the big RUN in the final stage would be replaced with something like `COPY --from=builder ...`.
    - verst 6y ago
      EDIT: Provide example below If you are doing multi stage builds it only matters to combined as many statements as possible in the last layer. I agree that for clarity it is nice not to optimize layers in the build stage - those will be thrown away anyway. I vastly prefer multi stage builds over having to chain install and cleanup statements Example: I usually want to use the python:3-slim image, but this doesn't have the tools to compile certain python libraries with C extensions. Generally I will use the python:3 image for my build stage to do my "pip install -r requirements.txt" and then copy the libraries over to my final stage based on the python:3-slim image Of course I could install and uninstall GCC and other tools in a single stage.. but that actually takes longer to do and is messier in my opinion.
westurner 6y ago
There are Alpine [1] and Debian [2] miniconda images (within which you can `conda install python==3.8` and 2.7 and 3.4 in different conda envs) [1] https://github.com/ContinuumIO/docker-images/blob/master/miniconda3/alpine/Dockerfile https://github.com/ContinuumIO/docker-images/blob/master/min... [2] https://github.com/ContinuumIO/docker-images/blob/master/miniconda3/debian/Dockerfile https://github.com/ContinuumIO/docker-images/blob/master/min... If you build manylinux wheels with auditwheel [3], they should install without needing compilation for {CentOS, Debian, Ubuntu, and Alpine}; though standard Alpine images have MUSL instead of glibc by default, this [4] may work: echo "manylinux1_compatible = True" > $PYTHON_PATH/_manylinux.py [3] https://github.com/pypa/auditwheel https://github.com/pypa/auditwheel [4] https://github.com/docker-library/docs/issues/904#issuecomment-544181321 https://github.com/docker-library/docs/issues/904#issuecomme... The miniforge docker images aren't yet [5][6] multi-arch, which means it's not as easy to take advantage of all of the ARM64 / aarch64 packages that conda-forge builds now. [5] https://github.com/conda-forge/docker-images/issues/102#issuecomment-619963279 https://github.com/conda-forge/docker-images/issues/102#issu... [6] https://github.com/conda-forge/miniforge/issues/20 https://github.com/conda-forge/miniforge/issues/20 There are i686 and x86-64 docker containers for building manylinux wheels that work with many distros: https://github.com/pypa/manylinux/tree/master/docker https://github.com/pypa/manylinux/tree/master/docker A multi-stage Dockerfile build can produce a wheel in the first stage and install that wheel (with `COPY --from=0`) in a later stage; leaving build dependencies out of the production environment for security and performance: https://docs.docker.com/develop/develop-images/multistage-build/ https://docs.docker.com/develop/develop-images/multistage-bu...
- athorax 6y ago
  Interesting! I use miniconda extensively for local development to manage virtual environments for different python versions and love it. I hardly ever actually use the conda packages though. I assume the main benefit of using these images would be if you are installing from conda repos instead of pip? Otherwise just using the official python images would be as good if not better Edit: I guess if you needed multiple python versions in a single container this would be a good solution for that as well
chromedev 6y ago
One thing I've learned is that rather than use a docker-entrypoint.sh, most Linux software can be ran just using the `--user 1000:1000` or whatever UID/GID you want to use, as long as you map a volume that can use those permissions. It is a lot cleaner this way.
- westurner 6y ago
  > Why Tini? > Using Tini has several benefits: > - It protects you from software that accidentally creates zombie processes, which can (over time!) starve your entire system for PIDs (and make it unusable). > - It ensures that the default signal handlers work for the software you run in your Docker image. For example, with Tini, SIGTERM properly terminates your process even if you didn't explicitly install a signal handler for it. > - It does so completely transparently! Docker images that work without Tini will work with Tini without any changes. [...] > NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just pass the `--init` flag to docker run. https://github.com/krallin/tini#why-tini https://github.com/krallin/tini#why-tini
  - chromedev 6y ago
    I didn't know it was included by default. I'll check it out, thanks!
willemmerson 6y ago
Why doesn't it just use the Debian python package?
- athorax 6y ago
  I can think of a couple of possibilities: 1. Better control of specific python version. There are debian builds for python 3.5, 3.6, etc. + other more specific minor versions. 2. Better isolation from the system python
- itamarst 6y ago
  Debian Stable currently ships Python 3.7. If that's what you want, great. But if you want Python 3.8, the soon-to-be-released Python 3.9, or other versions, you don't have that.
  - Hamuko 6y ago
    >Debian Stable currently ships Python 3.7. That's surprisingly new even. I remember Debian being very behind when it came to Python3.
    - kingosticks 6y ago
      But isn't that only because it was released just last year? Won't it stay stuck on Python 3.7 until bullseye is released in approx 2 years time? It'll likely be very dated by then.
      - sgerenser 6y ago
        Next Debian stable should be released in less than a year. It’s been more than a year since Buster (current stable) was released. But yes the point stands that 3.7 will be all you get built-in for the next several months regardless of how many Python releases occur.
        kingosticks 6y ago
redis_mlc 6y ago
Can somebody post a link to the actual Dockerfile in the article? Being breadcrumbed like that is plain rude.
mikepurvis 6y ago
An interesting aspect of this that isn't touched on in the article is about the much-maligned [1] scheme where Debian's Python is patched to look for packages in `dist-packages` instead of `site-packages`. The scenario in the Docker image is I believe the exact one that this is intended to provide sanity for, since you have two Pythons in play— the Debian-packaged python and also the built-from-source Python. Things get hairy once you start using the pip from either of these pythons to start installing pypi packages, especially if those packages or their dependencies have compiled parts which may not work for the other Python. Anyone who has used a Mac with brewed Python has also likely experienced this pain— especially in the days before wheels, where you basically had to use brewed Python for scientific work unless you wanted to compile big packages like numpy from source. So the deal is that the system-supplied Python can get deps from apt packages, or from pip (into ~/.local, or as root into /usr/local), but either way will install them into a dist-packages directory. This keeps them separate from the packages which the built-from-source Python installs into site-packages. Upstream Python has put a bunch of pieces in place to address this natively, for example with the "magic" tags that keep compiled assets separate in a directory of packages being potentially used by multiple different interpreters (see https://www.python.org/dev/peps/pep-3147/#proposal https://www.python.org/dev/peps/pep-3147/#proposal). And obviously the ideal solution where possible is to simply use a virtualenv and be totally isolated from the system python. But there are situations where that isn't possible or desirable, such as in this docker container, and so here we are. 1: eg https://github.com/pypa/setuptools/issues/2232 https://github.com/pypa/setuptools/issues/2232
- Kwpolska 6y ago
  > […] simply use a virtualenv and be totally isolated from the system python. But there are situations where that isn't possible or desirable, such as in this docker container, and so here we are. Why would it not be possible or desirable to use a venv in the Docker container? The tools are available and work fine (`venv` is built into modern Python 3, and they don’t break it unlike Debian). venv will give you separation from anything unusual on the system, and let you safely install your specifically pinned requirements.txt, with zero downsides.
fizixer 6y ago
After a few years' break from professional s/w dev, I'm really trying to get into the docker-based dev mindset. You can use docker (or, less efficiently, a full-blown VM) for any purpose, but it looks like the killer app that has emerged appears to be devops. I guess same goes with k8s, as well as Chef/Puppet/Salt/Ansible/whathaveyou. However, I'm noticing there is a new way of doing s/w dev emerging, let's call it modern software development, which utilizes some of these tools to maximize s/w dev productivity. I'm just not clear what is the best way to approach this. The core issue is obviously dependency hell, and install-reinstall-reconfigure hell. I guess a useful way to think is, what if I want do to web-dev, and android-dev, and, iOS dev, but I don't want these dev environments to interfere with each other, and all these dev environments should be available accessible on a single workstatation or powerful laptop. I guess I could have docker for web-dev, docker for android-dev, and so on. I came across docker compose, and then I heard it's known to be cumbersome for dev-environments, and someone created binci to address those problems (though it's not a well-known tool).
- harpratap 6y ago
  The solution for this is to do your development in cloud directly. Projects like Skaffold, tilt and VSCode in browser are one step in this direction. Developer machines would become just thin clients and GUIs for these remote machines.
  - fizixer 6y ago
    So you mean fire up an instance at AWS/Azure/GCP/etc that acts as an android-dev vm? So dev environment is on the cloud? and all the data (working dir, data files, pdfs, whatever is involved during the dev phase) is also on the cloud storage? That would be expensive, especially if you're a indie dev, and a tremendous waste of local compute/storage capabilities. Again, I'm really not sure what near-future looks like in this space.
    - harpratap 6y ago
      Indie devs are small minority, majority of the folks work in companies. You are already paying 2000-4000USD for your laptops, plus you also have to cost in lost productivity because of not being able to replicate your stack locally. Eventbrite is already doing this for all their employees - https://kelda.io/blog/eventbrite-interview/ https://kelda.io/blog/eventbrite-interview/ So the economics of this does check out.
bluntfang 6y ago
is it really a deep dive if it doesn't go into the base image at all? edit: ok reading the article further, there are handwavy explanations. Don't call it a deep dive if you're gonna say "There’s a lot in there, but the basic outcome is:..." and then not explain anything beyond that.
Earmouse 6y ago
Why would the image remove the *.pyc files?
- acdha 6y ago
  It keeps the image size down and they’ll be recreated on first load for what you actually use. If you use the compileall module you can generate them for your app in the final layer.
dirtnugget 6y ago
I would love to read this article on a phone but the top banner is not only highly confusing but also blocking a lot of the content ... please fix this

189 points by itamarst 6y ago

> Debian names all their releases after characters from Toy Story. I can’t believe I’m just now learning this, but that’s good to know next time someone asks why the names seem random.

lub 6y ago
And sid (unstable) is the neighbor who breaks the toys, which is why the name never changes :)
- beagle3 6y ago
  Story I heard is that it started with “sid” being an acronym for “still in development”, but once that caught, it blazed the trail for the rest of the Toy Story characters....
  - fanf2 6y ago
    I heard it was because when Bruce Perens was Debian Project Leader, he was working for Pixar around the time of the Toy Story release, and chose the naming scheme. The descriptive name for unstable has always been unstable, not "still in development".
- Igelau 6y ago
  Looking back, Sid was an artist being smothered by mundane suburbia, while Andy was a milquetoast who couldn't look after his stuff.
  - mikepurvis 6y ago
    Others have noticed this too— some examples: https://www.reddit.com/r/FanTheories/comments/1qmtoi/sid_from_toy_story_wasnt_actually_a_bad_kid_he/ https://www.reddit.com/r/FanTheories/comments/1qmtoi/sid_fro... https://matthewdicks.com/film/2018-5-23-sid-from-toy-story-was-a-completely-normal-person/ https://matthewdicks.com/film/2018-5-23-sid-from-toy-story-w...
    - emmelaich 6y ago
      and https://puzzling.org/entertainment/movies-entertainment/2013/04/team-sid-why-the-antagonist-became-my-favourite/ https://puzzling.org/entertainment/movies-entertainment/2013...
umvi 6y ago
Holy cow ...! I guess the reason I never noticed was because I started using Debian around version 7, and the majority of the iconic Toy Story names (Woody, Buzz, Slink, Potato, etc.) are all pre-7.
- R0b0t1 6y ago
  Same exact thing for me. I rue the day they run out of names :(.
  - LeoPanthera 6y ago
    Plenty more characters in the sequels.
pyuser583 6y ago
Yeah - wow.
dazzawazza 6y ago
It's cute but really frustrating. I wish they'd just use the version number or force everyone to use "<version #>-<stupidname>". but I'm a FreeBSD user so what do I know?
- kemotep 6y ago
  To be fair, you can just point to the `stable` repositories in the `/etc/apt/sources.list` configuration file and simply just refer to your install as "Debian Stable" ('s/stable/testing/g'). However every few years there might be a not so "stable" update around the new release.
  - dazzawazza 6y ago
    It's not for me, it's for when I get bug reports and I need to decipher what flavour of linux, what release of the kernel, what libc etc. "I'm running decrepit duck" doesn't really narrow it down too much. Especially when multiple distributions use these jovial monikers.
    - kemotep 6y ago
      I can understand that frustration if you are used to the BSD's. Windows 10 and macOS keep it pretty simple too.
      - black3r 6y ago
        Ubuntu is the most simplest.., 20.04 means it was released in april 2020. Windows 2004 means it was finalized in april 2020, but released some times later (may? june? who knows).., plus they both try to release every 6 months and in case of ubuntu you know that even aprils are LTS, so it's always predictable to know what's the latest version. mac 10.15 means what? and the next release is gonna be 11.0. someone who doesn't follow or use macs has no idea to figure out if the latest version is 10.15 or 10.14, or 11.0...
        6y ago
Hamuko 6y ago
I think even the Deb + Ian naming is still a fairly unknown piece of knowledge.

monkpit 6y ago

> A common mistake for people using this base image is to install Python again, by using Debian’s version of Python Is that really a common mistake?

JulianWasTaken 6y ago
16,328 results just on GitHub, though it fluctuates on refresh, and some are commented out: https://github.com/search?q=%22from+python%3A%22+apt-get+install+python3-dev&type=Code https://github.com/search?q=%22from+python%3A%22+apt-get+ins... (And there's likely what to clean up in the search query to make it filter more irrelevant results). Seems like though "yes" is the answer.
digitallogic 6y ago
Whoa boy, is it ever, but maybe not for the reason you're thinking. ie it isn't caused by people typing `apt-get install python`. There are many packages that have Python as a dependency these days. For example, on my Ubuntu system: > ~$ apt-cache rdepends python|wc -l > 4649 I think the best illustration of how this can happen is installing postgres libraries needed to build the psycopg2 PG client. If you know to install `libpq-dev` then you're great. But if you do something that on the surface feels totally reasonable, like installing the `postgresql-client` package... guess what? You just installed another Python interpreter. edit: formatting
itamarst 6y ago
Beyond the other two answers, I added this bit because I keep seeing people make this mistake in their StackOverflow questions about why their Dockerfile isn't working.

claytonjy 6y ago

> The packages—gcc and so on—needed to compile Python are removed once they are no longer needed. Is there a reason to prefer this method, where installation, usage, and removal all happen in one RUN, vs. using a multi-stage build? I tend to prefer the latter but am not aware of tradeoffs beyond the readability of the Dockerfile.

vbernat 6y ago
The reason is to minimize the number of layers, mostly the layers that are not used in the final image (in this example, a layer would include gcc while another would remove it, but you would still need to download the layer with gcc).
- claytonjy 6y ago
  I thought non-final stages in a multi-stage build are left out of the final artifact; is that incorrect? I'm thinking the "build step" would be done in an earlier stage; it could be the exact same RUN statement, or it could be split into multiple for readability, and wouldn't bother removing any installed packages, since they won't carry over to the final stage. Then the big RUN in the final stage would be replaced with something like `COPY --from=builder ...`.
  - verst 6y ago
    EDIT: Provide example below If you are doing multi stage builds it only matters to combined as many statements as possible in the last layer. I agree that for clarity it is nice not to optimize layers in the build stage - those will be thrown away anyway. I vastly prefer multi stage builds over having to chain install and cleanup statements Example: I usually want to use the python:3-slim image, but this doesn't have the tools to compile certain python libraries with C extensions. Generally I will use the python:3 image for my build stage to do my "pip install -r requirements.txt" and then copy the libraries over to my final stage based on the python:3-slim image Of course I could install and uninstall GCC and other tools in a single stage.. but that actually takes longer to do and is messier in my opinion.

westurner 6y ago

There are Alpine [1] and Debian [2] miniconda images (within which you can `conda install python==3.8` and 2.7 and 3.4 in different conda envs) [1] https://github.com/ContinuumIO/docker-images/blob/master/miniconda3/alpine/Dockerfile https://github.com/ContinuumIO/docker-images/blob/master/min... [2] https://github.com/ContinuumIO/docker-images/blob/master/miniconda3/debian/Dockerfile https://github.com/ContinuumIO/docker-images/blob/master/min... If you build manylinux wheels with auditwheel [3], they should install without needing compilation for {CentOS, Debian, Ubuntu, and Alpine}; though standard Alpine images have MUSL instead of glibc by default, this [4] may work: echo "manylinux1_compatible = True" > $PYTHON_PATH/_manylinux.py [3] https://github.com/pypa/auditwheel https://github.com/pypa/auditwheel [4] https://github.com/docker-library/docs/issues/904#issuecomment-544181321 https://github.com/docker-library/docs/issues/904#issuecomme... The miniforge docker images aren't yet [5][6] multi-arch, which means it's not as easy to take advantage of all of the ARM64 / aarch64 packages that conda-forge builds now. [5] https://github.com/conda-forge/docker-images/issues/102#issuecomment-619963279 https://github.com/conda-forge/docker-images/issues/102#issu... [6] https://github.com/conda-forge/miniforge/issues/20 https://github.com/conda-forge/miniforge/issues/20 There are i686 and x86-64 docker containers for building manylinux wheels that work with many distros: https://github.com/pypa/manylinux/tree/master/docker https://github.com/pypa/manylinux/tree/master/docker A multi-stage Dockerfile build can produce a wheel in the first stage and install that wheel (with `COPY --from=0`) in a later stage; leaving build dependencies out of the production environment for security and performance: https://docs.docker.com/develop/develop-images/multistage-build/ https://docs.docker.com/develop/develop-images/multistage-bu...

athorax 6y ago
Interesting! I use miniconda extensively for local development to manage virtual environments for different python versions and love it. I hardly ever actually use the conda packages though. I assume the main benefit of using these images would be if you are installing from conda repos instead of pip? Otherwise just using the official python images would be as good if not better Edit: I guess if you needed multiple python versions in a single container this would be a good solution for that as well

chromedev 6y ago

One thing I've learned is that rather than use a docker-entrypoint.sh, most Linux software can be ran just using the `--user 1000:1000` or whatever UID/GID you want to use, as long as you map a volume that can use those permissions. It is a lot cleaner this way.

westurner 6y ago
> Why Tini? > Using Tini has several benefits: > - It protects you from software that accidentally creates zombie processes, which can (over time!) starve your entire system for PIDs (and make it unusable). > - It ensures that the default signal handlers work for the software you run in your Docker image. For example, with Tini, SIGTERM properly terminates your process even if you didn't explicitly install a signal handler for it. > - It does so completely transparently! Docker images that work without Tini will work with Tini without any changes. [...] > NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just pass the `--init` flag to docker run. https://github.com/krallin/tini#why-tini https://github.com/krallin/tini#why-tini
- chromedev 6y ago
  I didn't know it was included by default. I'll check it out, thanks!

willemmerson 6y ago

Why doesn't it just use the Debian python package?

athorax 6y ago
I can think of a couple of possibilities: 1. Better control of specific python version. There are debian builds for python 3.5, 3.6, etc. + other more specific minor versions. 2. Better isolation from the system python
itamarst 6y ago
Debian Stable currently ships Python 3.7. If that's what you want, great. But if you want Python 3.8, the soon-to-be-released Python 3.9, or other versions, you don't have that.
- Hamuko 6y ago
  >Debian Stable currently ships Python 3.7. That's surprisingly new even. I remember Debian being very behind when it came to Python3.
  - kingosticks 6y ago
    But isn't that only because it was released just last year? Won't it stay stuck on Python 3.7 until bullseye is released in approx 2 years time? It'll likely be very dated by then.
    - sgerenser 6y ago
      Next Debian stable should be released in less than a year. It’s been more than a year since Buster (current stable) was released. But yes the point stands that 3.7 will be all you get built-in for the next several months regardless of how many Python releases occur.
      - kingosticks 6y ago

redis_mlc 6y ago

Can somebody post a link to the actual Dockerfile in the article? Being breadcrumbed like that is plain rude.

mikepurvis 6y ago

An interesting aspect of this that isn't touched on in the article is about the much-maligned [1] scheme where Debian's Python is patched to look for packages in `dist-packages` instead of `site-packages`. The scenario in the Docker image is I believe the exact one that this is intended to provide sanity for, since you have two Pythons in play— the Debian-packaged python and also the built-from-source Python. Things get hairy once you start using the pip from either of these pythons to start installing pypi packages, especially if those packages or their dependencies have compiled parts which may not work for the other Python. Anyone who has used a Mac with brewed Python has also likely experienced this pain— especially in the days before wheels, where you basically had to use brewed Python for scientific work unless you wanted to compile big packages like numpy from source. So the deal is that the system-supplied Python can get deps from apt packages, or from pip (into ~/.local, or as root into /usr/local), but either way will install them into a dist-packages directory. This keeps them separate from the packages which the built-from-source Python installs into site-packages. Upstream Python has put a bunch of pieces in place to address this natively, for example with the "magic" tags that keep compiled assets separate in a directory of packages being potentially used by multiple different interpreters (see https://www.python.org/dev/peps/pep-3147/#proposal https://www.python.org/dev/peps/pep-3147/#proposal). And obviously the ideal solution where possible is to simply use a virtualenv and be totally isolated from the system python. But there are situations where that isn't possible or desirable, such as in this docker container, and so here we are. 1: eg https://github.com/pypa/setuptools/issues/2232 https://github.com/pypa/setuptools/issues/2232

Kwpolska 6y ago
> […] simply use a virtualenv and be totally isolated from the system python. But there are situations where that isn't possible or desirable, such as in this docker container, and so here we are. Why would it not be possible or desirable to use a venv in the Docker container? The tools are available and work fine (`venv` is built into modern Python 3, and they don’t break it unlike Debian). venv will give you separation from anything unusual on the system, and let you safely install your specifically pinned requirements.txt, with zero downsides.

fizixer 6y ago

After a few years' break from professional s/w dev, I'm really trying to get into the docker-based dev mindset. You can use docker (or, less efficiently, a full-blown VM) for any purpose, but it looks like the killer app that has emerged appears to be devops. I guess same goes with k8s, as well as Chef/Puppet/Salt/Ansible/whathaveyou. However, I'm noticing there is a new way of doing s/w dev emerging, let's call it modern software development, which utilizes some of these tools to maximize s/w dev productivity. I'm just not clear what is the best way to approach this. The core issue is obviously dependency hell, and install-reinstall-reconfigure hell. I guess a useful way to think is, what if I want do to web-dev, and android-dev, and, iOS dev, but I don't want these dev environments to interfere with each other, and all these dev environments should be available accessible on a single workstatation or powerful laptop. I guess I could have docker for web-dev, docker for android-dev, and so on. I came across docker compose, and then I heard it's known to be cumbersome for dev-environments, and someone created binci to address those problems (though it's not a well-known tool).

harpratap 6y ago
The solution for this is to do your development in cloud directly. Projects like Skaffold, tilt and VSCode in browser are one step in this direction. Developer machines would become just thin clients and GUIs for these remote machines.
- fizixer 6y ago
  So you mean fire up an instance at AWS/Azure/GCP/etc that acts as an android-dev vm? So dev environment is on the cloud? and all the data (working dir, data files, pdfs, whatever is involved during the dev phase) is also on the cloud storage? That would be expensive, especially if you're a indie dev, and a tremendous waste of local compute/storage capabilities. Again, I'm really not sure what near-future looks like in this space.
  - harpratap 6y ago
    Indie devs are small minority, majority of the folks work in companies. You are already paying 2000-4000USD for your laptops, plus you also have to cost in lost productivity because of not being able to replicate your stack locally. Eventbrite is already doing this for all their employees - https://kelda.io/blog/eventbrite-interview/ https://kelda.io/blog/eventbrite-interview/ So the economics of this does check out.

bluntfang 6y ago

is it really a deep dive if it doesn't go into the base image at all? edit: ok reading the article further, there are handwavy explanations. Don't call it a deep dive if you're gonna say "There’s a lot in there, but the basic outcome is:..." and then not explain anything beyond that.

Earmouse 6y ago

Why would the image remove the *.pyc files?

acdha 6y ago
It keeps the image size down and they’ll be recreated on first load for what you actually use. If you use the compileall module you can generate them for your app in the final layer.

dirtnugget 6y ago

I would love to read this article on a phone but the top banner is not only highly confusing but also blocking a lot of the content ... please fix this