4 ms·

Develop in a container. It will remove all other issues from your machine/environment and let you declaratively specify the exact environment you need (i.e. pa

Develop in a container. It will remove all other issues from your machine/environment and let you declaratively specify the exact environment you need (i.e. packages to install, version of python, etc). Forget all about venv and all the other tools in the python ecosystem--in your container there is _one_ python and you control all of its dependencies.

nawgz 4y ago
Pragmatic advice, certainly, but maybe it's missing the point - you shouldn't need a clean OS every time you want to use a new language version. That's a ridiculously large failure of packaging.
- qbasic_forever 4y ago
  It's core to the design of python right now. Remember when python was designed in the 90s people had big multi-user Unix systems that were bespoke pets. It was assumed that python would be a system-level tool and all packages, etc. would be installed at the system level. Your Python install would be a living thing that was upgraded and administrated like any other software on the machine. Fast forward 30 years and we really don't use most systems like that anymore, especially in production environments. We run things in VMs or containers and build them up from scratch with ease--it's all just cattle. Python hasn't really adapted to that new reality.
  - dvdkon 4y ago
    The fact that many popular Python packages rely on system-level native tools also doesn't help. "Solving Python packaging" isn't just about distributing portable Python source code, that's easy. It's also about packaging and distributing unruly non-Python software for a myriad different platforms, all without user intervention. Sure, the solution's not great, but it's a really hard problem.
  - nonethewiser 4y ago
    Why aren’t virtual environments enough?
    - scruple 4y ago
      I want to hear the answer to this, too. I'm back around to Python professionally (haven't touched it since 2016) and I'm working with the basic tooling: pip, requirements.txt, and virtualenv. I'd like to know what sort of issues I'm going to run into and when I can expect them.
      - mixmastamyk 4y ago
        They are enough, just need care and sysad knowledge which are in short supply these days. Docker is an end-run around that, but you'll then have to know it as well.
    - qbasic_forever 4y ago
      You have to remember to activate them. Node/npm has a similar concept with its scripts execution, but it automatically runs them in the local node_modules environment (i.e. virtualenv). You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
      - wswope 4y ago
        It is silly. You put together a three-line bash script to activate the venv, run a pip install, and call your program. The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment https://packaging.python.org/en/latest/guides/installing-usi...
        jakewins 4y ago
        But the problem isn't my python environment - the problem is distributing packages, particularly libraries, to downstream users. That one over there uses virtualenvs and a bash script she wrote, that one just ran "pip" and has 4 different system python installs entangled into each other; that one ran `pipenv install`, that one `poetry add`. How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
  - bb88 4y ago
    You can do that today. You're not forced to run in a virtualenv if you don't want. Install the python as a system package, and then sudo pip install your way to happiness. That's how our production containers are built.
  - varjag 4y ago
    Other languages manage somehow. Python also could perhaps, if not the cultural dysfunction.
- morkalork 4y ago
  I work with a bunch of data scientists and they manage to go about 1.5 years between "I've totally fucked my Python environment and have to burn it down completely and reinstall it from scratch" events. So yes, I would agree with your statement about Python's packaging system being a ridiculous failure.
- Gordonjcp 4y ago
  I disagree. It doesn't matter which language you're using, any non-trivial code you have is going to have dependencies which you will need to pull in. If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" problems that used to plague people.
  - nawgz 4y ago
    > I disagree You disagree that "you shouldn't need a clean OS every time you want to use a new language version"!? ... I don't even know what to say, that's not something you can disagree with > If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" This is what testing is for. You don't need to cripple your development environment to have confidence your code works in other environments. Talk about the ends not justifying the means.
    - Gordonjcp 4y ago
      Can you explain why having a reliable, repeatable and well-tested development environment is "crippling" it?
      - nawgz 4y ago
        Because it's trivial to accomplish these things in other languages without containers?
monkellipse 4y ago
This was my solution. It feels like overkill at times but I never have to worry about packaging messes anymore. That being said I can understand the desire to solve it on the metal, it’s a mess.
pdpi 4y ago
I thoroughly resent that I'm being forced into that workflow. I'm trying to get enough distance to tell if it's actually overkill or just being a curmodgeonly old git.
- qbasic_forever 4y ago
  Well your alternative is to use a VM, which is a slower and clunkier container, or fight the myriad of bespoke python environment management tools so you can make one Python install work for X number of projects and all their unique dependencies.
  - pdpi 4y ago
    I meant in general, not for Python specifically (which I don't use nearly enough for it to be a problem). On the one hand, it bothers me that software is so brittle that you need a container around it so that everything is just right. On the other hand, you can argue that containerisation is a clever way to make it all more robust.
    - darepublic 4y ago
      Try it out you may find it makes you more productive then the old way of doing things
    - nerpderp82 4y ago
      The issue is with the OS and how dynamic linking works (or doesnt). The solution isn't for the maintainer of the dependencies to "try harder" or be better at what they do. The underlying system is broken and containerization is way to compartmentalize those flaws so they are less destructive. If you put everything into an Uber Container, the same problem would surface. Containers exist to solve the fragile dependency/dynamic linking problem.
- pharmakom 4y ago
  Agree. We don't need to use containers so often in other language stacks. this is a failure of the python ecosystem.
bick_nyers 4y ago
When I was first learning python/tensorflow I tried using docker containers, but got a huge headache trying to navigate using a GPU with a container. I know VMs also can't really share GPU resources that effectively (at least consumer cards in pedestrian setups). I built this resistance to docker about 5 years ago, is it (or was it) still correct? Is there a best practice for sharing GPUs with containers that's not a pain in the ass? Edit: I should specify that I typically develop on Ubuntu, but occasionally will do so on Windows (without WSL).

nawgz 4y ago

Pragmatic advice, certainly, but maybe it's missing the point - you shouldn't need a clean OS every time you want to use a new language version. That's a ridiculously large failure of packaging.

qbasic_forever 4y ago
It's core to the design of python right now. Remember when python was designed in the 90s people had big multi-user Unix systems that were bespoke pets. It was assumed that python would be a system-level tool and all packages, etc. would be installed at the system level. Your Python install would be a living thing that was upgraded and administrated like any other software on the machine. Fast forward 30 years and we really don't use most systems like that anymore, especially in production environments. We run things in VMs or containers and build them up from scratch with ease--it's all just cattle. Python hasn't really adapted to that new reality.
- dvdkon 4y ago
  The fact that many popular Python packages rely on system-level native tools also doesn't help. "Solving Python packaging" isn't just about distributing portable Python source code, that's easy. It's also about packaging and distributing unruly non-Python software for a myriad different platforms, all without user intervention. Sure, the solution's not great, but it's a really hard problem.
- nonethewiser 4y ago
  Why aren’t virtual environments enough?
  - scruple 4y ago
    I want to hear the answer to this, too. I'm back around to Python professionally (haven't touched it since 2016) and I'm working with the basic tooling: pip, requirements.txt, and virtualenv. I'd like to know what sort of issues I'm going to run into and when I can expect them.
    - mixmastamyk 4y ago
      They are enough, just need care and sysad knowledge which are in short supply these days. Docker is an end-run around that, but you'll then have to know it as well.
  - qbasic_forever 4y ago
    You have to remember to activate them. Node/npm has a similar concept with its scripts execution, but it automatically runs them in the local node_modules environment (i.e. virtualenv). You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
    - wswope 4y ago
      It is silly. You put together a three-line bash script to activate the venv, run a pip install, and call your program. The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment https://packaging.python.org/en/latest/guides/installing-usi...
      - jakewins 4y ago
        But the problem isn't my python environment - the problem is distributing packages, particularly libraries, to downstream users. That one over there uses virtualenvs and a bash script she wrote, that one just ran "pip" and has 4 different system python installs entangled into each other; that one ran `pipenv install`, that one `poetry add`. How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
- bb88 4y ago
  You can do that today. You're not forced to run in a virtualenv if you don't want. Install the python as a system package, and then sudo pip install your way to happiness. That's how our production containers are built.
- varjag 4y ago
  Other languages manage somehow. Python also could perhaps, if not the cultural dysfunction.
morkalork 4y ago
I work with a bunch of data scientists and they manage to go about 1.5 years between "I've totally fucked my Python environment and have to burn it down completely and reinstall it from scratch" events. So yes, I would agree with your statement about Python's packaging system being a ridiculous failure.
Gordonjcp 4y ago
I disagree. It doesn't matter which language you're using, any non-trivial code you have is going to have dependencies which you will need to pull in. If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" problems that used to plague people.
- nawgz 4y ago
  > I disagree You disagree that "you shouldn't need a clean OS every time you want to use a new language version"!? ... I don't even know what to say, that's not something you can disagree with > If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" This is what testing is for. You don't need to cripple your development environment to have confidence your code works in other environments. Talk about the ends not justifying the means.
  - Gordonjcp 4y ago
    Can you explain why having a reliable, repeatable and well-tested development environment is "crippling" it?
    - nawgz 4y ago
      Because it's trivial to accomplish these things in other languages without containers?

monkellipse 4y ago

This was my solution. It feels like overkill at times but I never have to worry about packaging messes anymore. That being said I can understand the desire to solve it on the metal, it’s a mess.

pdpi 4y ago

I thoroughly resent that I'm being forced into that workflow. I'm trying to get enough distance to tell if it's actually overkill or just being a curmodgeonly old git.

qbasic_forever 4y ago
Well your alternative is to use a VM, which is a slower and clunkier container, or fight the myriad of bespoke python environment management tools so you can make one Python install work for X number of projects and all their unique dependencies.
- pdpi 4y ago
  I meant in general, not for Python specifically (which I don't use nearly enough for it to be a problem). On the one hand, it bothers me that software is so brittle that you need a container around it so that everything is just right. On the other hand, you can argue that containerisation is a clever way to make it all more robust.
  - darepublic 4y ago
    Try it out you may find it makes you more productive then the old way of doing things
  - nerpderp82 4y ago
    The issue is with the OS and how dynamic linking works (or doesnt). The solution isn't for the maintainer of the dependencies to "try harder" or be better at what they do. The underlying system is broken and containerization is way to compartmentalize those flaws so they are less destructive. If you put everything into an Uber Container, the same problem would surface. Containers exist to solve the fragile dependency/dynamic linking problem.
pharmakom 4y ago
Agree. We don't need to use containers so often in other language stacks. this is a failure of the python ecosystem.

bick_nyers 4y ago

When I was first learning python/tensorflow I tried using docker containers, but got a huge headache trying to navigate using a GPU with a container. I know VMs also can't really share GPU resources that effectively (at least consumer cards in pedestrian setups). I built this resistance to docker about 5 years ago, is it (or was it) still correct? Is there a best practice for sharing GPUs with containers that's not a pain in the ass? Edit: I should specify that I typically develop on Ubuntu, but occasionally will do so on Windows (without WSL).