5 ms·

It's core to the design of python right now. Remember when python was designed in the 90s people had big multi-user Unix systems that were bespoke pets. It wa

It's core to the design of python right now. Remember when python was designed in the 90s people had big multi-user Unix systems that were bespoke pets. It was assumed that python would be a system-level tool and all packages, etc. would be installed at the system level. Your Python install would be a living thing that was upgraded and administrated like any other software on the machine. Fast forward 30 years and we really don't use most systems like that anymore, especially in production environments. We run things in VMs or containers and build them up from scratch with ease--it's all just cattle. Python hasn't really adapted to that new reality.

dvdkon 4y ago
The fact that many popular Python packages rely on system-level native tools also doesn't help. "Solving Python packaging" isn't just about distributing portable Python source code, that's easy. It's also about packaging and distributing unruly non-Python software for a myriad different platforms, all without user intervention. Sure, the solution's not great, but it's a really hard problem.
nonethewiser 4y ago
Why aren’t virtual environments enough?
- scruple 4y ago
  I want to hear the answer to this, too. I'm back around to Python professionally (haven't touched it since 2016) and I'm working with the basic tooling: pip, requirements.txt, and virtualenv. I'd like to know what sort of issues I'm going to run into and when I can expect them.
  - mixmastamyk 4y ago
    They are enough, just need care and sysad knowledge which are in short supply these days. Docker is an end-run around that, but you'll then have to know it as well.
- qbasic_forever 4y ago
  You have to remember to activate them. Node/npm has a similar concept with its scripts execution, but it automatically runs them in the local node_modules environment (i.e. virtualenv). You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
  - wswope 4y ago
    It is silly. You put together a three-line bash script to activate the venv, run a pip install, and call your program. The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment https://packaging.python.org/en/latest/guides/installing-usi...
    - jakewins 4y ago
      But the problem isn't my python environment - the problem is distributing packages, particularly libraries, to downstream users. That one over there uses virtualenvs and a bash script she wrote, that one just ran "pip" and has 4 different system python installs entangled into each other; that one ran `pipenv install`, that one `poetry add`. How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
      - wswope 4y ago
        Did you mean to respond here? This is a nonsequitor from the GP I was responding to. Regardless: If your question is "how can I ensure technical end-users have the same set of python packages as I do for the code they run using standard pip+venv?", the answer is to pin dependencies in a requirements.txt file. If your question is "how do I stop end users from installing dependencies for my software cowboy-style?", the answer is to write installation and usage instructions, and/or include an AIO run script. If your question is "how do I package my library so that end users' package managers know my library's downstream dependencies when they install it?", you build a wheel using `pip wheel`, which again relies on a requirements.txt. If I'm understanding you right, you're mistaken that you have to handle the package managers separately; they all use pip + wheels under the hood. Conda is a bit of an asterisk in that you can package things differently if you desire, but it plays nice with pip + wheel builds too. https://docs.conda.io/projects/conda-build/en/latest/user-guide/wheel-files.html https://docs.conda.io/projects/conda-build/en/latest/user-gu...
bb88 4y ago
You can do that today. You're not forced to run in a virtualenv if you don't want. Install the python as a system package, and then sudo pip install your way to happiness. That's how our production containers are built.
varjag 4y ago
Other languages manage somehow. Python also could perhaps, if not the cultural dysfunction.

dvdkon 4y ago

The fact that many popular Python packages rely on system-level native tools also doesn't help. "Solving Python packaging" isn't just about distributing portable Python source code, that's easy. It's also about packaging and distributing unruly non-Python software for a myriad different platforms, all without user intervention. Sure, the solution's not great, but it's a really hard problem.

nonethewiser 4y ago

Why aren’t virtual environments enough?

scruple 4y ago
I want to hear the answer to this, too. I'm back around to Python professionally (haven't touched it since 2016) and I'm working with the basic tooling: pip, requirements.txt, and virtualenv. I'd like to know what sort of issues I'm going to run into and when I can expect them.
- mixmastamyk 4y ago
  They are enough, just need care and sysad knowledge which are in short supply these days. Docker is an end-run around that, but you'll then have to know it as well.
qbasic_forever 4y ago
You have to remember to activate them. Node/npm has a similar concept with its scripts execution, but it automatically runs them in the local node_modules environment (i.e. virtualenv). You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
- wswope 4y ago
  It is silly. You put together a three-line bash script to activate the venv, run a pip install, and call your program. The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment https://packaging.python.org/en/latest/guides/installing-usi...
  - jakewins 4y ago
    But the problem isn't my python environment - the problem is distributing packages, particularly libraries, to downstream users. That one over there uses virtualenvs and a bash script she wrote, that one just ran "pip" and has 4 different system python installs entangled into each other; that one ran `pipenv install`, that one `poetry add`. How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
    - wswope 4y ago
      Did you mean to respond here? This is a nonsequitor from the GP I was responding to. Regardless: If your question is "how can I ensure technical end-users have the same set of python packages as I do for the code they run using standard pip+venv?", the answer is to pin dependencies in a requirements.txt file. If your question is "how do I stop end users from installing dependencies for my software cowboy-style?", the answer is to write installation and usage instructions, and/or include an AIO run script. If your question is "how do I package my library so that end users' package managers know my library's downstream dependencies when they install it?", you build a wheel using `pip wheel`, which again relies on a requirements.txt. If I'm understanding you right, you're mistaken that you have to handle the package managers separately; they all use pip + wheels under the hood. Conda is a bit of an asterisk in that you can package things differently if you desire, but it plays nice with pip + wheel builds too. https://docs.conda.io/projects/conda-build/en/latest/user-guide/wheel-files.html https://docs.conda.io/projects/conda-build/en/latest/user-gu...

bb88 4y ago

You can do that today. You're not forced to run in a virtualenv if you don't want. Install the python as a system package, and then sudo pip install your way to happiness. That's how our production containers are built.

varjag 4y ago

Other languages manage somehow. Python also could perhaps, if not the cultural dysfunction.