6 ms·

I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TI

I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TIMTOWTDI). As a tongue-in-cheek reaction, Python espoused "TOOWTDI", or There's Only One Way To Do It :) reference: https://wiki.python.org/moin/TOOWTDI https://wiki.python.org/moin/TOOWTDI The problem is, when it comes to the Python package management ecosystem, there are SO MANY ways to do it. And they aren't equal and require a deep understanding of each tool to select the correct choice. It is a problem for Python, although I often regrettably see it dismissed as not a true issue because $latest_package_management_system fixes it. Python has allowed me to be profoundly productive in many ways, but this is a huge sore spot for the Python ecosystem.

xdfgh1112 4y ago
I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?
- bloblaw 4y ago
  The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. Python scripting today requires shipping your development environment. Python is wonderful until you want to run that code on another machine. At that point, the target system has to venv their way into reproducing your environment...often including the specific Python interpreter you picked, and to download (and possibly compile) all modules and their dependencies. It can work beautifully and many of the large companies I've worked for have put oodles of engineering effort into making it "easy", as long as you follow their happy path and don't deviate. But, to your point, on my own machine with venv + pip it "just works". The pain comes when trying to venv + pip on another machine. How confident are you that you could venv + pip your moderately complex Python application on 200k machines without issue? What if it's a mix of Windows, Linux, and macOS? (this is a real scenario I've experienced). From my own experience I can share that it is painful. Your mileage may vary. And after a while your system is riddled with Python venvs. Leading to some mildly annoying problems with popular IDE's: https://www.jetbrains.com/help/pycharm/package-installation-issues.html https://www.jetbrains.com/help/pycharm/package-installation-...
  - elbigbad 4y ago
    Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.
    - behnamoh 4y ago
      I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?
      - deleted 4y ago
        [deleted]
      - bornfreddy 4y ago
        Yes, docker solves all these issues (because your app carries all of its environment in the image), however it is a bit cumbersome if you want to distribute your app to many unknown users. They can't just download something and run it, as Windows users are used to doing.
      - bloblaw 4y ago
        Docker simultaneously solves and creates problems here. Step 1: Docker must be installed on all targets. This is not a given and is a new piece of overhead. Step 2: The Dockerfile hopefully doesn't source from just "ubuntu:latest" and bring in the whole kitchen sink for this SINGLE APPLICATION. Effectively if you are using Docker and deploying your script to Windows or macOS, you are saying "Hey, this script requires you to install linux in a VM to run this. Docker makes it easy. Go download 500mb of Docker and a couple hundred more mb of images to run this 800k script.
  - andreareina 4y ago
    Ansible is a special example of this. Some Ansible modules (e.g. the postgresql one) require non-stdlib modules to be installed in the remote python. So either install it globally or figure out how to make Ansible work with environments. Even if the thing you're trying to set up ostensibly doesn't use python. sigh
- jjoonathan 4y ago
  Conda. Back in the day, before docker, it was a poor-man's docker combined with a repository of significantly newer and more compatible software than you could get from distro repositories. Python libraries with external dependencies used it heavily for this reason. Conda tended to Just Work in a bunch of cases where pip effectively forced you to go off and debug a bunch of autotools builds on your own. Yuck! However, it was pushy. It would put its own header/linker paths in front of the system paths (this is how it made "environments," its wannabe containers), which tended to create inadvertent cross dependencies if you didn't understand or didn't remember that the semantics of a conda environment extended beyond python. These dependencies could get baked into binaries and break far down the road, or they could get sucked in as a transitive dependency and trip over the shoelaces of a different build of the same software installed outside conda. Unfortunate. However, around 2019, the problems started growing beyond mere foot-guns. Conda uses a full SAT solver to provide a highly featured versioning system, and this worked great until the big conda channels grew to the point that it started getting really slow. Installing packages went from taking seconds to minutes to hours to forever. They tried caching, they tried fragmenting channels, but it was all very not-seamless. Eventually, people started migrating back to pip. It turns out that over the last decade distro repositories had gotten their shit together and now Docker existed to sweep up the last few use cases, so nobody needed conda's "poor-man's docker plus curated 3rd party repos" anymore. Now pip is the tool that Just Works, and it Just Works without any of conda's baggage. Virtualenv environments don't hook your system quite as aggressively, pip never stalls when resolving its version plans, and Docker can be used to reproducibly experiment and find the happy path. Be glad that you missed out on pre-conda pip and the conda arc.
- xdennis 4y ago
  > I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing? Compare this with Node. It always installs locally by default and always installs in node_modules regardless of which package manager you use. This is integrated into Node so you never need to modify the path like in Python. You don't have to guess whether your packages are installed in .venv, env, environ, or whatever someone else decides. `pip` actually has a lot of issues with regards to deciding which version to use. That's why people moved to pipenv... then pipenv stagnated and people moved to poetry. Using `requirements.txt` is dead simple but it ignores issues such as version locking. If you just add the packages you need to `requirements.txt`, then every time you install you could get a different set of packages. If you do `pip freeze > requirements.txt` then you don't know what comes from what.

xdfgh1112 4y ago

I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?

bloblaw 4y ago
The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. Python scripting today requires shipping your development environment. Python is wonderful until you want to run that code on another machine. At that point, the target system has to venv their way into reproducing your environment...often including the specific Python interpreter you picked, and to download (and possibly compile) all modules and their dependencies. It can work beautifully and many of the large companies I've worked for have put oodles of engineering effort into making it "easy", as long as you follow their happy path and don't deviate. But, to your point, on my own machine with venv + pip it "just works". The pain comes when trying to venv + pip on another machine. How confident are you that you could venv + pip your moderately complex Python application on 200k machines without issue? What if it's a mix of Windows, Linux, and macOS? (this is a real scenario I've experienced). From my own experience I can share that it is painful. Your mileage may vary. And after a while your system is riddled with Python venvs. Leading to some mildly annoying problems with popular IDE's: https://www.jetbrains.com/help/pycharm/package-installation-issues.html https://www.jetbrains.com/help/pycharm/package-installation-...
- elbigbad 4y ago
  Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.
  - behnamoh 4y ago
    I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?
    - deleted 4y ago
      [deleted]
    - bornfreddy 4y ago
      Yes, docker solves all these issues (because your app carries all of its environment in the image), however it is a bit cumbersome if you want to distribute your app to many unknown users. They can't just download something and run it, as Windows users are used to doing.
    - bloblaw 4y ago
      Docker simultaneously solves and creates problems here. Step 1: Docker must be installed on all targets. This is not a given and is a new piece of overhead. Step 2: The Dockerfile hopefully doesn't source from just "ubuntu:latest" and bring in the whole kitchen sink for this SINGLE APPLICATION. Effectively if you are using Docker and deploying your script to Windows or macOS, you are saying "Hey, this script requires you to install linux in a VM to run this. Docker makes it easy. Go download 500mb of Docker and a couple hundred more mb of images to run this 800k script.
- andreareina 4y ago
  Ansible is a special example of this. Some Ansible modules (e.g. the postgresql one) require non-stdlib modules to be installed in the remote python. So either install it globally or figure out how to make Ansible work with environments. Even if the thing you're trying to set up ostensibly doesn't use python. sigh
jjoonathan 4y ago
Conda. Back in the day, before docker, it was a poor-man's docker combined with a repository of significantly newer and more compatible software than you could get from distro repositories. Python libraries with external dependencies used it heavily for this reason. Conda tended to Just Work in a bunch of cases where pip effectively forced you to go off and debug a bunch of autotools builds on your own. Yuck! However, it was pushy. It would put its own header/linker paths in front of the system paths (this is how it made "environments," its wannabe containers), which tended to create inadvertent cross dependencies if you didn't understand or didn't remember that the semantics of a conda environment extended beyond python. These dependencies could get baked into binaries and break far down the road, or they could get sucked in as a transitive dependency and trip over the shoelaces of a different build of the same software installed outside conda. Unfortunate. However, around 2019, the problems started growing beyond mere foot-guns. Conda uses a full SAT solver to provide a highly featured versioning system, and this worked great until the big conda channels grew to the point that it started getting really slow. Installing packages went from taking seconds to minutes to hours to forever. They tried caching, they tried fragmenting channels, but it was all very not-seamless. Eventually, people started migrating back to pip. It turns out that over the last decade distro repositories had gotten their shit together and now Docker existed to sweep up the last few use cases, so nobody needed conda's "poor-man's docker plus curated 3rd party repos" anymore. Now pip is the tool that Just Works, and it Just Works without any of conda's baggage. Virtualenv environments don't hook your system quite as aggressively, pip never stalls when resolving its version plans, and Docker can be used to reproducibly experiment and find the happy path. Be glad that you missed out on pre-conda pip and the conda arc.
xdennis 4y ago
> I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing? Compare this with Node. It always installs locally by default and always installs in node_modules regardless of which package manager you use. This is integrated into Node so you never need to modify the path like in Python. You don't have to guess whether your packages are installed in .venv, env, environ, or whatever someone else decides. `pip` actually has a lot of issues with regards to deciding which version to use. That's why people moved to pipenv... then pipenv stagnated and people moved to poetry. Using `requirements.txt` is dead simple but it ignores issues such as version locking. If you just add the packages you need to `requirements.txt`, then every time you install you could get a different set of packages. If you do `pip freeze > requirements.txt` then you don't know what comes from what.