8 ms·

Python is a fun language but the ecosystem around it is horrible. It's a shame. I just want to pip install like I would a package manager

newuser4321 4y ago
I think you mean the package management rather than the ecosystem. For all its failings, the python ecosystem (for machine learning / data science in particular) is unmatched, which is why python is so popular. As a "professional" python user (who got there in a roundabout way), I'd say the biggest problem with package management is all the conflicting advice and different ways to accomplish the same thing (there are other problems, but they are surmountable). Once you get a workflow down, it ends up being pretty easy to set up and manage an environment for a given project. Not that it's perfect, but I think a lot of the stereotypes about how bad it is come from how bad it appears as you try to converge on a setup that works for you.
- bloblaw 4y ago
  I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TIMTOWTDI). As a tongue-in-cheek reaction, Python espoused "TOOWTDI", or There's Only One Way To Do It :) reference: https://wiki.python.org/moin/TOOWTDI https://wiki.python.org/moin/TOOWTDI The problem is, when it comes to the Python package management ecosystem, there are SO MANY ways to do it. And they aren't equal and require a deep understanding of each tool to select the correct choice. It is a problem for Python, although I often regrettably see it dismissed as not a true issue because $latest_package_management_system fixes it. Python has allowed me to be profoundly productive in many ways, but this is a huge sore spot for the Python ecosystem.
  - xdfgh1112 4y ago
    I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?
    - bloblaw 4y ago
      The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. Python scripting today requires shipping your development environment. Python is wonderful until you want to run that code on another machine. At that point, the target system has to venv their way into reproducing your environment...often including the specific Python interpreter you picked, and to download (and possibly compile) all modules and their dependencies. It can work beautifully and many of the large companies I've worked for have put oodles of engineering effort into making it "easy", as long as you follow their happy path and don't deviate. But, to your point, on my own machine with venv + pip it "just works". The pain comes when trying to venv + pip on another machine. How confident are you that you could venv + pip your moderately complex Python application on 200k machines without issue? What if it's a mix of Windows, Linux, and macOS? (this is a real scenario I've experienced). From my own experience I can share that it is painful. Your mileage may vary. And after a while your system is riddled with Python venvs. Leading to some mildly annoying problems with popular IDE's: https://www.jetbrains.com/help/pycharm/package-installation-issues.html https://www.jetbrains.com/help/pycharm/package-installation-...
      - elbigbad 4y ago
        Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.
        behnamoh 4y ago
        I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?
        deleted 4y ago
- jmcgough 4y ago
  As someone two months into their first python job - the ecosystem is solid for ML / data science, but there's a lot of places where it's painfully lacking. I hate the most popular ORM (sqlalchemy) and alembic has a lot of footguns: for instance, if you autogenerate a migration where you change a table name, it will try to drop the old table and create a new one. In web dev a lot of the OSS community has moved on to more appropriate or exciting languages, so the tools I'd normally reach for frequently are OSS projects that haven't been maintained in seven years. As for pip: it needs a major version bump that creates a lockfile, or it needs to be intentionally sunset. It's easy for a beginner to start using (part of why it's heavily used), but extremely dangerous because of the lack of guarantees. I wish it didn't have as big of a footprint as it does.
  - wombatpm 4y ago
    Much prefer the Django ORM, which is a pain to use if you are not building web apps
    - mickeyk 4y ago
      I've used peewee in the past which worked well.
  - m_mueller 4y ago
    agreed on the lockfile. we use pip-tools to achieve this, which gives the `pip-compile` command. a Makefile is tying together the different development and build workflows we have, such that it can be executed both locally as well as in our CICD build server. complexity is manageable, but it's roughly comparable to a compiled language if you want locked-down reproducible builds (which I recommend for anything production grade).
- eslaught 4y ago
  To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sigh, these are usually the ones I get stuck working with, due to outside constraints). A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? It depends on which one of the multitude of options you pick. But at least a couple of the most popular ones basically don't do this (pip) or do this in a hacky, ugly way that makes you want to tear your hair out (Conda). As a result, at least in the projects I work on, people tend to skip this. I hope it should be obvious what bad things can happen if you don't pin your dependencies, but for the uninitiated: I'm talking about things like projects breaking inexplicably after 6 months because you rebuilt some Docker container and something or other got upgraded and is now incompatible. Beyond the technical issues, there's a human engineering problem of getting everyone on the same page about the right processes, and Python makes that vastly harder by not having a One True Solution that everyone just uses.
  - gjvc 4y ago
    "pip freeze" generates a versioned list of packages to install in the same format as requirements.txt -- in this example below i've called it versions.txt. I have a bash wrapper "bin/venv-create" which essentially does this python3 -m venv .venv/ if [[ -f versions.txt ]] && [[ versions.txt -nt requirements.txt ]]; then pip install --requirement versions.txt else pip install --requirement requirements.txt pip freeze > versions.txt fi (I place the files in etc/pip/ in my projects (and check them into git), but I've omitted the paths for clarity. One could embellish this by including python version number in the filename, as package requirements can change between python versions.) I also have a bin/venv-python wrapper which sets PYTHONPATH, PYTHONDONTWRITEBYTCODE before chain calling .venv/bin/python3 with the arguments, and this is how pip above is called. (again, omitted above for clarity.) This won't cover everyone's usage scenario, but it works for me. YMMV. https://iam.georgecox.com/2021/09/25/python-3-venv/ https://iam.georgecox.com/2021/09/25/python-3-venv/ explains the details.
    - jvolkman 4y ago
      `pip freeze` doesn't generate hashes, so you can't be sure that the package contents haven't actually changed but maintained the same version string. Also, `pip freeze` doesn't include platform-specific dependencies for other platforms. So if you run the following on linux and again on macos, you'll get different results because `ipython` depends on `appnope` only when running on macos. python -m venv env && env/bin/pip install ipython==8.4.0 --quiet && env/bin/pip freeze
- thrashh 4y ago
  The reason for the conflicting advice is because people keep making new package managers that still suck overall. Like they fix one major package manager problem and somehow neglect the other 60% of problems. So you end up with all these different Python package managers that are all good at individual different things but no single package manager that is good at all of the things. I swear that is a major problem with most libraries or tools (and products overall actually) that people make. It’s like people fix their pet peeve but forget that the whole picture matters way more.
- hoosieree 4y ago
  Python's package management is to package managers as C++ is to programming languages: everyone who uses it understands about 10% of it, and no two people understand the same 10%.
  - wheelerof4te 4y ago
    The most relatable comment on Python's packaging ever. BTW, I know setup.py method. How much is that in percents?
- woodruffw 4y ago
  The conflicting advice is a serious problem. I hope you'll forgive me for adding one additional piece of advice: for many Python packages, the only packaging metadata you need is `pyproject.toml`. You don't even need `setup.py` anymore, so long as you're using a build backend that supports editable installs with `pyproject.toml`. Here's an example of a Python package that does everything in `pyproject.toml`[1]. You should be able to copy that into any of your projects, edit it to match your metadata, and everything will work exactly as if you have a `setup.cfg` or `setup.py`. [1]: https://github.com/sigstore/sigstore-python https://github.com/sigstore/sigstore-python
  - Hackbraten 4y ago
    Thanks for the link!
- lovingCranberry 4y ago
  I wish there was a "standard" package manager like npm. While npm might not be the most efficient one, it just works.
- gernb 4y ago
  Could you please link to your favorite way to install python at specific version and all dependencies without have to put anything in a global place. Ideally on MacOS (since it's what I'm on most). I don't have homebrew or macports since eventually those always lead to me getting a borked system since they want everything installed globally.
  - takeda 4y ago
    Not the GP, but I became a big fan of using poetry for managing python package dependencies. For managing python itself and binary libraries I started using Nix package manager. It allows to describe all dependencies via code, but with time that code became a boilerplate, so I created this: https://github.com/takeda/nix-cde https://github.com/takeda/nix-cde It works very well for me so far. You do need to have Nix[1] installed, but hopefully that should be the only thing needed and everything you can just list in project.nix file (in the example in README.md you should have access to `dive` command even if you never installed it on your system). Which will make it available only for that project. Here's also an example of a very simple project: [2] [1] https://nixos.org/download.html#nix-install-macos https://nixos.org/download.html#nix-install-macos [2] https://github.com/takeda/nix-cde/tree/master/tools/aws_assume https://github.com/takeda/nix-cde/tree/master/tools/aws_assu...
  - Hackbraten 4y ago
    Not OP but anecdotally, this is how my workflow looks like: 1. Install pyenv to a central location of my choice, e. g. ~/.pyenv (this helps manage all those different Python versions piling up from all the isolated projects I have.) 2. Install pipenv as a stand-alone tool. Doesn’t matter which Python I’m using, I just want to have it in my PATH. 3. Now I’m ready to create fully isolated projects using pipenv. Not only does pipenv create a venv right inside my project directory for me, it also helps me manage and lock dependencies. It also talks to pyenv so it can fetch the per-project Python version for me; this is especially useful when collaborating with others on the same project; they can reliably check out the project without having to manage specific Python versions themselves. YMMV but I’ve had a stable experience using this setup so far.
vorpalhex 4y ago
Poetry or pipenv.
bschwindHN 4y ago
Yep. Every time I try out a python-based tool, it _never_ runs successfully on the first try. I always look for Go/Rust/C based alternatives, especially when it's a CLI tool.
jeroenhd 4y ago
The problem is that the operating system packages itself often depends on some version of a dependency. You can perfectly pip install something globally, but globally installing a library means everything else on your system is affected too. There are solutions to this (pipenv being a popular one, though sadly often not explained to Python newcomers) and there are alternatives for that as well. You probably don't actually want to install a package like a package manager, most of the time you want to install an application and the specific requirements for that application, which is exactly what pipenv is for. Your global package manager and your global Python package manager don't agree on what version is "the latest stable version" of a package so you just can't expect global package installs to work like that. It's like trying to make pacman, apt, and dnf work on the same file tree: you can probably get it done, but the end result will grow unstable or unusable within days.
jefurii 4y ago
I think this has become a meme. It's true that people have made a lot of different ways to install Python packages. Each of these people has an axe to grind and they have supporters who are trying to boost their preferred solution and keep people away from the others. That part really is a mess. The standard virtualenv+pip has always been rock solid for me. Maybe this is because I do all my development inside virtualenvs and I only ever use pip inside a venv. And I run Linux, though I do ship my Python code to a fleet of servers. Sometimes I think maybe this Python-ecosystem-is-horrible stuff comes from people who are trying to steer people away from Python towards their preferred language.

newuser4321 4y ago

I think you mean the package management rather than the ecosystem. For all its failings, the python ecosystem (for machine learning / data science in particular) is unmatched, which is why python is so popular. As a "professional" python user (who got there in a roundabout way), I'd say the biggest problem with package management is all the conflicting advice and different ways to accomplish the same thing (there are other problems, but they are surmountable). Once you get a workflow down, it ends up being pretty easy to set up and manage an environment for a given project. Not that it's perfect, but I think a lot of the stereotypes about how bad it is come from how bad it appears as you try to converge on a setup that works for you.

bloblaw 4y ago
I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TIMTOWTDI). As a tongue-in-cheek reaction, Python espoused "TOOWTDI", or There's Only One Way To Do It :) reference: https://wiki.python.org/moin/TOOWTDI https://wiki.python.org/moin/TOOWTDI The problem is, when it comes to the Python package management ecosystem, there are SO MANY ways to do it. And they aren't equal and require a deep understanding of each tool to select the correct choice. It is a problem for Python, although I often regrettably see it dismissed as not a true issue because $latest_package_management_system fixes it. Python has allowed me to be profoundly productive in many ways, but this is a huge sore spot for the Python ecosystem.
- xdfgh1112 4y ago
  I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?
  - bloblaw 4y ago
    The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern...to me. Python scripting today requires shipping your development environment. Python is wonderful until you want to run that code on another machine. At that point, the target system has to venv their way into reproducing your environment...often including the specific Python interpreter you picked, and to download (and possibly compile) all modules and their dependencies. It can work beautifully and many of the large companies I've worked for have put oodles of engineering effort into making it "easy", as long as you follow their happy path and don't deviate. But, to your point, on my own machine with venv + pip it "just works". The pain comes when trying to venv + pip on another machine. How confident are you that you could venv + pip your moderately complex Python application on 200k machines without issue? What if it's a mix of Windows, Linux, and macOS? (this is a real scenario I've experienced). From my own experience I can share that it is painful. Your mileage may vary. And after a while your system is riddled with Python venvs. Leading to some mildly annoying problems with popular IDE's: https://www.jetbrains.com/help/pycharm/package-installation-issues.html https://www.jetbrains.com/help/pycharm/package-installation-...
    - elbigbad 4y ago
      Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.
      - behnamoh 4y ago
        I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?
        deleted 4y ago
jmcgough 4y ago
As someone two months into their first python job - the ecosystem is solid for ML / data science, but there's a lot of places where it's painfully lacking. I hate the most popular ORM (sqlalchemy) and alembic has a lot of footguns: for instance, if you autogenerate a migration where you change a table name, it will try to drop the old table and create a new one. In web dev a lot of the OSS community has moved on to more appropriate or exciting languages, so the tools I'd normally reach for frequently are OSS projects that haven't been maintained in seven years. As for pip: it needs a major version bump that creates a lockfile, or it needs to be intentionally sunset. It's easy for a beginner to start using (part of why it's heavily used), but extremely dangerous because of the lack of guarantees. I wish it didn't have as big of a footprint as it does.
- wombatpm 4y ago
  Much prefer the Django ORM, which is a pain to use if you are not building web apps
  - mickeyk 4y ago
    I've used peewee in the past which worked well.
- m_mueller 4y ago
  agreed on the lockfile. we use pip-tools to achieve this, which gives the `pip-compile` command. a Makefile is tying together the different development and build workflows we have, such that it can be executed both locally as well as in our CICD build server. complexity is manageable, but it's roughly comparable to a compiled language if you want locked-down reproducible builds (which I recommend for anything production grade).
eslaught 4y ago
To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sigh, these are usually the ones I get stuck working with, due to outside constraints). A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? It depends on which one of the multitude of options you pick. But at least a couple of the most popular ones basically don't do this (pip) or do this in a hacky, ugly way that makes you want to tear your hair out (Conda). As a result, at least in the projects I work on, people tend to skip this. I hope it should be obvious what bad things can happen if you don't pin your dependencies, but for the uninitiated: I'm talking about things like projects breaking inexplicably after 6 months because you rebuilt some Docker container and something or other got upgraded and is now incompatible. Beyond the technical issues, there's a human engineering problem of getting everyone on the same page about the right processes, and Python makes that vastly harder by not having a One True Solution that everyone just uses.
- gjvc 4y ago
  "pip freeze" generates a versioned list of packages to install in the same format as requirements.txt -- in this example below i've called it versions.txt. I have a bash wrapper "bin/venv-create" which essentially does this python3 -m venv .venv/ if [[ -f versions.txt ]] && [[ versions.txt -nt requirements.txt ]]; then pip install --requirement versions.txt else pip install --requirement requirements.txt pip freeze > versions.txt fi (I place the files in etc/pip/ in my projects (and check them into git), but I've omitted the paths for clarity. One could embellish this by including python version number in the filename, as package requirements can change between python versions.) I also have a bin/venv-python wrapper which sets PYTHONPATH, PYTHONDONTWRITEBYTCODE before chain calling .venv/bin/python3 with the arguments, and this is how pip above is called. (again, omitted above for clarity.) This won't cover everyone's usage scenario, but it works for me. YMMV. https://iam.georgecox.com/2021/09/25/python-3-venv/ https://iam.georgecox.com/2021/09/25/python-3-venv/ explains the details.
  - jvolkman 4y ago
    `pip freeze` doesn't generate hashes, so you can't be sure that the package contents haven't actually changed but maintained the same version string. Also, `pip freeze` doesn't include platform-specific dependencies for other platforms. So if you run the following on linux and again on macos, you'll get different results because `ipython` depends on `appnope` only when running on macos. python -m venv env && env/bin/pip install ipython==8.4.0 --quiet && env/bin/pip freeze
thrashh 4y ago
The reason for the conflicting advice is because people keep making new package managers that still suck overall. Like they fix one major package manager problem and somehow neglect the other 60% of problems. So you end up with all these different Python package managers that are all good at individual different things but no single package manager that is good at all of the things. I swear that is a major problem with most libraries or tools (and products overall actually) that people make. It’s like people fix their pet peeve but forget that the whole picture matters way more.
hoosieree 4y ago
Python's package management is to package managers as C++ is to programming languages: everyone who uses it understands about 10% of it, and no two people understand the same 10%.
- wheelerof4te 4y ago
  The most relatable comment on Python's packaging ever. BTW, I know setup.py method. How much is that in percents?
woodruffw 4y ago
The conflicting advice is a serious problem. I hope you'll forgive me for adding one additional piece of advice: for many Python packages, the only packaging metadata you need is `pyproject.toml`. You don't even need `setup.py` anymore, so long as you're using a build backend that supports editable installs with `pyproject.toml`. Here's an example of a Python package that does everything in `pyproject.toml`[1]. You should be able to copy that into any of your projects, edit it to match your metadata, and everything will work exactly as if you have a `setup.cfg` or `setup.py`. [1]: https://github.com/sigstore/sigstore-python https://github.com/sigstore/sigstore-python
- Hackbraten 4y ago
  Thanks for the link!
lovingCranberry 4y ago
I wish there was a "standard" package manager like npm. While npm might not be the most efficient one, it just works.
gernb 4y ago
Could you please link to your favorite way to install python at specific version and all dependencies without have to put anything in a global place. Ideally on MacOS (since it's what I'm on most). I don't have homebrew or macports since eventually those always lead to me getting a borked system since they want everything installed globally.
- takeda 4y ago
  Not the GP, but I became a big fan of using poetry for managing python package dependencies. For managing python itself and binary libraries I started using Nix package manager. It allows to describe all dependencies via code, but with time that code became a boilerplate, so I created this: https://github.com/takeda/nix-cde https://github.com/takeda/nix-cde It works very well for me so far. You do need to have Nix[1] installed, but hopefully that should be the only thing needed and everything you can just list in project.nix file (in the example in README.md you should have access to `dive` command even if you never installed it on your system). Which will make it available only for that project. Here's also an example of a very simple project: [2] [1] https://nixos.org/download.html#nix-install-macos https://nixos.org/download.html#nix-install-macos [2] https://github.com/takeda/nix-cde/tree/master/tools/aws_assume https://github.com/takeda/nix-cde/tree/master/tools/aws_assu...
- Hackbraten 4y ago
  Not OP but anecdotally, this is how my workflow looks like: 1. Install pyenv to a central location of my choice, e. g. ~/.pyenv (this helps manage all those different Python versions piling up from all the isolated projects I have.) 2. Install pipenv as a stand-alone tool. Doesn’t matter which Python I’m using, I just want to have it in my PATH. 3. Now I’m ready to create fully isolated projects using pipenv. Not only does pipenv create a venv right inside my project directory for me, it also helps me manage and lock dependencies. It also talks to pyenv so it can fetch the per-project Python version for me; this is especially useful when collaborating with others on the same project; they can reliably check out the project without having to manage specific Python versions themselves. YMMV but I’ve had a stable experience using this setup so far.

vorpalhex 4y ago

Poetry or pipenv.

bschwindHN 4y ago

Yep. Every time I try out a python-based tool, it _never_ runs successfully on the first try. I always look for Go/Rust/C based alternatives, especially when it's a CLI tool.

jeroenhd 4y ago

The problem is that the operating system packages itself often depends on some version of a dependency. You can perfectly pip install something globally, but globally installing a library means everything else on your system is affected too. There are solutions to this (pipenv being a popular one, though sadly often not explained to Python newcomers) and there are alternatives for that as well. You probably don't actually want to install a package like a package manager, most of the time you want to install an application and the specific requirements for that application, which is exactly what pipenv is for. Your global package manager and your global Python package manager don't agree on what version is "the latest stable version" of a package so you just can't expect global package installs to work like that. It's like trying to make pacman, apt, and dnf work on the same file tree: you can probably get it done, but the end result will grow unstable or unusable within days.

jefurii 4y ago

I think this has become a meme. It's true that people have made a lot of different ways to install Python packages. Each of these people has an axe to grind and they have supporters who are trying to boost their preferred solution and keep people away from the others. That part really is a mess. The standard virtualenv+pip has always been rock solid for me. Maybe this is because I do all my development inside virtualenvs and I only ever use pip inside a venv. And I run Linux, though I do ship my Python code to a fleet of servers. Sometimes I think maybe this Python-ecosystem-is-horrible stuff comes from people who are trying to steer people away from Python towards their preferred language.