6 ms·

> sudo apt-get install python3.10 python3.11 python3.12 This assumes the Python version you need is available from your package manager's repo. This won't work

by athrun 2y ago

> sudo apt-get install python3.10 python3.11 python3.12 This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available. > You are incorrect about needing to use an additional tool to install a "global" tool like `ruff`; `pip` does this by default when you're not using a virtual environment. True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle. And it gets even more complex if you need different tools that have different Python version requirements.

coldtea 2y ago
>This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available. And of course you could be working with multiple distros and versions of the same distro, production and dev might be different environment and tons of others concerns. You need something that just works across.
- nightpool 2y ago
  Surely you just use Docker for production, right?
  - nicoburns 2y ago
    You almost need to use Docker for deploying Python because the tooling is so bad that it's otherwise very difficult to get a reproducible environment. For many other languages the tooling works well enough that there's relatively little advantage to be had from Docker (although you can of course still use it).
  - wombatpm 2y ago
    And how do you know everything is ok when you build your new docker image?
zahlman 2y ago
> pip doesn't resolve dependencies of dependencies. This is simply incorrect. In fact the reason it gets stuck on resolution sometimes is exactly because it resolved transitive dependencies and found that they were mutually incompatible. Here's an example which will also help illustrate the rest of my reply. I make a venv for Python 3.8, and set up a new project with a deliberately poorly-thought-out pyproject.toml: [project] name="example" version="0.1.0" dependencies=["pandas==2.0.3", "numpy==1.17.3"] I've specified the oldest version of Numpy that has a manylinux wheel for Python 3.8 and the newest version of Pandas similarly. These are both acceptable for the venv separately, but mutually incompatible on purpose. When I try to `pip install -e .` in the venv, Pip happily explains (granted the first line is a bit strange): ERROR: Cannot install example and example==0.1.0 because these package versions have conflicting dependencies. The conflict is caused by: example 0.1.0 depends on numpy==1.17.3 pandas 2.0.3 depends on numpy>=1.20.3; python_version < "3.10" To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip to attempt to solve the dependency conflict If I change the Numpy pin to 1.20.3, that's the version that gets installed. (`python-dateutil`, `pytz`, `six` and `tzdata` are also installed.) If I remove the Numpy requirement completely and start over, Numpy 1.24.4 is installed instead - the latest version compatible with Pandas' transitive specification of the dependency. Similarly if I unpin Pandas and ask for any version - Pip will try to install the latest version it can, and it turns out that the latest Pandas version that declares compatibility with 3.8, indeed allows for fetching 3.8-compatible dependencies. (Good job not breaking it, Pandas maintainers! Although usually this is trivial, because your dependencies are also actively maintained.) > pip will only respect version pinning for dependencies you explicitly specify. So for example, say I am using pandas and I pin it to version X. If a dependency of pandas (say, numpy) isn't pinned as well, the underlying version of numpy can still change when I reinstall dependencies. Well, sure; Pip can't respect a version pin that doesn't exist anywhere in your project. If the specific version of Pandas you want says that it's okay with a range of Numpy versions, then of course Pip has freedom to choose one of those versions. If that matters, you explicitly specify it. Other programs like uv can't fix this. They can only choose different resolution strategies, such as "don't update the transitive dependency if the environment already contains a compatible version", versus "try to use the most recent versions of everything that meet the specified compatibility requirements". > To get around this with pip you would need an additional tool like pip-tools, which allows you to pin all dependencies, explicit and nested, to a lock file for true reproducibility. No, you just use Pip's options to determine what's already in the environment (`pip list`, `pip freeze` etc.) and pin everything that needs pinning (whether with a Pip requirements file or with `pyproject.toml`). Nothing prevents you from listing your transitive dependencies in e.g. the [project.dependencies] of your pyproject.toml, and if you pin them, Pip will take that constraint into consideration. Lock files are for when you need to care about alternate package sources, checking hashes etc.; or for when you want an explicit representation of your dependency graph in metadata for the sake of other tooling. > This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available. I have built versions 3.5 through 3.13 inclusive from source and have them installed in /opt and the binaries symlinked in /usr/local/bin. It's not difficult at all. > True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle. What brittleness are you talking about? There's no reason why the tool needs to run in the same environment as the code it's operating on. You can install it in its own virtual environment, too. Since tools generally are applications, I use Pipx for this (which really just wraps a bit of environment management around Pip). It works great; for example I always have the standard build-frontend `build` (as `pyproject-build`) and the uploader `twine` available. They run from a guaranteed-compatible Python. And they would if they were installed for the system Python, too. (I just, you know, don't want to do that because the system Python is the system package manager's responsibility.) The separate environment don't matter because the tool's code and the operated-on project's code don't even need to run at the same time, let alone in the same process. In fact, it would make no sense to be running the code while actively trying to build or upload it. > And it gets even more complex if you need different tools that have different Python version requirements. No, you just let each tool have the virtual environment it requires. And you can update them in-place in those environments, too.
mixmastamyk 2y ago
>> You are incorrect about needing to use an additional tool to install a "global" tool like `ruff`; `pip` does this by default when you're not using a virtual environment. >True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle. "Globally" means installed with sudo. These are installed into the user folder under ~/.local/ and called a user install by pip. I wouldn't call it "extremely brittle" either. It works fine until you upgrade to a new version of python, in which case you install the package again. Happens once a year perhaps. The good part of this is that unused cruft will get left behind and then you can delete old folders in ~/.local/lib/python3.? etc. I've been doing this over a decade without issue.
- lmm 2y ago
  > It works fine until you upgrade to a new version of python, in which case you install the package again. Or you install a second global tool that depends on an incompatible version of a library.
  - mixmastamyk 2y ago
    Never happened, and exceedingly unlikely to because your user-wide tools should be few.
    - lmm 2y ago
      > exceedingly unlikely to because your user-wide tools should be few. Why "should"? I think it's the other way around - Python culture has shied away from user-wide tools because it's known that they cause problems if you have more than a handful of them, and so e.g. Python profilers remain very underdeveloped.
      - mixmastamyk 2y ago
        There are simply few, I don't shy away from them. Other than tools replaced by ruff, httpie, twine, ptpython, yt-dlp, and my own tools I don't need anything else. Most "user" tools are provided by the system package manager. All the other project-specific things go in venvs where they belong. This is all a non-issue despite constant "end of the world" folks who never learned sysadmin and are terrified of an error. If a libraries conflict, uninstall them, and put them in a venv. Why do all the work up front? I haven't had to do that in so long I forget how long it was. Early this century.

by athrun 2y ago

coldtea 2y ago

>This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available. And of course you could be working with multiple distros and versions of the same distro, production and dev might be different environment and tons of others concerns. You need something that just works across.

nightpool 2y ago
Surely you just use Docker for production, right?
- nicoburns 2y ago
  You almost need to use Docker for deploying Python because the tooling is so bad that it's otherwise very difficult to get a reproducible environment. For many other languages the tooling works well enough that there's relatively little advantage to be had from Docker (although you can of course still use it).
- wombatpm 2y ago
  And how do you know everything is ok when you build your new docker image?

zahlman 2y ago

> pip doesn't resolve dependencies of dependencies. This is simply incorrect. In fact the reason it gets stuck on resolution sometimes is exactly because it resolved transitive dependencies and found that they were mutually incompatible. Here's an example which will also help illustrate the rest of my reply. I make a venv for Python 3.8, and set up a new project with a deliberately poorly-thought-out pyproject.toml: [project] name="example" version="0.1.0" dependencies=["pandas==2.0.3", "numpy==1.17.3"] I've specified the oldest version of Numpy that has a manylinux wheel for Python 3.8 and the newest version of Pandas similarly. These are both acceptable for the venv separately, but mutually incompatible on purpose. When I try to `pip install -e .` in the venv, Pip happily explains (granted the first line is a bit strange): ERROR: Cannot install example and example==0.1.0 because these package versions have conflicting dependencies. The conflict is caused by: example 0.1.0 depends on numpy==1.17.3 pandas 2.0.3 depends on numpy>=1.20.3; python_version < "3.10" To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip to attempt to solve the dependency conflict If I change the Numpy pin to 1.20.3, that's the version that gets installed. (`python-dateutil`, `pytz`, `six` and `tzdata` are also installed.) If I remove the Numpy requirement completely and start over, Numpy 1.24.4 is installed instead - the latest version compatible with Pandas' transitive specification of the dependency. Similarly if I unpin Pandas and ask for any version - Pip will try to install the latest version it can, and it turns out that the latest Pandas version that declares compatibility with 3.8, indeed allows for fetching 3.8-compatible dependencies. (Good job not breaking it, Pandas maintainers! Although usually this is trivial, because your dependencies are also actively maintained.) > pip will only respect version pinning for dependencies you explicitly specify. So for example, say I am using pandas and I pin it to version X. If a dependency of pandas (say, numpy) isn't pinned as well, the underlying version of numpy can still change when I reinstall dependencies. Well, sure; Pip can't respect a version pin that doesn't exist anywhere in your project. If the specific version of Pandas you want says that it's okay with a range of Numpy versions, then of course Pip has freedom to choose one of those versions. If that matters, you explicitly specify it. Other programs like uv can't fix this. They can only choose different resolution strategies, such as "don't update the transitive dependency if the environment already contains a compatible version", versus "try to use the most recent versions of everything that meet the specified compatibility requirements". > To get around this with pip you would need an additional tool like pip-tools, which allows you to pin all dependencies, explicit and nested, to a lock file for true reproducibility. No, you just use Pip's options to determine what's already in the environment (`pip list`, `pip freeze` etc.) and pin everything that needs pinning (whether with a Pip requirements file or with `pyproject.toml`). Nothing prevents you from listing your transitive dependencies in e.g. the [project.dependencies] of your pyproject.toml, and if you pin them, Pip will take that constraint into consideration. Lock files are for when you need to care about alternate package sources, checking hashes etc.; or for when you want an explicit representation of your dependency graph in metadata for the sake of other tooling. > This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available. I have built versions 3.5 through 3.13 inclusive from source and have them installed in /opt and the binaries symlinked in /usr/local/bin. It's not difficult at all. > True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle. What brittleness are you talking about? There's no reason why the tool needs to run in the same environment as the code it's operating on. You can install it in its own virtual environment, too. Since tools generally are applications, I use Pipx for this (which really just wraps a bit of environment management around Pip). It works great; for example I always have the standard build-frontend `build` (as `pyproject-build`) and the uploader `twine` available. They run from a guaranteed-compatible Python. And they would if they were installed for the system Python, too. (I just, you know, don't want to do that because the system Python is the system package manager's responsibility.) The separate environment don't matter because the tool's code and the operated-on project's code don't even need to run at the same time, let alone in the same process. In fact, it would make no sense to be running the code while actively trying to build or upload it. > And it gets even more complex if you need different tools that have different Python version requirements. No, you just let each tool have the virtual environment it requires. And you can update them in-place in those environments, too.

mixmastamyk 2y ago

>> You are incorrect about needing to use an additional tool to install a "global" tool like `ruff`; `pip` does this by default when you're not using a virtual environment. >True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle. "Globally" means installed with sudo. These are installed into the user folder under ~/.local/ and called a user install by pip. I wouldn't call it "extremely brittle" either. It works fine until you upgrade to a new version of python, in which case you install the package again. Happens once a year perhaps. The good part of this is that unused cruft will get left behind and then you can delete old folders in ~/.local/lib/python3.? etc. I've been doing this over a decade without issue.

lmm 2y ago
> It works fine until you upgrade to a new version of python, in which case you install the package again. Or you install a second global tool that depends on an incompatible version of a library.
- mixmastamyk 2y ago
  Never happened, and exceedingly unlikely to because your user-wide tools should be few.
  - lmm 2y ago
    > exceedingly unlikely to because your user-wide tools should be few. Why "should"? I think it's the other way around - Python culture has shied away from user-wide tools because it's known that they cause problems if you have more than a handful of them, and so e.g. Python profilers remain very underdeveloped.
    - mixmastamyk 2y ago
      There are simply few, I don't shy away from them. Other than tools replaced by ruff, httpie, twine, ptpython, yt-dlp, and my own tools I don't need anything else. Most "user" tools are provided by the system package manager. All the other project-specific things go in venvs where they belong. This is all a non-issue despite constant "end of the world" folks who never learned sysadmin and are terrified of an error. If a libraries conflict, uninstall them, and put them in a venv. Why do all the work up front? I haven't had to do that in so long I forget how long it was. Early this century.