3 ms·

To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sig

To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sigh, these are usually the ones I get stuck working with, due to outside constraints). A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? It depends on which one of the multitude of options you pick. But at least a couple of the most popular ones basically don't do this (pip) or do this in a hacky, ugly way that makes you want to tear your hair out (Conda). As a result, at least in the projects I work on, people tend to skip this. I hope it should be obvious what bad things can happen if you don't pin your dependencies, but for the uninitiated: I'm talking about things like projects breaking inexplicably after 6 months because you rebuilt some Docker container and something or other got upgraded and is now incompatible. Beyond the technical issues, there's a human engineering problem of getting everyone on the same page about the right processes, and Python makes that vastly harder by not having a One True Solution that everyone just uses.

gjvc 4y ago
"pip freeze" generates a versioned list of packages to install in the same format as requirements.txt -- in this example below i've called it versions.txt. I have a bash wrapper "bin/venv-create" which essentially does this python3 -m venv .venv/ if [[ -f versions.txt ]] && [[ versions.txt -nt requirements.txt ]]; then pip install --requirement versions.txt else pip install --requirement requirements.txt pip freeze > versions.txt fi (I place the files in etc/pip/ in my projects (and check them into git), but I've omitted the paths for clarity. One could embellish this by including python version number in the filename, as package requirements can change between python versions.) I also have a bin/venv-python wrapper which sets PYTHONPATH, PYTHONDONTWRITEBYTCODE before chain calling .venv/bin/python3 with the arguments, and this is how pip above is called. (again, omitted above for clarity.) This won't cover everyone's usage scenario, but it works for me. YMMV. https://iam.georgecox.com/2021/09/25/python-3-venv/ https://iam.georgecox.com/2021/09/25/python-3-venv/ explains the details.
- jvolkman 4y ago
  `pip freeze` doesn't generate hashes, so you can't be sure that the package contents haven't actually changed but maintained the same version string. Also, `pip freeze` doesn't include platform-specific dependencies for other platforms. So if you run the following on linux and again on macos, you'll get different results because `ipython` depends on `appnope` only when running on macos. python -m venv env && env/bin/pip install ipython==8.4.0 --quiet && env/bin/pip freeze
  - gjvc 4y ago
    these are legitimate weaknesses -- thank you for highlighting them. On the other hand, if one can accept these weaknesses, keeping separate the root requirements file and the versions file goes a long way to making the process manageable, and not as chaotic as some might have you believe. The beef I have with so many explanations of pip is that they tell you to source ".venv/bin/activate" and then "just run pip install whatever" without a) separating the root requirements from the effective/complete requirements, and b) they don't suggest using a wrapper for the ".venv/bin/python3" binary so that execution is the same in all environments.
- eslaught 4y ago
  Thanks, I appreciate this. I don't think this practice is widespread, which is exactly why I made a point about human engineering in my original post. But I do appreciate that solutions like this exist, and I should look into driving more of this sort of thing in my projects.
  - gjvc 4y ago
    the blog post is missing a link to the template github repo; i'll fix that
viraptor 4y ago
> A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? Does it matter? I get a different way to define dependencies. But the lock file itself is an implementation detail. You use the package manager the project uses and the lock file can be opaque. It's the same for package-lock.json / yarn.lock in js land.
- bee_rider 4y ago
  The rest of the sentences in that paragraph present an argument for why it might matter.
rat87 4y ago
python's poetry has poetry.lock I think it's slowly becoming semi-standard. But still slowly and still not fully ready(bugs). Also its odd that pyproject.toml (not poetry.toml which is the poetry settings for repo) is the dependency definition file and poetry.lock is the lock file for pip file I usually make a requirements-to-freeze.txt, make a fresh virtualenv, install requirements-to-freeze.txt and then pip freeze > requirements.txt poetry does this automatically
- returnzero 4y ago
  > But still slowly and still not fully ready(bugs). I use poetry every day and so far it's been pretty great. The main complications I have had are with accessing private pypi repos in Azure DevOps pipelines which use short lived tokens but it just looks a little clunky, still works. > Also its odd that pyproject.toml (not poetry.toml which is the poetry settings for repo) is the dependency definition file and poetry.lock is the lock file This is because poetry is using Python's PEP 518[1] specification rather than define their own build requirements format. It also isn't limited to just building, you can also include the configuration for other python tools like `pytest`[2]. [1] https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [2] https://docs.pytest.org/en/stable/reference/customize.html#pyproject-toml https://docs.pytest.org/en/stable/reference/customize.html#p...
  - rat87 4y ago
    > This is because poetry is using Python's PEP 518[1] specification rather than define their own build requirements format. It also isn't limited to just building, you can also include the configuration for other python tools like `pytest`[2]. I sort of get this but its still a bit odd. especially since I have a small poetry.toml as well. > I use poetry every day and so far it's been pretty great. I like poetry, it works a lot better for me then pipenv did(to be fair that was a few years ago). But I do seem to have to delete lock file to update it and sometimes poetry add/install breaks oddly(or at least with awkward error messages). Also sometimes it interacts with venvs in odd ways. python really needs to ship with it
- mickeyk 4y ago
rootusrootus 4y ago
I must live a sheltered life. Everything I've run into so far uses requirements.txt to pin dependencies. Well supported and easy is probably why.
- eslaught 4y ago
  Yes, exactly. And how many of these projects go to any effort to pin their transitive dependencies? In my experience, this is basically no one; in fact I've seen many (most?) projects not even pin the versions on their top-level dependencies. (Ask yourself how many times you've seen just "numpy" as a dependency without any version bound. Far too often in my experience.) And this is exactly why stuff breaks: even when the root dependencies are pinned (and are they?), a transitive dependency could get upgraded and break something (or fail to build entirely).

gjvc 4y ago

"pip freeze" generates a versioned list of packages to install in the same format as requirements.txt -- in this example below i've called it versions.txt. I have a bash wrapper "bin/venv-create" which essentially does this python3 -m venv .venv/ if [[ -f versions.txt ]] && [[ versions.txt -nt requirements.txt ]]; then pip install --requirement versions.txt else pip install --requirement requirements.txt pip freeze > versions.txt fi (I place the files in etc/pip/ in my projects (and check them into git), but I've omitted the paths for clarity. One could embellish this by including python version number in the filename, as package requirements can change between python versions.) I also have a bin/venv-python wrapper which sets PYTHONPATH, PYTHONDONTWRITEBYTCODE before chain calling .venv/bin/python3 with the arguments, and this is how pip above is called. (again, omitted above for clarity.) This won't cover everyone's usage scenario, but it works for me. YMMV. https://iam.georgecox.com/2021/09/25/python-3-venv/ https://iam.georgecox.com/2021/09/25/python-3-venv/ explains the details.

jvolkman 4y ago
`pip freeze` doesn't generate hashes, so you can't be sure that the package contents haven't actually changed but maintained the same version string. Also, `pip freeze` doesn't include platform-specific dependencies for other platforms. So if you run the following on linux and again on macos, you'll get different results because `ipython` depends on `appnope` only when running on macos. python -m venv env && env/bin/pip install ipython==8.4.0 --quiet && env/bin/pip freeze
- gjvc 4y ago
  these are legitimate weaknesses -- thank you for highlighting them. On the other hand, if one can accept these weaknesses, keeping separate the root requirements file and the versions file goes a long way to making the process manageable, and not as chaotic as some might have you believe. The beef I have with so many explanations of pip is that they tell you to source ".venv/bin/activate" and then "just run pip install whatever" without a) separating the root requirements from the effective/complete requirements, and b) they don't suggest using a wrapper for the ".venv/bin/python3" binary so that execution is the same in all environments.
eslaught 4y ago
Thanks, I appreciate this. I don't think this practice is widespread, which is exactly why I made a point about human engineering in my original post. But I do appreciate that solutions like this exist, and I should look into driving more of this sort of thing in my projects.
- gjvc 4y ago
  the blog post is missing a link to the template github repo; i'll fix that

viraptor 4y ago

> A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? Does it matter? I get a different way to define dependencies. But the lock file itself is an implementation detail. You use the package manager the project uses and the lock file can be opaque. It's the same for package-lock.json / yarn.lock in js land.

bee_rider 4y ago
The rest of the sentences in that paragraph present an argument for why it might matter.

rat87 4y ago

python's poetry has poetry.lock I think it's slowly becoming semi-standard. But still slowly and still not fully ready(bugs). Also its odd that pyproject.toml (not poetry.toml which is the poetry settings for repo) is the dependency definition file and poetry.lock is the lock file for pip file I usually make a requirements-to-freeze.txt, make a fresh virtualenv, install requirements-to-freeze.txt and then pip freeze > requirements.txt poetry does this automatically

returnzero 4y ago
> But still slowly and still not fully ready(bugs). I use poetry every day and so far it's been pretty great. The main complications I have had are with accessing private pypi repos in Azure DevOps pipelines which use short lived tokens but it just looks a little clunky, still works. > Also its odd that pyproject.toml (not poetry.toml which is the poetry settings for repo) is the dependency definition file and poetry.lock is the lock file This is because poetry is using Python's PEP 518[1] specification rather than define their own build requirements format. It also isn't limited to just building, you can also include the configuration for other python tools like `pytest`[2]. [1] https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [2] https://docs.pytest.org/en/stable/reference/customize.html#pyproject-toml https://docs.pytest.org/en/stable/reference/customize.html#p...
- rat87 4y ago
  > This is because poetry is using Python's PEP 518[1] specification rather than define their own build requirements format. It also isn't limited to just building, you can also include the configuration for other python tools like `pytest`[2]. I sort of get this but its still a bit odd. especially since I have a small poetry.toml as well. > I use poetry every day and so far it's been pretty great. I like poetry, it works a lot better for me then pipenv did(to be fair that was a few years ago). But I do seem to have to delete lock file to update it and sometimes poetry add/install breaks oddly(or at least with awkward error messages). Also sometimes it interacts with venvs in odd ways. python really needs to ship with it
mickeyk 4y ago

rootusrootus 4y ago

I must live a sheltered life. Everything I've run into so far uses requirements.txt to pin dependencies. Well supported and easy is probably why.

eslaught 4y ago
Yes, exactly. And how many of these projects go to any effort to pin their transitive dependencies? In my experience, this is basically no one; in fact I've seen many (most?) projects not even pin the versions on their top-level dependencies. (Ask yourself how many times you've seen just "numpy" as a dependency without any version bound. Far too often in my experience.) And this is exactly why stuff breaks: even when the root dependencies are pinned (and are they?), a transitive dependency could get upgraded and break something (or fail to build entirely).