9 ms·

Boring Python: dependency management (2022)

114 points by bruh2 3y ago

djstein 3y ago
I’m happy others are writing on this subject! I appreciate your enthusiasm for trying to do the most “basic” things in Python. While I personally enjoy a bit more management with tools like Poetry, I believe all Python programmers should know how pip and setuptools work before trying their supersets. To add to this discussion, I recently wrote this less wordy guide on macOS Python setup https://steins.studio/technical/01-python-setup https://steins.studio/technical/01-python-setup
- dabber 3y ago
  > To add to this discussion, I recently wrote this less wordy guide on macOS Python setup https://steins.studio/technical/01-python-setup https://steins.studio/technical/01-python-setup Thanks for this. It's exactly the the format and depth I wanted. I haven't been able to muster the time or energy to start digging into the quagmire that is the Python ecosystem but this seems like the perfect place to start (and hopefully stay for a while.)
Humphrey 3y ago
I've landed on Poetry after having tried many different options over the years. Being able to specify my relatively open dependencies (Eg "django==4.0.*") but having the exact version of every subdependency versioned has proved to be very reliable and reproducable. Docker multistage builds allow me to ship a production container without Poetry installed.
- selcuka 3y ago
  Note that you can also accomplish the same with pip using the tilde operator in your requirements.txt (e.g. Django~=4.0), and Constraints Files [1] for the subdependencies. A multistage build is still recommended as building your dependencies might need gcc or other tools. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
irjustin 3y ago
With the move to pyproject.toml in 2022[0], Poetry has become our goto method. With the lock file being default, we don't worry about different installs in different envs. Having come from the Rails world, Bundler system was solved for... a decade? So I was surprised it was such a mess in Python until so recently. At the core, the thing that makes Poetry and Bundler system so predictable is 1, lock file. 2, ability to install different versions in diff locations and referencing the version you need to load. Each alone isn't enough. npm had the same problem pip suffered from, you may have a version installed different what the req.txt, proj.js or even lockfile says but because it exists inside the location, it gets loaded. It wasn't until yarn2 did node_modules finally get moved out such that side-by-side versions wasn't awkward. [EDIT] If you're not using Poetry + Docker for deployment yet, I 100% recommend it as the "boring" method. RUN curl -sSL https://install.python-poetry.org | python - RUN /root/.local/bin/poetry config virtualenvs.create false COPY poetry.lock pyproject.toml ./ RUN --mount=type=secret,id=gh_priv_key,target=/root/.ssh/id_rsa \ /root/.local/bin/poetry install --without dev --no-root [0] https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...
- EdwardDiego 3y ago
  Notably Poetry lockfiles are farrrrrr better cross-env then Pipenv's, which if you lock on an ARM Mac, will not install on a x64 Linux box. Poetry just works.
- andenacitelli 3y ago
  Hard agree. This article makes good points but I feel like Poetry just simplifies so much of it and makes it so much less manual. We’ve used it at $DAYJOB for months and it’s been awesome. In the same sense as the article — it’s been sufficiently boring to rarely be the source of problems, and when it is, it’s almost always our fault rather than a design flaw in Poetry.
- doctorpangloss 3y ago
  Despite these comments, Poetry is pretty limited. You cannot install PyTorch with it for example, and the developers do not care: https://github.com/python-poetry/poetry/issues/6409 https://github.com/python-poetry/poetry/issues/6409 So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.
EdwardDiego 3y ago
The biggest problem in Python dep management is Pip. When upgrading pip breaks tools wrapping/integrating with it, it's a bad time. I've had to pin pip itself a few times due to resolution that used to work failing, and sometimes there's breaking API changes at the module level. Oh and also because setup.pys in packages are somehow tied to pip apis. It's a weak foundation to build from.
mattbillenstein 3y ago
The python core team endorses pip, but pip solves something like 80% of the problem - we need them to endorse the other 20% of the solution. I've been using pip-tools for some time, I think it solves the other 20% of the problem for me in a simple way that I like. Poetry et al seem to be trying to do too much - ymmv. The iterations on packaging that don't really seem to ever get it right are, I think, frustrating to the community where the core likes to advertise a "Zen of Python" "one way to do things" mantra, but can never really get 100% of the packaging problem sorted out in a clean way in spite of several communities seeming to figure it out.
- doctorpangloss 3y ago
  The core team endorses everything, not only pip, but good luck finding an example setup.py with example repositories from any of the documentation pages you find on Google. It’s results 1-3 and is always marked Obsolete. The communication on what to do is a disaster. Look at non obsolete https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... - my dude, their pyproject.toml doesn’t even specify a dependencies section, which is 99% of the value proposition of packaging.
  - mattbillenstein 3y ago
    Yeah, this too. I have been impressed that they're making progress though - wheels solve some problems conda used to solve; so doing ML stuff that is based on conda I can usually just use regular pip packages now which is very nice.
  - voidnap 3y ago
    > their pyproject.toml doesn’t even specify a dependencies section Doesn't it though? https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#a-full-example https://packaging.python.org/en/latest/guides/writing-pyproj...
- globular-toast 3y ago
  I like that pip-tools is a separate thing. Most Python packages don't need and shouldn't use a requirements.txt file. It makes more sense for npm to have it by default, but not Python.
yijiahe 3y ago
The author mentions using pip freeze or pip-tools pip-compile as a solution to the indirect dependencies which are reliant on the Python environment, i.e. the platform and Python version. But from what I understand pip cross-environment usage needs the requirements.txt file to be generated on the environment it is going to be run on. The solution of copying in the same requirements text for installing the packages locally might not work in the container.
- jvolkman 3y ago
  You are correct. Poetry and PDM solve this problem.
  - geethree 3y ago
    As does the PEX(python executable) project.
ktbwrestler 3y ago
This is great! Does anyone know of a similar resource for ruby?
- rcaught 3y ago
  You don't need one, because Ruby dependency management isn't a clusterf#!k like Python's.
- yxhuvud 3y ago
  You use bundler to manage what dependencies are installed and use that to get a lock file. You also create a .ruby-version file that specify the language version. There are a whole bunch of different tools to select what ruby version, but all works with that file and what you chose really doesn't matter.
nurettin 3y ago
pip install, pip freeze, let's return to our roots and whatnot. But what if there are dependency constraints?
- laichzeit0 3y ago
  How does one install development only dependencies using that approach? My production Dockerfiles are always: RUN poetry install --only main
  - vindex10 3y ago
    There is an open discussion on this in Python community: https://peps.python.org/pep-0735/ https://peps.python.org/pep-0735/
  - nurettin 3y ago
    you would have to maintain two requirements files, and automate their concatenation. It's messy. With a versionless requirements.txt file, there is no guarantee that your project will work on pip install next year.
quickthrower2 3y ago
Of course you can tame your snake with a container.
ildjarn 3y ago
Has anyone figured out to how to “cross-compile” Python? By this I mean creating an app bundle that contains the dependencies but for another platform than the one we are bundling on.
- jvolkman 3y ago
  Probably not what you're looking for, but I put together a demo of actually cross-compiling a wheel the other day: https://github.com/jvolkman/bazel-pycross-zstandard-example https://github.com/jvolkman/bazel-pycross-zstandard-example But if you just mean that you want to gather the dependencies for a platform other than your build host: this should be possible with the help of Poetry and PDM since they both perform cross-platform resolution.
  - ildjarn 3y ago
    rules_pycross looks very interesting and is sorely needed in the Python ecosystem. I will follow its development!
globular-toast 3y ago
Here's my "boring" workflow: 1. Start project (mkdir, git init), 2. Make virtualenv using virtualenvwrapper, 3. Write project.toml file for setuptools, 4. pip install -e . 5. To add deps, add them to pyproject.toml and repeat step 4. Do not pip install deps directly. Do not pin deps to any particular version, but if you have to you can add constraints like >=5 (I need a feature introduced in v5). 6. If you are writing a package to be pip installed by others then you're done. Read setuptools docs for how to build etc. 7. If you also want to build an environment to run your code (e.g. docker image for deployment or serverless deployment etc) use pip-tools to pin your dependencies. (This is the only reason you need requirements.txt). 8. For test dependencies (e.g. pytest) or dev dependencies (e.g. test server) leverage optional dependencies in the pyproject.toml file. This plays very nicely with tools like tox, which you should use. Use pre-commit for linting etc.
atoav 3y ago
The biggest problem about Pythons dependency managment in 2024 is that it stil feels like an afterthought. It is just not straightforward and it is not pythonic. I would go as far as saying that dependency managment with Python is likely more complicated than anything you would normally encounter within the language. As of 2024 poetry is the best solution we have, but even it can come to its limits at times. I work in a position where I develope with poetry and have to deploy without it (using venv), and I do not wish the journey of learning how to do that on anybody.
dagw 3y ago
I'm personally afraid that the python community has bifurcated so far that a single 'correct' solution is doomed to fail. The needs of people writing and deploying HTTP/REST servers is just so different from the people writing PyTorch models and numerical simulations that no tool will ever satisfy the very different needs of both camps. The worst part is that many people developing these tools don't seem to realise this and blindly claim that their tool is The Tool! without having any deep insights into the needs of the other camps.
- hereonout2 3y ago
  > The needs of people writing and deploying HTTP/REST servers is just so different from the people writing PyTorch models and numerical simulations that no tool will ever satisfy the very different needs of both camps. Is this true though? At the end of it they both need python packages and some system dependencies installed (let's ignore models and data for now). Why is there umpteen tools for doing this in python when there isn't in other languages? I have to deploy both web apps and ML models, the first thing I do is convert any project to use pyproject and poetry. Whilst Poetry comes with it's own issues I've not had a project yet that this doesn't work for and wish the wider community would just settle on one method. Instead we have stuff coming in with various conda incantations, pip, pipx, poetry, setuptools, setup.py! Deployment of ML models needs a decent solution too, half the ML code I get goes and fetches stuff from NLTK or Huggingface at runtime. Some of it (like the LLAMA models) needs various API keys set and EULAs agreed to before it'll run, then pings back to Huggingface each time you run it to check the EULA again! This makes life difficult when trying to deploy and adds this massive dependency on 3rd party services.
  - aragilar 3y ago
    But how are you handling those system dependencies (is R and its various packages one?), and how do you know that the packages you install/manage via the python ecosystem work with them? I think the reason Python has so many things is simply because most other languages throw their arms up and say "system stuff, not my problem" (rust is an excellent example of this, the build.rs is basically the same thing as a setup.py, except less standard, and currently lacks from what I've seen is any kind of systematic solution like cibuildwheel), whereas Python has always been trying to do something to address it.
sebra 3y ago
For me it feels like people have always been over-engineering dependency management where the practical issues from not doing it are pretty much non-existent. My approach is to just use Docker, no virtualenvs. I get that you might run into the multiple interpreters issue in theory but across multiple projects in the past 5 years I haven't seen that. Also, this might no longer be true but avoid using Alpine. If you're deploying Django there is no reason to optimize image size and Alpine has a lot of things which are missing (i.e. at least a couple of years ago, wheels where not supported leading very slow build times). I only do a single requirements.txt. Anything which makes your prod and local environment differ is a bad thing in my opinion. Fine black might make my image a couple of mbs larger but why would it matter? On the other hand attempting to figure out why something which works on my machine does not work on prod is always a nightmare. Setting requirements as a range in requirements.txt allows me to automatically get bugfixes without spending time on it (e.g. django>=4.2.3,<4.2.99 django-ninja>=1.0.1,<1.0.99) Again, I might have run into 1-2 issues over the past couple of years from this and I've saved a lot of time. Getting a project running locally should not take more than 1 minute (a couple of .env vars + docker-compose up -d should be enough). The biggest practical issue in dependency management in python is dependencies not pinning their dependencies correctly.
- dagw 3y ago
  Anything which makes your prod and local environment differ is a bad thing in my opinion Unless you're writing code that only you will deploy to machines that you control, "prod" and "local" will always be different. If you're only targeting a fixed version of a fixed OS on a fixed architecture, then most things are easy. For me "local" is a Mac running ARM, for the person pip installing my tool "prod" might be Linux or Windows. I cannot punt (or I can, but it would greatly diminish the usefulness of the stuff I develop) and say "your prod must equal my local or it won't work", I have to deal with it and I want tools that make this hard problem as easy as possible.
taraparo 3y ago
Coming from Java/Maven I was amazed to see the obscure mess in Python world regarding dependency management. After trying all available tools I finally settled on pdm. I also found it to be more intuitive than poetry.
- rq1 3y ago
  PDM is a better tool in general, the documentation is not great though. The big plus is: it supports mono-repos with multiple projects, unlike poetry. The answer to some issues by the poetry team (including monorepo one) was along the line: “deal with it”. Fine, switch to PDM.
nicornk 3y ago
Pixi is great, highly recommended!
- banditelol 3y ago
  Have you compared it with poetry or pip-tools? I'm thinking of trying pixi but still can't muster up energy to do it. Especially since for my use case poetry and pip-tools cover most of it.
- baggiponte 3y ago
  I like pixi, but I am not likely to make the switch. They don't support pyproject.toml and other standards. This disqualifies it from being a potential "recommended tool" by PyPA or whatever.
posix_monad 3y ago
The "boring" set of tools described here does not enable reproducible artefacts. This is a huge weakness of the ecosystem.
- linkdd 3y ago
  Of this specific solution yes, of the "ecosystem" no. This has been solved by many other solutions (pipenv, poetry, pdm, etc...).
  - barbazoo 3y ago
    I haven’t tested this yet but what’s better about pipfile.lock over whatever pip-compile spits out? It sounds like both are exact versions of packages, no?
    - linkdd 3y ago
      I don't know about pipfile.lock nor pip-compile, once I tested poetry I stuck to it without rethinking it (because I have better things to do than test all the package managers out there). But the poetry.lock contains hashes as well.
    - EdwardDiego 3y ago
      Pipfile.lock is broken if you have wheels wrapping compiled code as it captures the arch in the lock file! Poetry doesn't do this, so you can lock on your M2 Mac and install on x86 fine. Pip-compile in the most common use just creates a requirements.txt with everything pinned to a given version. I think you can do hash stuff with it, but haven't used that part.
      - o11c 3y ago
        If you're building for multiple platforms, it doesn't make sense to lock your dependencies. "Some platforms need newer versions" is the default case; don't make your tools fight it. That said, package managers that do timestamp-based version filtering would be very useful.

114 points by bruh2 3y ago

I’m happy others are writing on this subject! I appreciate your enthusiasm for trying to do the most “basic” things in Python. While I personally enjoy a bit more management with tools like Poetry, I believe all Python programmers should know how pip and setuptools work before trying their supersets. To add to this discussion, I recently wrote this less wordy guide on macOS Python setup https://steins.studio/technical/01-python-setup https://steins.studio/technical/01-python-setup

dabber 3y ago
> To add to this discussion, I recently wrote this less wordy guide on macOS Python setup https://steins.studio/technical/01-python-setup https://steins.studio/technical/01-python-setup Thanks for this. It's exactly the the format and depth I wanted. I haven't been able to muster the time or energy to start digging into the quagmire that is the Python ecosystem but this seems like the perfect place to start (and hopefully stay for a while.)

Humphrey 3y ago

I've landed on Poetry after having tried many different options over the years. Being able to specify my relatively open dependencies (Eg "django==4.0.*") but having the exact version of every subdependency versioned has proved to be very reliable and reproducable. Docker multistage builds allow me to ship a production container without Poetry installed.

selcuka 3y ago
Note that you can also accomplish the same with pip using the tilde operator in your requirements.txt (e.g. Django~=4.0), and Constraints Files [1] for the subdependencies. A multistage build is still recommended as building your dependencies might need gcc or other tools. [1] https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files

irjustin 3y ago

With the move to pyproject.toml in 2022[0], Poetry has become our goto method. With the lock file being default, we don't worry about different installs in different envs. Having come from the Rails world, Bundler system was solved for... a decade? So I was surprised it was such a mess in Python until so recently. At the core, the thing that makes Poetry and Bundler system so predictable is 1, lock file. 2, ability to install different versions in diff locations and referencing the version you need to load. Each alone isn't enough. npm had the same problem pip suffered from, you may have a version installed different what the req.txt, proj.js or even lockfile says but because it exists inside the location, it gets loaded. It wasn't until yarn2 did node_modules finally get moved out such that side-by-side versions wasn't awkward. [EDIT] If you're not using Poetry + Docker for deployment yet, I 100% recommend it as the "boring" method. RUN curl -sSL https://install.python-poetry.org | python - RUN /root/.local/bin/poetry config virtualenvs.create false COPY poetry.lock pyproject.toml ./ RUN --mount=type=secret,id=gh_priv_key,target=/root/.ssh/id_rsa \ /root/.local/bin/poetry install --without dev --no-root [0] https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...

EdwardDiego 3y ago
Notably Poetry lockfiles are farrrrrr better cross-env then Pipenv's, which if you lock on an ARM Mac, will not install on a x64 Linux box. Poetry just works.
andenacitelli 3y ago
Hard agree. This article makes good points but I feel like Poetry just simplifies so much of it and makes it so much less manual. We’ve used it at $DAYJOB for months and it’s been awesome. In the same sense as the article — it’s been sufficiently boring to rarely be the source of problems, and when it is, it’s almost always our fault rather than a design flaw in Poetry.
doctorpangloss 3y ago
Despite these comments, Poetry is pretty limited. You cannot install PyTorch with it for example, and the developers do not care: https://github.com/python-poetry/poetry/issues/6409 https://github.com/python-poetry/poetry/issues/6409 So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.

EdwardDiego 3y ago

The biggest problem in Python dep management is Pip. When upgrading pip breaks tools wrapping/integrating with it, it's a bad time. I've had to pin pip itself a few times due to resolution that used to work failing, and sometimes there's breaking API changes at the module level. Oh and also because setup.pys in packages are somehow tied to pip apis. It's a weak foundation to build from.

mattbillenstein 3y ago

The python core team endorses pip, but pip solves something like 80% of the problem - we need them to endorse the other 20% of the solution. I've been using pip-tools for some time, I think it solves the other 20% of the problem for me in a simple way that I like. Poetry et al seem to be trying to do too much - ymmv. The iterations on packaging that don't really seem to ever get it right are, I think, frustrating to the community where the core likes to advertise a "Zen of Python" "one way to do things" mantra, but can never really get 100% of the packaging problem sorted out in a clean way in spite of several communities seeming to figure it out.

doctorpangloss 3y ago
The core team endorses everything, not only pip, but good luck finding an example setup.py with example repositories from any of the documentation pages you find on Google. It’s results 1-3 and is always marked Obsolete. The communication on what to do is a disaster. Look at non obsolete https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... - my dude, their pyproject.toml doesn’t even specify a dependencies section, which is 99% of the value proposition of packaging.
- mattbillenstein 3y ago
  Yeah, this too. I have been impressed that they're making progress though - wheels solve some problems conda used to solve; so doing ML stuff that is based on conda I can usually just use regular pip packages now which is very nice.
- voidnap 3y ago
  > their pyproject.toml doesn’t even specify a dependencies section Doesn't it though? https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#a-full-example https://packaging.python.org/en/latest/guides/writing-pyproj...
globular-toast 3y ago
I like that pip-tools is a separate thing. Most Python packages don't need and shouldn't use a requirements.txt file. It makes more sense for npm to have it by default, but not Python.

yijiahe 3y ago

The author mentions using pip freeze or pip-tools pip-compile as a solution to the indirect dependencies which are reliant on the Python environment, i.e. the platform and Python version. But from what I understand pip cross-environment usage needs the requirements.txt file to be generated on the environment it is going to be run on. The solution of copying in the same requirements text for installing the packages locally might not work in the container.

jvolkman 3y ago
You are correct. Poetry and PDM solve this problem.
- geethree 3y ago
  As does the PEX(python executable) project.

ktbwrestler 3y ago

This is great! Does anyone know of a similar resource for ruby?

rcaught 3y ago
You don't need one, because Ruby dependency management isn't a clusterf#!k like Python's.
yxhuvud 3y ago
You use bundler to manage what dependencies are installed and use that to get a lock file. You also create a .ruby-version file that specify the language version. There are a whole bunch of different tools to select what ruby version, but all works with that file and what you chose really doesn't matter.

nurettin 3y ago

pip install, pip freeze, let's return to our roots and whatnot. But what if there are dependency constraints?

laichzeit0 3y ago
How does one install development only dependencies using that approach? My production Dockerfiles are always: RUN poetry install --only main
- vindex10 3y ago
  There is an open discussion on this in Python community: https://peps.python.org/pep-0735/ https://peps.python.org/pep-0735/
- nurettin 3y ago
  you would have to maintain two requirements files, and automate their concatenation. It's messy. With a versionless requirements.txt file, there is no guarantee that your project will work on pip install next year.

quickthrower2 3y ago

Of course you can tame your snake with a container.

ildjarn 3y ago

Has anyone figured out to how to “cross-compile” Python? By this I mean creating an app bundle that contains the dependencies but for another platform than the one we are bundling on.

jvolkman 3y ago
Probably not what you're looking for, but I put together a demo of actually cross-compiling a wheel the other day: https://github.com/jvolkman/bazel-pycross-zstandard-example https://github.com/jvolkman/bazel-pycross-zstandard-example But if you just mean that you want to gather the dependencies for a platform other than your build host: this should be possible with the help of Poetry and PDM since they both perform cross-platform resolution.
- ildjarn 3y ago
  rules_pycross looks very interesting and is sorely needed in the Python ecosystem. I will follow its development!

globular-toast 3y ago

Here's my "boring" workflow: 1. Start project (mkdir, git init), 2. Make virtualenv using virtualenvwrapper, 3. Write project.toml file for setuptools, 4. pip install -e . 5. To add deps, add them to pyproject.toml and repeat step 4. Do not pip install deps directly. Do not pin deps to any particular version, but if you have to you can add constraints like >=5 (I need a feature introduced in v5). 6. If you are writing a package to be pip installed by others then you're done. Read setuptools docs for how to build etc. 7. If you also want to build an environment to run your code (e.g. docker image for deployment or serverless deployment etc) use pip-tools to pin your dependencies. (This is the only reason you need requirements.txt). 8. For test dependencies (e.g. pytest) or dev dependencies (e.g. test server) leverage optional dependencies in the pyproject.toml file. This plays very nicely with tools like tox, which you should use. Use pre-commit for linting etc.

atoav 3y ago

The biggest problem about Pythons dependency managment in 2024 is that it stil feels like an afterthought. It is just not straightforward and it is not pythonic. I would go as far as saying that dependency managment with Python is likely more complicated than anything you would normally encounter within the language. As of 2024 poetry is the best solution we have, but even it can come to its limits at times. I work in a position where I develope with poetry and have to deploy without it (using venv), and I do not wish the journey of learning how to do that on anybody.

dagw 3y ago

I'm personally afraid that the python community has bifurcated so far that a single 'correct' solution is doomed to fail. The needs of people writing and deploying HTTP/REST servers is just so different from the people writing PyTorch models and numerical simulations that no tool will ever satisfy the very different needs of both camps. The worst part is that many people developing these tools don't seem to realise this and blindly claim that their tool is The Tool! without having any deep insights into the needs of the other camps.

hereonout2 3y ago
> The needs of people writing and deploying HTTP/REST servers is just so different from the people writing PyTorch models and numerical simulations that no tool will ever satisfy the very different needs of both camps. Is this true though? At the end of it they both need python packages and some system dependencies installed (let's ignore models and data for now). Why is there umpteen tools for doing this in python when there isn't in other languages? I have to deploy both web apps and ML models, the first thing I do is convert any project to use pyproject and poetry. Whilst Poetry comes with it's own issues I've not had a project yet that this doesn't work for and wish the wider community would just settle on one method. Instead we have stuff coming in with various conda incantations, pip, pipx, poetry, setuptools, setup.py! Deployment of ML models needs a decent solution too, half the ML code I get goes and fetches stuff from NLTK or Huggingface at runtime. Some of it (like the LLAMA models) needs various API keys set and EULAs agreed to before it'll run, then pings back to Huggingface each time you run it to check the EULA again! This makes life difficult when trying to deploy and adds this massive dependency on 3rd party services.
- aragilar 3y ago
  But how are you handling those system dependencies (is R and its various packages one?), and how do you know that the packages you install/manage via the python ecosystem work with them? I think the reason Python has so many things is simply because most other languages throw their arms up and say "system stuff, not my problem" (rust is an excellent example of this, the build.rs is basically the same thing as a setup.py, except less standard, and currently lacks from what I've seen is any kind of systematic solution like cibuildwheel), whereas Python has always been trying to do something to address it.

sebra 3y ago

For me it feels like people have always been over-engineering dependency management where the practical issues from not doing it are pretty much non-existent. My approach is to just use Docker, no virtualenvs. I get that you might run into the multiple interpreters issue in theory but across multiple projects in the past 5 years I haven't seen that. Also, this might no longer be true but avoid using Alpine. If you're deploying Django there is no reason to optimize image size and Alpine has a lot of things which are missing (i.e. at least a couple of years ago, wheels where not supported leading very slow build times). I only do a single requirements.txt. Anything which makes your prod and local environment differ is a bad thing in my opinion. Fine black might make my image a couple of mbs larger but why would it matter? On the other hand attempting to figure out why something which works on my machine does not work on prod is always a nightmare. Setting requirements as a range in requirements.txt allows me to automatically get bugfixes without spending time on it (e.g. django>=4.2.3,<4.2.99 django-ninja>=1.0.1,<1.0.99) Again, I might have run into 1-2 issues over the past couple of years from this and I've saved a lot of time. Getting a project running locally should not take more than 1 minute (a couple of .env vars + docker-compose up -d should be enough). The biggest practical issue in dependency management in python is dependencies not pinning their dependencies correctly.

dagw 3y ago
Anything which makes your prod and local environment differ is a bad thing in my opinion Unless you're writing code that only you will deploy to machines that you control, "prod" and "local" will always be different. If you're only targeting a fixed version of a fixed OS on a fixed architecture, then most things are easy. For me "local" is a Mac running ARM, for the person pip installing my tool "prod" might be Linux or Windows. I cannot punt (or I can, but it would greatly diminish the usefulness of the stuff I develop) and say "your prod must equal my local or it won't work", I have to deal with it and I want tools that make this hard problem as easy as possible.

taraparo 3y ago

Coming from Java/Maven I was amazed to see the obscure mess in Python world regarding dependency management. After trying all available tools I finally settled on pdm. I also found it to be more intuitive than poetry.

rq1 3y ago
PDM is a better tool in general, the documentation is not great though. The big plus is: it supports mono-repos with multiple projects, unlike poetry. The answer to some issues by the poetry team (including monorepo one) was along the line: “deal with it”. Fine, switch to PDM.

nicornk 3y ago

Pixi is great, highly recommended!

banditelol 3y ago
Have you compared it with poetry or pip-tools? I'm thinking of trying pixi but still can't muster up energy to do it. Especially since for my use case poetry and pip-tools cover most of it.
baggiponte 3y ago
I like pixi, but I am not likely to make the switch. They don't support pyproject.toml and other standards. This disqualifies it from being a potential "recommended tool" by PyPA or whatever.

posix_monad 3y ago

The "boring" set of tools described here does not enable reproducible artefacts. This is a huge weakness of the ecosystem.

linkdd 3y ago
Of this specific solution yes, of the "ecosystem" no. This has been solved by many other solutions (pipenv, poetry, pdm, etc...).
- barbazoo 3y ago
  I haven’t tested this yet but what’s better about pipfile.lock over whatever pip-compile spits out? It sounds like both are exact versions of packages, no?
  - linkdd 3y ago
    I don't know about pipfile.lock nor pip-compile, once I tested poetry I stuck to it without rethinking it (because I have better things to do than test all the package managers out there). But the poetry.lock contains hashes as well.
  - EdwardDiego 3y ago
    Pipfile.lock is broken if you have wheels wrapping compiled code as it captures the arch in the lock file! Poetry doesn't do this, so you can lock on your M2 Mac and install on x86 fine. Pip-compile in the most common use just creates a requirements.txt with everything pinned to a given version. I think you can do hash stuff with it, but haven't used that part.
    - o11c 3y ago
      If you're building for multiple platforms, it doesn't make sense to lock your dependencies. "Some platforms need newer versions" is the default case; don't make your tools fight it. That said, package managers that do timestamp-based version filtering would be very useful.