8 ms·

Specifying Installation Requirements for Python Projects

55 points by JetSpiegel 5y ago

PaulHoule 5y ago
The deepest problem I see w/ the Python dependency system is the need to be able to declare substitute or alternative dependencies. For instance you should be able to install a numpy that is compiled to take advantage of your specific CPU. You should be able to pick at install time which one you want WITHOUT patching the the source code directory in any way that would make it back to version control.
- marius_k 5y ago
  Do you have an example of how it would be solved in other languages? In python I would create different requirements files together with freeze versions of those. One could contain the standard "numpy" and another would be "numpy @ my-custom-built-numpy-url". I haven't tried it and don't know how it would work in practice.
  - ivoflipse 5y ago
    Yarn has the following: https://classic.yarnpkg.com/en/docs/selective-version-resolutions/#toc-why-would-you-want-to-do-this https://classic.yarnpkg.com/en/docs/selective-version-resolu... This doesn't allow you to use an alternative index, but it does allow you to override packages that restrict their dependencies versions too strictly
- nonameiguess 5y ago
  Assuming you don't want to actually patch Numpy and just want to make sure it's compiled with support for processor features you have available like AVX-512 and SSE2 and what not, you can already do this by just telling pip to prefer source, which means it'll compile it at install time and the underlying build system will detect and compile in support for whatever processor features you can use. I suspect it's the features of the BLAS that will matter, though, a lot more than Numpy itself. The Numpy build system by default prefers MKL, OpenBLAS, then ATLAS, and I don't remember what after that, and it's up to you to compile those with the features you need.
  - PaulHoule 5y ago
    I DO NOT want to get anywhere near compiling dependencies at install time, that is how you make your Python build process entirely irreproducible! I didn’t feel in control of Python builds until I had a wheels only policy. (For one things you can download about 2% of a wheel and get the dependencies to put into an SMT solver and globally solve the dependencies before you build anything. pip, poetry and similar systems use eggs and have to run setup.py to figure out deps which means they solver locally and eagerly and back themselves into corners that they may or may not be able to get out of.) If I have to build from source I want to build a wheel, archive it, then reference the wheel in my repository. I don’t quite have the tooling to make it as easy as it really is.
dannyz 5y ago
I pretty much only do scientific computing with Python so take this with a grain of salt, but ever since I have started using conda I have never looked back. Especially for cross platform projects conda makes things much easier. I would love for the official Python packaging tools to be in a better state so I hope this helps.
- ivoflipse 5y ago
  Conda has the advantage that conda packages are typically repackaged from the Pypi package. Especially om conda-forge, they are curated by people with more experience in packaging, that are more likely to use appropriate version pinning for libraries. Furthermore, conda-forge has the ability to create a new build of the same package with improved version pinnings, which typically doesn't happen on Pypi for already released packages.
- misnome 5y ago
  With the exception that the conda tool itself is so darned slow it can make you give up in frustration rather than wait another ten minutes to be told that the packages conflict. So use https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba , which is an alternative client to conda and a million times faster. (If you are creating environments more than once per month or so).
- acidburnNSA 5y ago
  I also do scientific computing all day every day with Python. I've used the built-in virtual environments using `python -m venv` for years on a project-by-project basis and have never had too much trouble. Yes, finding the right package compiled on Windows sometimes requires a trip to Christoph Gohlke's website, but beyond that it's never been a problem. What does conda give you that is so much better than venvs?
  - 41b696ef1113 5y ago
    I utilize a package that only distributes source or binary through Conda. Building it is very much non-trivial, so as to not go insane, I have to use Conda. Would really rather not use Conda, but bespoke packages + the trivial ability to define a specific Python version ($WORK servers use ancient RedHat with dated Python) has forced my hand. Poetry feels better, if only because it separates out explicit vs transitive dependencies.
mch82 5y ago
> `pyproject-lock.d` Since this PEP is a draft, any comments/thoughts about naming the directory that stores lock files `pyproject-lock.d`? Is it normal to end a directory name with `.d`?
- yodelshady 5y ago
  Reasonably so within linux filesystems where a file with that name might already exist, e.g. within Debian /etc/apt/sources.list is a file and /etc/apt/sources.list.d is a folder. There's probably a more specific convention so I'll await murphy's law.
  - deleted 5y ago
    [deleted]
- egberts 5y ago
  The dot-d notation is there for applications/packages to affect a subsystem without messing with that subsystems’ own configuration settings. Debian makes good use of it but i found that it introduces a smaller problem of ordering (which of the app’s setting to apply firstly for that subsystem’s dot-d collections of apps’ settings. But such conflict of inter-app configuration settings is seemingly rare and often best done directly into such subsystem’s own main config settings.
wodenokoto 5y ago
Any plans to add a toml reader to the standard library?
- Xophmeister 5y ago
  Exactly! I really don’t understand why they went with TOML — with pypackage.toml and now this — when there’s no stdlib TOML support. Given that it therefore requires an extra package, why not go with YAML? If they don’t care about optimising for human-readability, then stick to JSON.
smitty1e 5y ago
As 3.10 is two months away from yeeting, this PEP665 would seem to be part of the 3.11 sizzle. Packaging has long been a weaker suit for python. As with so many other language features (e.g. dataclasses) python seems to let a bunch of art settle in the wild and then offer a nicely refined PEP to capture and implement it.
- ivoflipse 5y ago
  Packaging PEPs aren't really tied to the CPython release cycle unless they impact CPython, like the deprecation of distutils. So if this PEP gets approved, you'll see projects using it much sooner than the release of 3.11

55 points by JetSpiegel 5y ago

The deepest problem I see w/ the Python dependency system is the need to be able to declare substitute or alternative dependencies. For instance you should be able to install a numpy that is compiled to take advantage of your specific CPU. You should be able to pick at install time which one you want WITHOUT patching the the source code directory in any way that would make it back to version control.

marius_k 5y ago
Do you have an example of how it would be solved in other languages? In python I would create different requirements files together with freeze versions of those. One could contain the standard "numpy" and another would be "numpy @ my-custom-built-numpy-url". I haven't tried it and don't know how it would work in practice.
- ivoflipse 5y ago
  Yarn has the following: https://classic.yarnpkg.com/en/docs/selective-version-resolutions/#toc-why-would-you-want-to-do-this https://classic.yarnpkg.com/en/docs/selective-version-resolu... This doesn't allow you to use an alternative index, but it does allow you to override packages that restrict their dependencies versions too strictly
nonameiguess 5y ago
Assuming you don't want to actually patch Numpy and just want to make sure it's compiled with support for processor features you have available like AVX-512 and SSE2 and what not, you can already do this by just telling pip to prefer source, which means it'll compile it at install time and the underlying build system will detect and compile in support for whatever processor features you can use. I suspect it's the features of the BLAS that will matter, though, a lot more than Numpy itself. The Numpy build system by default prefers MKL, OpenBLAS, then ATLAS, and I don't remember what after that, and it's up to you to compile those with the features you need.
- PaulHoule 5y ago
  I DO NOT want to get anywhere near compiling dependencies at install time, that is how you make your Python build process entirely irreproducible! I didn’t feel in control of Python builds until I had a wheels only policy. (For one things you can download about 2% of a wheel and get the dependencies to put into an SMT solver and globally solve the dependencies before you build anything. pip, poetry and similar systems use eggs and have to run setup.py to figure out deps which means they solver locally and eagerly and back themselves into corners that they may or may not be able to get out of.) If I have to build from source I want to build a wheel, archive it, then reference the wheel in my repository. I don’t quite have the tooling to make it as easy as it really is.

dannyz 5y ago

I pretty much only do scientific computing with Python so take this with a grain of salt, but ever since I have started using conda I have never looked back. Especially for cross platform projects conda makes things much easier. I would love for the official Python packaging tools to be in a better state so I hope this helps.

ivoflipse 5y ago
Conda has the advantage that conda packages are typically repackaged from the Pypi package. Especially om conda-forge, they are curated by people with more experience in packaging, that are more likely to use appropriate version pinning for libraries. Furthermore, conda-forge has the ability to create a new build of the same package with improved version pinnings, which typically doesn't happen on Pypi for already released packages.
misnome 5y ago
With the exception that the conda tool itself is so darned slow it can make you give up in frustration rather than wait another ten minutes to be told that the packages conflict. So use https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba , which is an alternative client to conda and a million times faster. (If you are creating environments more than once per month or so).
acidburnNSA 5y ago
I also do scientific computing all day every day with Python. I've used the built-in virtual environments using `python -m venv` for years on a project-by-project basis and have never had too much trouble. Yes, finding the right package compiled on Windows sometimes requires a trip to Christoph Gohlke's website, but beyond that it's never been a problem. What does conda give you that is so much better than venvs?
- 41b696ef1113 5y ago
  I utilize a package that only distributes source or binary through Conda. Building it is very much non-trivial, so as to not go insane, I have to use Conda. Would really rather not use Conda, but bespoke packages + the trivial ability to define a specific Python version ($WORK servers use ancient RedHat with dated Python) has forced my hand. Poetry feels better, if only because it separates out explicit vs transitive dependencies.

mch82 5y ago

> `pyproject-lock.d` Since this PEP is a draft, any comments/thoughts about naming the directory that stores lock files `pyproject-lock.d`? Is it normal to end a directory name with `.d`?

yodelshady 5y ago
Reasonably so within linux filesystems where a file with that name might already exist, e.g. within Debian /etc/apt/sources.list is a file and /etc/apt/sources.list.d is a folder. There's probably a more specific convention so I'll await murphy's law.
- deleted 5y ago
  [deleted]
egberts 5y ago
The dot-d notation is there for applications/packages to affect a subsystem without messing with that subsystems’ own configuration settings. Debian makes good use of it but i found that it introduces a smaller problem of ordering (which of the app’s setting to apply firstly for that subsystem’s dot-d collections of apps’ settings. But such conflict of inter-app configuration settings is seemingly rare and often best done directly into such subsystem’s own main config settings.

wodenokoto 5y ago

Any plans to add a toml reader to the standard library?

Xophmeister 5y ago
Exactly! I really don’t understand why they went with TOML — with pypackage.toml and now this — when there’s no stdlib TOML support. Given that it therefore requires an extra package, why not go with YAML? If they don’t care about optimising for human-readability, then stick to JSON.

smitty1e 5y ago

As 3.10 is two months away from yeeting, this PEP665 would seem to be part of the 3.11 sizzle. Packaging has long been a weaker suit for python. As with so many other language features (e.g. dataclasses) python seems to let a bunch of art settle in the wild and then offer a nicely refined PEP to capture and implement it.

ivoflipse 5y ago
Packaging PEPs aren't really tied to the CPython release cycle unless they impact CPython, like the deprecation of distutils. So if this PEP gets approved, you'll see projects using it much sooner than the release of 3.11