15 ms·

Distributions vs. Releases: Why Python Packaging Is Hard

131 points by BerislavLopac 8y ago

alexbecker 8y ago
Author here, happy to answer any questions.
- daliusd 8y ago
  Node with npm must have the same problem. How they handle that and why python is not doing the same? If it should of course.
  - alexbecker 8y ago
    NPM doesn't have the same problem, because they only allow a single file per release (which also means there is no way to distribute platform-specific builds).
    - daliusd 8y ago
      There is a way. It seems node-gyp and node-pre-gyp is used for that. Here is project that has precompiled binaries http://sharp.pixelplumbing.com/en/stable/install/ http://sharp.pixelplumbing.com/en/stable/install/ .
      - alexbecker 8y ago
        It looks like they don't include the binary in the package at all, and instead the package has custom install code that downloads it during install. Which works, but it completely bypasses the package manager and would certainly take me by surprise as a user.
        je42 8y ago
        jep. it can cause hard to trace errors and makes reproducible build more difficult.
        daliusd 8y ago
        That's my point - it works. What's wrong with this approach? I have not investigated neither python nor node way (while I use both) and as user I don't care (maybe I should, I don't know).
        alexbecker 8y ago
        Expanding on jep42's answer, there are a few problems: - The package manager can't tell whether the package will be able to support your system or not, so if this is e.g. an optional dependency your install will probably break instead of the package being skipped. - This results in an unexpected and hard-to-foresee dependency on some other non-npm server, which could disappear, be blocked by a firewall, etc. - Standard tools will not understand what this package is doing and their assumptions may be broken, e.g. when trying to make builds repeatable.
        daliusd 8y ago
        Looked into this a little bit deeper. There are multiple packages in node that can do prebuilts handling, e.g. sharp is using this https://github.com/prebuild/prebuild-install https://github.com/prebuild/prebuild-install so there are multiple solutions. Now answering your notes: - It is not very different in python from practical point of view. I understand that while from theoretical point of view it should be different but it is not. E.g. when I was working on Windows I met situations where package is supported but there is no binary package. In such situations it falls back to compilation from source code and for some packages you need to perform "ancient Zulu dance" to get it compiled on Windows. Sometimes I was in the mood to do that sometimes I was not. However from user perspective even if package manager can tell that it is supported it does not help much. In practice, both python and node packages compile successfully quite frequently and in the end package is almost always supported. Node supports optional dependencies and failure to install package either as prebuilt or built from source code will simply skip it and will not fail the whole install. - In practice node packages usually use github and sometimes amazon s3 for prebuilts (at least based on quick analysis). There are still at least two systems but github and amazon s3 both seems to be good enough. - I don't see problem with repeatable builds in node's case. Could you elaborate more here? I see one problem with node's approach however. In case you want to have your own prebuilts (e.g. you have your own server) then you will end up with one big problem. You will need to have override each package with prebuilts separately. Overall I think node did good thing by not trying to do everything and allowing community to figure out prebuilt solution. Problem is that there are multiple solutions. Python approach most probably will work out as better solution in the long run. In the end, most probably both solutions will be equally good and will look quite similar from user perspective.
- vamega 8y ago
  What is the key advantage of PyDist over Artifactory/bintray ? That's the system most companies I know of use, and I was wondering what makes PyDist compelling.
  - alexbecker 8y ago
    PyDist is specifically for Python packages, so it's 1) a better fit in this niche (for example, it mirrors PyPI so you can install public and private packages through a single index, and you won't be broken by packages deleted from PyPI), and 2) it's much cheaper if you don't need everything else Artifactory offers.
    - sl1ck731 8y ago
      I believe Artifactory proxies and caches PyPi as well via remote repos.
      - alexbecker 8y ago
        Ah, looks like you can set that up. But it's not as convenient, because 1) you have to provision the remote repository and 2) instead of transparently installing from PyPI, you have to specify the remote repository each time you install from it.
        jbaruch_s 8y ago
        It is exactly as convenient (if not more). You set up a set of 3 repositories (local for your modules, remove to proxy the PyPI, and virtual, which unifies them under a single URL (which solves the #2 you mentioned). So, both points are incorrect.
- 4dahalibut 8y ago
  Why isn't the obvious answer to do this change? https://github.com/pypa/packaging.python.org/issues/564#issuecomment-428553391 https://github.com/pypa/packaging.python.org/issues/564#issu...
  - alexbecker 8y ago
    Because right now you can only upload individual distributions to PyPI. A release is implicitly created when you upload the first one. If PyPI implemented that change, you could only have one distribution per release. The proper fix would be to make publishing a release a separate operation. But that breaks all existing tooling and workflows.
    - MrOxiMoron 8y ago
      Make it a configuration option for the package on pypi, then tooling can migrate slowly and at some point it can become the default for new packages. If then at some point someone uses old tooling the downside would be that they might need to do a manual publishing step to actually publish their package, but at least you don't have the problem of publishing before you are ready.
- falsedan 8y ago
  Do you have any internal guidelines for ensuring these sorts of public announcements have a neutral, compassionate tone? To me, it feels like it would be easy to read this article as being more empathetic to the tools than the users, and to interpret it as blaming the users for using the tools wrong: authors for adding a new distribution to a release post-hoc, instead of creating a new release; or consumers for not carefully vetting packages to ensure there’s a matching distribution for their platform & not using source distros. I feel like, as a representative of the tool it’s easy for readers to default to understanding any criticism (even the slightest) of user behaviour & no discussion about tool behaviour as being defensive of the tools. I know the Python package tools community value their users and would never dream of suggesting that they’re using the tools the wrong way. How important do you feel it is to recognise/appreciate those users in user-facing messaging?
- kalefranz 8y ago
  Honest question, is pydist just a hosted version of devpi (https://github.com/devpi/devpi https://github.com/devpi/devpi)? What features distinguish pydist from devpi?
boapnuaput 8y ago
Nix's usage of source hashes to pin every package is more and more prescient, as each language's custom-written package manager reinvents the wheel. (Pun indented!)
- Godel_unicode 8y ago
  There are plenty of situations where the output of the build process is not deterministic on only the source.
  - coherentpony 8y ago
    Example?
    - Godel_unicode 8y ago
      Install Python from source with and without readline-devel in the ld library path. Or any other piece of C software with macros that depend on the build environment. Same .c file in, different binary out.
      - mikepurvis 8y ago
        Homebrew has had to deal with this in its sometimes-painful transition over the years from being a build-from-source system to a binary package manager. Suddenly you can't just have some package require the boost package be built using the --with-python flag, you either need to have it on in the bottled version (but wait, then boost has a dependency on Python, is that okay?), or you have to ship a separate boost-python, like Debian (but wait, then it's no longer a 1:1 mapping of recipes to bottles, is that okay?).
      - m_mueller 8y ago
        This sort of thing is IMO exactly why containers have merit. Let the software vendor decide what to distribute or others jump in and fill the gap. Powerusers can still read the dockerfiles and decide what do do or even build themselves. When it's built the image is well defined, i.e. not depending on host environment, including files stem except volume mounts. When will we get a whole desktop distro like this?
your-nanny 8y ago
https://github.com/pypa/pip/issues/5874 https://github.com/pypa/pip/issues/5874 This seems like reasonable suggestion.
mlthoughts2018 8y ago
It’s beyond disappointing that the article does not discuss conda and conda’s build variant capabilities [0]. Maybe 2 years ago there was still room for honest debate among Python package tooling. There just sincerely isn’t anymore. [0]: https://conda.io/projects/conda-build/en/latest/resources/variants.html#build-variants https://conda.io/projects/conda-build/en/latest/resources/va...
- phowon 8y ago
  As someone who aggressively uses conda, I do think one of its downsides is how heavy it is. I agree that it's a good one-stop-shop if you want to get an environment up and running with no issues. But if you're not doing any scientific computing for a given project, it might help to use something more lightweight.
  - mlthoughts2018 8y ago
    I use it with good success managing non-scientific and non-Python packages. I think especially with conda-forge, it is just an all purpose package manager and package distributor. Using the miniconda installer often helps keep it lightweight too, no need for Anaconda in many use cases.
  - lsorber 8y ago
    What makes conda heavy?
  - nirse 8y ago
    I've had to 'debug' quite a few (ana)conda setups, mostly ones that were used for scientific work, where the whole python setup got so messed up that nothing was quite working anymore, both system and conda pythons. I advise against it.
    - slfnflctd 8y ago
      > the whole python setup got so messed up that nothing was quite working anymore, both system and conda pythons This seems like the biggest downside I've seen mentioned thus far. Can you (or anyone) offer more detail as to how using Conda resulted in corrupting your Python installation? It it similar to what can happen with game mods, or is there something else going on?
malkia 8y ago
If you take this further, and can compile any of the "C" code (open source), then you can link that existing code as one single binary - e.g. "python" - basically python with all compiled code needed, thus: - You no longer need .dll, .so, etc. to go along, you can package the rest of the .py files into a .zip and slap it at the end of the now "fat" python binary, hence effectively having one executable - a bit of a "go" style releasing... - This can even have smaller sizes, but careful (with FFI) - because only the code that is exposed to be used from Python would get linked in the fat "python" binary.. that is - with "FFI" is more tricky, as you have to force symbol to be kept - You can obviously do that with other runtimes - java - e.g. your whole "java" can be down to one executable really But to get there, you need sane build system, in order to explain these dependencies outside of python/java/whatever... And obviously the source code (although precompiled static libs should work in this scheme too)... The goal again - just to have one thing, and most importantly no .so, .dll, .dylib lying around,
- icebraining 8y ago
  This is what tools like py2exe and PyInstaller do.
  - paavoova 8y ago
    You can do this with plain zip files as Python can unpack and execute in-memory. But you have to extract non-Python code such as C libraries at runtime somewhere so the OS can link them. I would be interested to know how the packagers you mentioned get around this, or if they too use temporary files and unpack.
    - dead_mall 8y ago
      This is exactly how I package my python scripts. The answer to your question: Memfd https://magisterquis.github.io/2018/03/31/in-memory-only-elf-execution.html https://magisterquis.github.io/2018/03/31/in-memory-only-elf... And zipimport pip package
      - 8y ago
nirse 8y ago
https://xkcd.com/1987/ https://xkcd.com/1987/ (edit: this is for anyone who mentions conda in here)
- mlthoughts2018 8y ago
  How so? Conda is strictly simpler than alternatives, by a solid order of magnitude at least.
perlgeek 8y ago
Most of my pain with python packaging comes from incompatible changes in the toolchain. Some years ago, pip started distrusting HTTP mirrors, and while you could add some options to force it use HTTP, these options weren't present in previous pip versions. Which meant that you now had to provide options depending on the pip version -- which is harder if you don't call pip directly (for example through dh-virtaualenv). We switched to HTTPS, but one with a TLS cert signed by a company-internal CA. Getting pip to trust another CA is non-obvious, and again depends on the pip version. So another version-dependent treatment necessary. Then pip suddenly moved all of its implementation from pip.something to pip._internal.something, stating that it was documented all along that you shouldn't import anything from the pip.* namespace. But, the package names all didn't start with an underscore, so, by python convention, looked like perfectly fine public classes and methods. Moreover there simply isn't any public API to do things you really don't want to reimplement yourself, like parsing `requirements.txt` files. As soon as you want to do anything slightly non-standard with pip/distutils/setuptools, you find a solution pretty quickly, and that solution turns into a nightmare when you upgrade any of the components involved. Also, finding a local pypi mirror that lets you inject your own packages and deals with wheels correctly and doesn't generally suck and is somewhat easy to use... not easy.
- icebraining 8y ago
  Agreed; for something that managed to become the de-facto solution for Python package installation, pip always felt surprisingly unreliable to me. My solution at a small company, where we couldn't waste time with this crap, was to vendor everything: when a developer updated the requirements.txt, they would also install the packages in a project directory (using "pip install -r requirements.txt -t ./dependencies") and commit that. That way, the rest of the process (CI, packaging, etc) could just checkout the repo, set PYTHONPATH=./dependencies and ignore pip completely.
MrOxiMoron 8y ago
shouldn't pip then also look at the hashes (if provided) to determine which package to install? If it worked at one time it should keep working if there are "better" options available now. Consistency is important here.
- di 8y ago
  The article links to https://github.com/pypa/pip/issues/5874 https://github.com/pypa/pip/issues/5874 which proposes exactly this.
  - alexbecker 8y ago
    In fact I'm hoping to get some time this weekend to make a PR implementing this! (I'm assuming the reason you/Donald Stufft haven't done so is time, rather than disagreement over whether this is a good solution or some unseen obstacle to implementing it.)
xiaodai 8y ago
How does R avoid this then?
- nestorD 8y ago
  I believe that, for R, releases are synonymous with distributions. The code is compiled on the user's machine (I once had a fortran error message while installing an R package). If reproducibility matters, you can also use Microsoft R Open to get your packages from a frozen snapshot of CRAN : https://mran.microsoft.com/ https://mran.microsoft.com/
- kristjansson 8y ago
  R / CRAN provides precompiled distributions for macOS and windows machines, but builds those distributions on its own infrastructure so they’re available more-or-less the same time the source distribution is. Additionally, R/CRAN do not try to solve the problem of binary releases on Linux distros, and instead default to source distributions or defer to the OS package manager
olah_1 8y ago
I only got interested in Python again after I became aware of tools like Poetry https://poetry.eustace.io/ https://poetry.eustace.io/
- brightball 8y ago
  That looks really sharp.
- ptx 8y ago
  Looks very interesting, but the part of the installation instructions where it says that its directory "will be in your $PATH environment variable" worries me – is the installer taking the liberty of messing with my configuration without asking first?

131 points by BerislavLopac 8y ago

alexbecker 8y ago

Author here, happy to answer any questions.

daliusd 8y ago
Node with npm must have the same problem. How they handle that and why python is not doing the same? If it should of course.
- alexbecker 8y ago
  NPM doesn't have the same problem, because they only allow a single file per release (which also means there is no way to distribute platform-specific builds).
  - daliusd 8y ago
    There is a way. It seems node-gyp and node-pre-gyp is used for that. Here is project that has precompiled binaries http://sharp.pixelplumbing.com/en/stable/install/ http://sharp.pixelplumbing.com/en/stable/install/ .
    - alexbecker 8y ago
      It looks like they don't include the binary in the package at all, and instead the package has custom install code that downloads it during install. Which works, but it completely bypasses the package manager and would certainly take me by surprise as a user.
      - je42 8y ago
        jep. it can cause hard to trace errors and makes reproducible build more difficult.
      - daliusd 8y ago
        That's my point - it works. What's wrong with this approach? I have not investigated neither python nor node way (while I use both) and as user I don't care (maybe I should, I don't know).
        alexbecker 8y ago
        Expanding on jep42's answer, there are a few problems: - The package manager can't tell whether the package will be able to support your system or not, so if this is e.g. an optional dependency your install will probably break instead of the package being skipped. - This results in an unexpected and hard-to-foresee dependency on some other non-npm server, which could disappear, be blocked by a firewall, etc. - Standard tools will not understand what this package is doing and their assumptions may be broken, e.g. when trying to make builds repeatable.
        daliusd 8y ago
        Looked into this a little bit deeper. There are multiple packages in node that can do prebuilts handling, e.g. sharp is using this https://github.com/prebuild/prebuild-install https://github.com/prebuild/prebuild-install so there are multiple solutions. Now answering your notes: - It is not very different in python from practical point of view. I understand that while from theoretical point of view it should be different but it is not. E.g. when I was working on Windows I met situations where package is supported but there is no binary package. In such situations it falls back to compilation from source code and for some packages you need to perform "ancient Zulu dance" to get it compiled on Windows. Sometimes I was in the mood to do that sometimes I was not. However from user perspective even if package manager can tell that it is supported it does not help much. In practice, both python and node packages compile successfully quite frequently and in the end package is almost always supported. Node supports optional dependencies and failure to install package either as prebuilt or built from source code will simply skip it and will not fail the whole install. - In practice node packages usually use github and sometimes amazon s3 for prebuilts (at least based on quick analysis). There are still at least two systems but github and amazon s3 both seems to be good enough. - I don't see problem with repeatable builds in node's case. Could you elaborate more here? I see one problem with node's approach however. In case you want to have your own prebuilts (e.g. you have your own server) then you will end up with one big problem. You will need to have override each package with prebuilts separately. Overall I think node did good thing by not trying to do everything and allowing community to figure out prebuilt solution. Problem is that there are multiple solutions. Python approach most probably will work out as better solution in the long run. In the end, most probably both solutions will be equally good and will look quite similar from user perspective.
vamega 8y ago
What is the key advantage of PyDist over Artifactory/bintray ? That's the system most companies I know of use, and I was wondering what makes PyDist compelling.
- alexbecker 8y ago
  PyDist is specifically for Python packages, so it's 1) a better fit in this niche (for example, it mirrors PyPI so you can install public and private packages through a single index, and you won't be broken by packages deleted from PyPI), and 2) it's much cheaper if you don't need everything else Artifactory offers.
  - sl1ck731 8y ago
    I believe Artifactory proxies and caches PyPi as well via remote repos.
    - alexbecker 8y ago
      Ah, looks like you can set that up. But it's not as convenient, because 1) you have to provision the remote repository and 2) instead of transparently installing from PyPI, you have to specify the remote repository each time you install from it.
      - jbaruch_s 8y ago
        It is exactly as convenient (if not more). You set up a set of 3 repositories (local for your modules, remove to proxy the PyPI, and virtual, which unifies them under a single URL (which solves the #2 you mentioned). So, both points are incorrect.
4dahalibut 8y ago
Why isn't the obvious answer to do this change? https://github.com/pypa/packaging.python.org/issues/564#issuecomment-428553391 https://github.com/pypa/packaging.python.org/issues/564#issu...
- alexbecker 8y ago
  Because right now you can only upload individual distributions to PyPI. A release is implicitly created when you upload the first one. If PyPI implemented that change, you could only have one distribution per release. The proper fix would be to make publishing a release a separate operation. But that breaks all existing tooling and workflows.
  - MrOxiMoron 8y ago
    Make it a configuration option for the package on pypi, then tooling can migrate slowly and at some point it can become the default for new packages. If then at some point someone uses old tooling the downside would be that they might need to do a manual publishing step to actually publish their package, but at least you don't have the problem of publishing before you are ready.
falsedan 8y ago
Do you have any internal guidelines for ensuring these sorts of public announcements have a neutral, compassionate tone? To me, it feels like it would be easy to read this article as being more empathetic to the tools than the users, and to interpret it as blaming the users for using the tools wrong: authors for adding a new distribution to a release post-hoc, instead of creating a new release; or consumers for not carefully vetting packages to ensure there’s a matching distribution for their platform & not using source distros. I feel like, as a representative of the tool it’s easy for readers to default to understanding any criticism (even the slightest) of user behaviour & no discussion about tool behaviour as being defensive of the tools. I know the Python package tools community value their users and would never dream of suggesting that they’re using the tools the wrong way. How important do you feel it is to recognise/appreciate those users in user-facing messaging?
kalefranz 8y ago
Honest question, is pydist just a hosted version of devpi (https://github.com/devpi/devpi https://github.com/devpi/devpi)? What features distinguish pydist from devpi?

boapnuaput 8y ago

Nix's usage of source hashes to pin every package is more and more prescient, as each language's custom-written package manager reinvents the wheel. (Pun indented!)

Godel_unicode 8y ago
There are plenty of situations where the output of the build process is not deterministic on only the source.
- coherentpony 8y ago
  Example?
  - Godel_unicode 8y ago
    Install Python from source with and without readline-devel in the ld library path. Or any other piece of C software with macros that depend on the build environment. Same .c file in, different binary out.
    - mikepurvis 8y ago
      Homebrew has had to deal with this in its sometimes-painful transition over the years from being a build-from-source system to a binary package manager. Suddenly you can't just have some package require the boost package be built using the --with-python flag, you either need to have it on in the bottled version (but wait, then boost has a dependency on Python, is that okay?), or you have to ship a separate boost-python, like Debian (but wait, then it's no longer a 1:1 mapping of recipes to bottles, is that okay?).
    - m_mueller 8y ago
      This sort of thing is IMO exactly why containers have merit. Let the software vendor decide what to distribute or others jump in and fill the gap. Powerusers can still read the dockerfiles and decide what do do or even build themselves. When it's built the image is well defined, i.e. not depending on host environment, including files stem except volume mounts. When will we get a whole desktop distro like this?

your-nanny 8y ago

https://github.com/pypa/pip/issues/5874 https://github.com/pypa/pip/issues/5874 This seems like reasonable suggestion.

mlthoughts2018 8y ago

It’s beyond disappointing that the article does not discuss conda and conda’s build variant capabilities [0]. Maybe 2 years ago there was still room for honest debate among Python package tooling. There just sincerely isn’t anymore. [0]: https://conda.io/projects/conda-build/en/latest/resources/variants.html#build-variants https://conda.io/projects/conda-build/en/latest/resources/va...

phowon 8y ago
As someone who aggressively uses conda, I do think one of its downsides is how heavy it is. I agree that it's a good one-stop-shop if you want to get an environment up and running with no issues. But if you're not doing any scientific computing for a given project, it might help to use something more lightweight.
- mlthoughts2018 8y ago
  I use it with good success managing non-scientific and non-Python packages. I think especially with conda-forge, it is just an all purpose package manager and package distributor. Using the miniconda installer often helps keep it lightweight too, no need for Anaconda in many use cases.
- lsorber 8y ago
  What makes conda heavy?
- nirse 8y ago
  I've had to 'debug' quite a few (ana)conda setups, mostly ones that were used for scientific work, where the whole python setup got so messed up that nothing was quite working anymore, both system and conda pythons. I advise against it.
  - slfnflctd 8y ago
    > the whole python setup got so messed up that nothing was quite working anymore, both system and conda pythons This seems like the biggest downside I've seen mentioned thus far. Can you (or anyone) offer more detail as to how using Conda resulted in corrupting your Python installation? It it similar to what can happen with game mods, or is there something else going on?

malkia 8y ago

If you take this further, and can compile any of the "C" code (open source), then you can link that existing code as one single binary - e.g. "python" - basically python with all compiled code needed, thus: - You no longer need .dll, .so, etc. to go along, you can package the rest of the .py files into a .zip and slap it at the end of the now "fat" python binary, hence effectively having one executable - a bit of a "go" style releasing... - This can even have smaller sizes, but careful (with FFI) - because only the code that is exposed to be used from Python would get linked in the fat "python" binary.. that is - with "FFI" is more tricky, as you have to force symbol to be kept - You can obviously do that with other runtimes - java - e.g. your whole "java" can be down to one executable really But to get there, you need sane build system, in order to explain these dependencies outside of python/java/whatever... And obviously the source code (although precompiled static libs should work in this scheme too)... The goal again - just to have one thing, and most importantly no .so, .dll, .dylib lying around,

icebraining 8y ago
This is what tools like py2exe and PyInstaller do.
- paavoova 8y ago
  You can do this with plain zip files as Python can unpack and execute in-memory. But you have to extract non-Python code such as C libraries at runtime somewhere so the OS can link them. I would be interested to know how the packagers you mentioned get around this, or if they too use temporary files and unpack.
  - dead_mall 8y ago
    This is exactly how I package my python scripts. The answer to your question: Memfd https://magisterquis.github.io/2018/03/31/in-memory-only-elf-execution.html https://magisterquis.github.io/2018/03/31/in-memory-only-elf... And zipimport pip package
    - 8y ago

nirse 8y ago

https://xkcd.com/1987/ https://xkcd.com/1987/ (edit: this is for anyone who mentions conda in here)

mlthoughts2018 8y ago
How so? Conda is strictly simpler than alternatives, by a solid order of magnitude at least.

perlgeek 8y ago

Most of my pain with python packaging comes from incompatible changes in the toolchain. Some years ago, pip started distrusting HTTP mirrors, and while you could add some options to force it use HTTP, these options weren't present in previous pip versions. Which meant that you now had to provide options depending on the pip version -- which is harder if you don't call pip directly (for example through dh-virtaualenv). We switched to HTTPS, but one with a TLS cert signed by a company-internal CA. Getting pip to trust another CA is non-obvious, and again depends on the pip version. So another version-dependent treatment necessary. Then pip suddenly moved all of its implementation from pip.something to pip._internal.something, stating that it was documented all along that you shouldn't import anything from the pip.* namespace. But, the package names all didn't start with an underscore, so, by python convention, looked like perfectly fine public classes and methods. Moreover there simply isn't any public API to do things you really don't want to reimplement yourself, like parsing `requirements.txt` files. As soon as you want to do anything slightly non-standard with pip/distutils/setuptools, you find a solution pretty quickly, and that solution turns into a nightmare when you upgrade any of the components involved. Also, finding a local pypi mirror that lets you inject your own packages and deals with wheels correctly and doesn't generally suck and is somewhat easy to use... not easy.

icebraining 8y ago
Agreed; for something that managed to become the de-facto solution for Python package installation, pip always felt surprisingly unreliable to me. My solution at a small company, where we couldn't waste time with this crap, was to vendor everything: when a developer updated the requirements.txt, they would also install the packages in a project directory (using "pip install -r requirements.txt -t ./dependencies") and commit that. That way, the rest of the process (CI, packaging, etc) could just checkout the repo, set PYTHONPATH=./dependencies and ignore pip completely.

MrOxiMoron 8y ago

shouldn't pip then also look at the hashes (if provided) to determine which package to install? If it worked at one time it should keep working if there are "better" options available now. Consistency is important here.

di 8y ago
The article links to https://github.com/pypa/pip/issues/5874 https://github.com/pypa/pip/issues/5874 which proposes exactly this.
- alexbecker 8y ago
  In fact I'm hoping to get some time this weekend to make a PR implementing this! (I'm assuming the reason you/Donald Stufft haven't done so is time, rather than disagreement over whether this is a good solution or some unseen obstacle to implementing it.)

xiaodai 8y ago

How does R avoid this then?

nestorD 8y ago
I believe that, for R, releases are synonymous with distributions. The code is compiled on the user's machine (I once had a fortran error message while installing an R package). If reproducibility matters, you can also use Microsoft R Open to get your packages from a frozen snapshot of CRAN : https://mran.microsoft.com/ https://mran.microsoft.com/
kristjansson 8y ago
R / CRAN provides precompiled distributions for macOS and windows machines, but builds those distributions on its own infrastructure so they’re available more-or-less the same time the source distribution is. Additionally, R/CRAN do not try to solve the problem of binary releases on Linux distros, and instead default to source distributions or defer to the OS package manager

olah_1 8y ago

I only got interested in Python again after I became aware of tools like Poetry https://poetry.eustace.io/ https://poetry.eustace.io/

brightball 8y ago
That looks really sharp.
ptx 8y ago
Looks very interesting, but the part of the installation instructions where it says that its directory "will be in your $PATH environment variable" worries me – is the installer taking the liberty of messing with my configuration without asking first?