5 ms·

If only its package management were as easy as its syntax... I wish pip worked the same way as npm: -g flag installs it globally, otherwise it creates a local

by umvi 7y ago

If only its package management were as easy as its syntax... I wish pip worked the same way as npm: -g flag installs it globally, otherwise it creates a local "python_modules" folder I can delete at any time. And optionally I can save the dependency versioning info to some package.json... Instead, pip is a nightmarish experience where it fails half the time and I have no idea where anything is being installed to and I'm not sure if I'm supposed to use sudo or not and I'm not sure if I'm supposed to use pip or pip3, etc.

jbay808 7y ago
Use Conda envs!
- tachyonbeam 7y ago
  I feel uncomfortable with the fact that people feel a third-party solution is the best way to solve this mess. It can also get messy when packages installed with pip, pip3, conda and apt are all entangled with one another in various ways.
  - adamson 7y ago
    It’s unfortunate that it’s third party, but conda has the unquestionable advantage of being the only Python-centric packaging system that has a reasonable shared binary library story
    - geofft 7y ago
      I'm curious, do you not find wheels + manylinux reasonable? I agree that until recently, Conda definitely had that advantage, but now that you can `pip install scipy` and have that get you a working library and not try to compile things on your machine what does Conda offer beyond that? I guess one thing Conda has that the pip ecosystem doesn't is that it supports installing non-Python shared libraries like libcurl on their own. Is that an advantage? (We absolutely could replicate that in the pip ecosystem if that was worth doing, and it's even not totally unprecedented to have non-Python binaries on PyPI.)
      - jbay808 7y ago
        Does pip allow version number dependencies? Conda is able to upgrade/downgrade packages to resolve conflicts, whereas pip just seems to check if a package exists and shrugs when there's a version conflict.
        geofft 7y ago
        pip does handle versioned dependencies and ranges, and know enough to upgrade existing packages when needed to resolve an upgrade. Its resolver isn't currently as complete as Conda's - see https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 . (On the other hand, the fact that Conda uses a real constraint solver has apparently been causing trouble at my day job when it gets stuck exploring some area of the solution space and doesn't install your packages.... so for both pip and conda you're probably better off not relying too hard on dependency resolution and specifying the versions of as many things as you can.)
      - tachyonbeam 7y ago
        I think it would definitely be great if pip could install non-python dependencies. One problem right now is that many projects will tell you to just pip install xyz. You execute that, things start building, and the process fails partway with some cryptic message because you're missing an external dependency. You figure out which one, you install it, start again, and another dependency is missing. Rinse and repeat. It's definitely not a turnkey solution, and this issue trips up newcomers all the time. With respect to versioning, I think pip should be way more strict. It should force you to freeze dependency versions before uploading to pipy, not accept "libxyz > 3.5", but require a fixed range or single version. That would make packages much less likely to break later because newer versions of their dependencies don't work the same way anymore.
  - krapht 7y ago
    ... the same thing happens if you mix stuff in your usr/bin directory that isn't managed by your system package manager. The solution is: don't mix your package environments. Use a conda environment. Just like in Linux, you'd use a container. If you wait for the Python steering committee to fix pip you'll be waiting a long time.
- sidlls 7y ago
  Conda doesn't really solve the packaging problems of python. It can make them worse if conda doesn't have a package you need.
  - dagw 7y ago
    Conda supports pip installing into a conda environment if there is no conda package.
    - sidlls 7y ago
      Yes, exactly my point: then you have to potentially deal with conflicting dependencies between pip and conda packages. This happens and it's a pain to deal with.
- KaiserPro 7y ago
  please no. As someone who has to look after a bunch of servers used by researchers, conda is the biggest cause of things failing for everyone. conda installs stuff all over the place and its very easy to install conflicting stuff in the same place. I just wish that either venv was automatic, or at least the second thing you learn in python.
  - jbay808 7y ago
    Are you sure you're talking about Conda's virtual envs? Everything gets installed into the folder /envs/my-env-name.
dangom 7y ago
Is there any particular bottleneck towards getting this to work with Python? Would not a simple pip wrapper be enough for the task?
deleted 7y ago
[deleted]
gilad 7y ago
Packaging is definitely recognized as an issue; e.g. here: https://cecinestpasun.com/entries/where-do-you-see-python-in-10-years/ https://cecinestpasun.com/entries/where-do-you-see-python-in...
kingbirdy 7y ago
That's what virtualenvs are for. Admittedly they're not a perfect solution but the tooling is quite good nowadays.
sametmax 7y ago
Use virtualenv. Always. Python 3 comes with it as "python -m venv". Once in the virtualenv, you don't have to worry about the various pip forms and effects, you can just pio install. You can get fancier than that of course, but that's what works with stock python, on all OS.
geofft 7y ago
Here's a simple but less-than-completely-documented way to keep Python package management under control: 1. Don't install anything globally. Don't pip install --user, definitely don't sudo pip install. 2. For each project you want to work on, create a venv. Yes, there are tools for this, but the base venv tool is totally fine. (venv should be included in your Python, but a few distributors like Debian put it in a separate package - install it from them if needed.) Use python3 -m venv ~/some/directory to create a venv. From here on out 3. As a first step, upgrade pip: ~/some/directory/bin/pip install -U pip. 4. Install things with ~/some/directory/bin/pip install. 5. Run Python with ~/some/directory/bin/python. Slightly advanced move: make a requirements.txt file (you can use .../bin/pip freeze as a starting point) and use .../pip install -r requirements.txt. That way, if you get any sort of package resolution error, you can just delete your venv and make a new one. (Downloads are cached, so this isn't super annoying to do.) A "project" can either be actual Python development, or just a place to install some Python programs and run them out of the resulting bin/. (Edit: Yes, I know about activate, see the replies below for why I don't recommend it. With these rules, you get to say "Never ever type pip, only .../bin/pip", which is a good safety measure.)
- nexuist 7y ago
  Herein lies my problem. If I want to start a Node project I run `npm init` and then `npm install --save` to my heart's content. If I somehow manage to mess it up I just delete node_modules/ and install again. If I want to start a Python project I have to set up venv and remember to put relative paths in front of every command or else it'll run the system version. Sounds simple, but it's still something to always remember.
  - rootusrootus 7y ago
    Just source the activate script and it'll prepend the correct path so that you don't need to do anything else.
  - odonnellryan 7y ago
    You have to use something like anaconda. It's basically the same as npm_modules (sorta).
cfors 7y ago
What you are describing is a virtualenv and a requirements.txt file. There are tons of other options for Python that do the same thing.
gedy 7y ago
I agree but is it intentional? Seems like a lot of folks rail about how npm works (not just because of the "micro package" philosophy)
fredthomsen 7y ago
I couldn't agree more with this. I was forced into doing some UI coding and although I could never full embrace js, the package management aspects (esp having the sane default of not installing packages globally) were definitely superior to python.
_hardwaregeek 7y ago
Exactly. NPM gets a lot of hate but lockfiles and local install by default is great. The default mode should not be global installation. Also imo virtual environments aren't amazing. Having some mode you have to remember to flip on that changes essentially the semantics of your pip commands seems a little brittle. Tools that work on top of pip and venv like Pipenv or Poetry seem a lot better.
- yawaramin 7y ago
  The default should actually be a shared cache that is symlinked to local projects, like pnpm does.
vonseel 7y ago
Meanwhile, all the python developers who have started doing any Js work are saying “I wish npm worked as easy as pip/virtualenv”... It really isn’t that difficult, it’s just different. Different always seems wrong, at first.
- odonnellryan 7y ago
  I haven't seriously JavaScript'd in a couple of years but my problem with it then was different versions of node or npm. Nice thing about python virtual environments is that problem never exists (can make environments with whatever version you want).
  - vonseel 7y ago
    pipenv really solves the Python version problem, IIRC. I don't actually use pipenv, myself, since I haven't had time to thoroughly figure out the new magic and I prefer to know exactly what's going on with my Python's. npm doesn't solve the duplicating of deps any better than python/pip, as far as I know. The react-native app I'm currently working on has a node_modules folder at 960Mb. That's probably bigger than nearly every virtualenv I've ever seen. A react-native node_modules on a bare project with `--template typescript` is at least 350Mb (created one a few minutes ago). I'm using nvm for node version management. No problems so far.
j88439h84 7y ago
Just use Poetry. https://poetry.eustace.io https://poetry.eustace.io
- michaelmrose 7y ago
  Man that looks fantastic. It looks so useful that it ought to be an official part of Python.
rleigh 7y ago
This isn't even the start of the problems with pip and pypi. If I install pylibtiff, it embeds a copy of an old libtiff which is binary incompatible with the libraries on my system. I have to build it from source, then it works just fine. But I can't inflict this level of brokenness on my end users. This applies to many packages embedding C libraries, including numpy, hdf5 and all the rest. There has been minimal thought put into binary compatibility here, meaning that it's a lottery if it works today, or will break in the future.

by umvi 7y ago

jbay808 7y ago

Use Conda envs!

tachyonbeam 7y ago
I feel uncomfortable with the fact that people feel a third-party solution is the best way to solve this mess. It can also get messy when packages installed with pip, pip3, conda and apt are all entangled with one another in various ways.
- adamson 7y ago
  It’s unfortunate that it’s third party, but conda has the unquestionable advantage of being the only Python-centric packaging system that has a reasonable shared binary library story
  - geofft 7y ago
    I'm curious, do you not find wheels + manylinux reasonable? I agree that until recently, Conda definitely had that advantage, but now that you can `pip install scipy` and have that get you a working library and not try to compile things on your machine what does Conda offer beyond that? I guess one thing Conda has that the pip ecosystem doesn't is that it supports installing non-Python shared libraries like libcurl on their own. Is that an advantage? (We absolutely could replicate that in the pip ecosystem if that was worth doing, and it's even not totally unprecedented to have non-Python binaries on PyPI.)
    - jbay808 7y ago
      Does pip allow version number dependencies? Conda is able to upgrade/downgrade packages to resolve conflicts, whereas pip just seems to check if a package exists and shrugs when there's a version conflict.
      - geofft 7y ago
        pip does handle versioned dependencies and ranges, and know enough to upgrade existing packages when needed to resolve an upgrade. Its resolver isn't currently as complete as Conda's - see https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 . (On the other hand, the fact that Conda uses a real constraint solver has apparently been causing trouble at my day job when it gets stuck exploring some area of the solution space and doesn't install your packages.... so for both pip and conda you're probably better off not relying too hard on dependency resolution and specifying the versions of as many things as you can.)
    - tachyonbeam 7y ago
      I think it would definitely be great if pip could install non-python dependencies. One problem right now is that many projects will tell you to just pip install xyz. You execute that, things start building, and the process fails partway with some cryptic message because you're missing an external dependency. You figure out which one, you install it, start again, and another dependency is missing. Rinse and repeat. It's definitely not a turnkey solution, and this issue trips up newcomers all the time. With respect to versioning, I think pip should be way more strict. It should force you to freeze dependency versions before uploading to pipy, not accept "libxyz > 3.5", but require a fixed range or single version. That would make packages much less likely to break later because newer versions of their dependencies don't work the same way anymore.
- krapht 7y ago
  ... the same thing happens if you mix stuff in your usr/bin directory that isn't managed by your system package manager. The solution is: don't mix your package environments. Use a conda environment. Just like in Linux, you'd use a container. If you wait for the Python steering committee to fix pip you'll be waiting a long time.
sidlls 7y ago
Conda doesn't really solve the packaging problems of python. It can make them worse if conda doesn't have a package you need.
- dagw 7y ago
  Conda supports pip installing into a conda environment if there is no conda package.
  - sidlls 7y ago
    Yes, exactly my point: then you have to potentially deal with conflicting dependencies between pip and conda packages. This happens and it's a pain to deal with.
KaiserPro 7y ago
please no. As someone who has to look after a bunch of servers used by researchers, conda is the biggest cause of things failing for everyone. conda installs stuff all over the place and its very easy to install conflicting stuff in the same place. I just wish that either venv was automatic, or at least the second thing you learn in python.
- jbay808 7y ago
  Are you sure you're talking about Conda's virtual envs? Everything gets installed into the folder /envs/my-env-name.

dangom 7y ago

Is there any particular bottleneck towards getting this to work with Python? Would not a simple pip wrapper be enough for the task?

deleted 7y ago

[deleted]

gilad 7y ago

Packaging is definitely recognized as an issue; e.g. here: https://cecinestpasun.com/entries/where-do-you-see-python-in-10-years/ https://cecinestpasun.com/entries/where-do-you-see-python-in...

kingbirdy 7y ago

That's what virtualenvs are for. Admittedly they're not a perfect solution but the tooling is quite good nowadays.

sametmax 7y ago

Use virtualenv. Always. Python 3 comes with it as "python -m venv". Once in the virtualenv, you don't have to worry about the various pip forms and effects, you can just pio install. You can get fancier than that of course, but that's what works with stock python, on all OS.

geofft 7y ago

Here's a simple but less-than-completely-documented way to keep Python package management under control: 1. Don't install anything globally. Don't pip install --user, definitely don't sudo pip install. 2. For each project you want to work on, create a venv. Yes, there are tools for this, but the base venv tool is totally fine. (venv should be included in your Python, but a few distributors like Debian put it in a separate package - install it from them if needed.) Use python3 -m venv ~/some/directory to create a venv. From here on out 3. As a first step, upgrade pip: ~/some/directory/bin/pip install -U pip. 4. Install things with ~/some/directory/bin/pip install. 5. Run Python with ~/some/directory/bin/python. Slightly advanced move: make a requirements.txt file (you can use .../bin/pip freeze as a starting point) and use .../pip install -r requirements.txt. That way, if you get any sort of package resolution error, you can just delete your venv and make a new one. (Downloads are cached, so this isn't super annoying to do.) A "project" can either be actual Python development, or just a place to install some Python programs and run them out of the resulting bin/. (Edit: Yes, I know about activate, see the replies below for why I don't recommend it. With these rules, you get to say "Never ever type pip, only .../bin/pip", which is a good safety measure.)

nexuist 7y ago
Herein lies my problem. If I want to start a Node project I run `npm init` and then `npm install --save` to my heart's content. If I somehow manage to mess it up I just delete node_modules/ and install again. If I want to start a Python project I have to set up venv and remember to put relative paths in front of every command or else it'll run the system version. Sounds simple, but it's still something to always remember.
- rootusrootus 7y ago
  Just source the activate script and it'll prepend the correct path so that you don't need to do anything else.
- odonnellryan 7y ago
  You have to use something like anaconda. It's basically the same as npm_modules (sorta).

cfors 7y ago

What you are describing is a virtualenv and a requirements.txt file. There are tons of other options for Python that do the same thing.

gedy 7y ago

I agree but is it intentional? Seems like a lot of folks rail about how npm works (not just because of the "micro package" philosophy)

fredthomsen 7y ago

I couldn't agree more with this. I was forced into doing some UI coding and although I could never full embrace js, the package management aspects (esp having the sane default of not installing packages globally) were definitely superior to python.

_hardwaregeek 7y ago

Exactly. NPM gets a lot of hate but lockfiles and local install by default is great. The default mode should not be global installation. Also imo virtual environments aren't amazing. Having some mode you have to remember to flip on that changes essentially the semantics of your pip commands seems a little brittle. Tools that work on top of pip and venv like Pipenv or Poetry seem a lot better.

yawaramin 7y ago
The default should actually be a shared cache that is symlinked to local projects, like pnpm does.

vonseel 7y ago

Meanwhile, all the python developers who have started doing any Js work are saying “I wish npm worked as easy as pip/virtualenv”... It really isn’t that difficult, it’s just different. Different always seems wrong, at first.

odonnellryan 7y ago
I haven't seriously JavaScript'd in a couple of years but my problem with it then was different versions of node or npm. Nice thing about python virtual environments is that problem never exists (can make environments with whatever version you want).
- vonseel 7y ago
  pipenv really solves the Python version problem, IIRC. I don't actually use pipenv, myself, since I haven't had time to thoroughly figure out the new magic and I prefer to know exactly what's going on with my Python's. npm doesn't solve the duplicating of deps any better than python/pip, as far as I know. The react-native app I'm currently working on has a node_modules folder at 960Mb. That's probably bigger than nearly every virtualenv I've ever seen. A react-native node_modules on a bare project with `--template typescript` is at least 350Mb (created one a few minutes ago). I'm using nvm for node version management. No problems so far.

j88439h84 7y ago

Just use Poetry. https://poetry.eustace.io https://poetry.eustace.io

michaelmrose 7y ago
Man that looks fantastic. It looks so useful that it ought to be an official part of Python.

rleigh 7y ago

This isn't even the start of the problems with pip and pypi. If I install pylibtiff, it embeds a copy of an old libtiff which is binary incompatible with the libraries on my system. I have to build it from source, then it works just fine. But I can't inflict this level of brokenness on my end users. This applies to many packages embedding C libraries, including numpy, hdf5 and all the rest. There has been minimal thought put into binary compatibility here, meaning that it's a lottery if it works today, or will break in the future.