10 ms·

Use `Python -m Pip`

131 points by behnamoh 4y ago

valbaca 4y ago
Where this conversation is inevitably going to go: https://xkcd.com/1987/ https://xkcd.com/1987/
- sidpatil 4y ago
  Contrast with https://xkcd.com/353/ https://xkcd.com/353/.
  - josefx 4y ago
    They broke that with python 3. print "Hello World!" no longer works and import antigravity now only links to the comic.
    - Liquid_Fire 4y ago
      > import antigravity now only links to the comic "now"? Hasn't it always?
- deleted 4y ago
  [deleted]
- ActorNightly 4y ago
  Note how that is the environment under Mac OS This issue does not exist under Linux.
whichquestion 4y ago
Pyenv[1] solves the multiple versions of python problem in my experience. You can install the version of python you want and then can set that version globally, per directory, and then use that versions pip or take it further and use a virtualenv/poetry shell. 1: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv
- sph 4y ago
  Not to be confused with pipenv!
- romeoblade 4y ago
  This! I use a combination pyenv, direnv and poetry for my projects.
- aynawn 4y ago
  asdf works as a universal tool for any cli app i.e. python, terraform, ruby, golang, etc https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf asdf plugin add python asdf install python 3.9.1 asdf global python 3.9.1 asdf local python 3.9.1
revskill 4y ago
I have many friends, who's not working as a programmer, but treat programming as a side job only. And what i observe the common theme from them is, they don't mind software engineering practice. What made me confusing and i think, Python actually did a good job at bringing programming to mass audience. What's not going good, is, in Python, software engineering practice is not something to be a huge concern, so end-user hardly learn the same way about how to do thing "correctly". That made me sad though.
Thrymr 4y ago
(2019) Not that anything has fundamentally changed, there are just more other ways to install and manage python packages.
woodruffw 4y ago
This is great advice. To tack on: Python has had built-in virtualenv support since 3.3[1], meaning that you can do this: python -m venv env/ ...on any version of Python released in the last decade and get a reasonable virtual environment. To go one step further, you can also have `venv` automatically bring `pip` and `setuptools` to their latest versions: python -m venv --upgrade-deps env/ ...which you should almost always do, since the versions bundled with your Python distribution are likely to be behind the latest.
- jonatron 4y ago
  If you're on Debian/Ubuntu, you'd first need to install python3-virtualenv. It does say that though IIRC, if you try without it installed.
  - deleted 4y ago
    [deleted]
  - woodruffw 4y ago
    Almost! `python3-virtualenv` is virtualenv, the third-party tool that ultimately inspired `venv`. `python3-venv` is the Ubuntu/Debian breakout package for `python -m venv`. TL;DR: Install python3-venv, not python3-virtualenv. (Ubuntu and Debian's decision to break out core parts of the Python standard library has been an unending source of problems. PEP 668[1] is an in-progress effort to better categorize these kinds of distro changes and improve the user experience around them.) Edit: I should also qualify: virtualenv is still maintained and useful; venv is contains the subset of virtualenv's functionality that 99% of users use. [1]: https://peps.python.org/pep-0668/ https://peps.python.org/pep-0668/
- CobrastanJorji 4y ago
  Yeah, I strongly advise ALWAYS using a virtualenv for any Python project. It's pretty much always a good idea.
- srvmshr 4y ago
  As someone who uses environment & containers fairly regularly, to this day, I didn't know `virtualenv` & `venv` were actually two different packages. I always used these interchangeably. I saw the difference pointed out in the article.
gammalost 4y ago
Python is a fun language but the ecosystem around it is horrible. It's a shame. I just want to pip install like I would a package manager
- newuser4321 4y ago
  I think you mean the package management rather than the ecosystem. For all its failings, the python ecosystem (for machine learning / data science in particular) is unmatched, which is why python is so popular. As a "professional" python user (who got there in a roundabout way), I'd say the biggest problem with package management is all the conflicting advice and different ways to accomplish the same thing (there are other problems, but they are surmountable). Once you get a workflow down, it ends up being pretty easy to set up and manage an environment for a given project. Not that it's perfect, but I think a lot of the stereotypes about how bad it is come from how bad it appears as you try to converge on a setup that works for you.
  - bloblaw 4y ago
    I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TIMTOWTDI). As a tongue-in-cheek reaction, Python espoused "TOOWTDI", or There's Only One Way To Do It :) reference: https://wiki.python.org/moin/TOOWTDI https://wiki.python.org/moin/TOOWTDI The problem is, when it comes to the Python package management ecosystem, there are SO MANY ways to do it. And they aren't equal and require a deep understanding of each tool to select the correct choice. It is a problem for Python, although I often regrettably see it dismissed as not a true issue because $latest_package_management_system fixes it. Python has allowed me to be profoundly productive in many ways, but this is a huge sore spot for the Python ecosystem.
    - xdfgh1112 4y ago
      I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?
cmcconomy 4y ago
I wish there was a way to lock the global python so you couldn't install packages to it by accident
- dragonwriter 4y ago
  On unixy systems, won't the system Python usually require root/sudo to install packages, whereas the ones in your environments will be owned by a less-privileged user? And on Windows, you may have a globally installed Python (though there is little reason to have one instead of the Py launcher), but even if you do it's not a system Python that system components are relying on.
  - remram 4y ago
    Unfortunately the pip developers had the genius idea to add a "user install" mode which pip will default to in this situation. So if you're not root, you won't be able to mess up your system-global Python, but you can still mess up your user-global Python (deleting some ~/.local subfolders takes care of this).
    - mixmastamyk 4y ago
      Useful feature that should have existed from the beginning. If it did, the sudo mistake wouldn’t have happened.
- __mharrison__ 4y ago
  You can do this in UNIX systems by dropping a line like this in your ~/.bashrc . /home/matt/envs/menv/bin/activate
- kstrauser 4y ago
  I have: $ cat ~/.pip/pip.conf [global] require-virtualenv = true ...which makes pip refuse to install anything unless I'm in an activated virtualenv. That, plus running as a regular user that doesn't have write permission to /usr, goes a long way.
jwilk 4y ago
Beware that "python -m" is insecure in untrusted cwd: https://bugs.python.org/issue33053 https://bugs.python.org/issue33053 E.g.: $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -m pip --version _______ < pwned > ------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
- cgriswald 4y ago
  You can use -I to have the interpreter not include the directory in sys.path. (This also eschews site-packages and any environment variables.) $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -Im pip --version pip 22.0.2 ...
- ericholscher 4y ago
  Isn’t that true of command execution as well? I guess you could use a full path to the pip executable, but similar issue.
  - jjnoakes 4y ago
    Do you mean there might be a 'pip' command in the local directory that might get called by accident, or do you mean the real python pip command might load a python module from the local directory by accident?
    - romanows 4y ago
      The second thing. Python includes the current working directory first in the module search path. It leads to the above issue and occasionally some a tricky debugging problem.
      - jwilk 4y ago
        If you run a script, such as "pip", sys.path[0] is the script's directory, not cwd: https://docs.python.org/3/using/cmdline.html https://docs.python.org/3/using/cmdline.html > If the script name refers directly to a Python file, the directory containing that file is added to the start of sys.path
bxparks 4y ago
What is the best way for an author of a Python package to develop and test on multiple versions of Python (e.g. 3.6, 3.7, 3.8, 3.9, 3.10), and be able to switch easily among the various Python versions? Every time I try to research this, I get lost in the chaos of Python packaging and environments. Currently I do a `pip3 install -e .`, which uses the default Python provided by the OS. Then I hope that my continuous integration matrix catches any problems in the other Python versions.
- StopHammoTime 4y ago
  asdf is good, and supports more than Python https://asdf-vm.com/ https://asdf-vm.com/
- maest 4y ago
  I think tox was designed to solve that problem. https://pypi.org/project/tox/ https://pypi.org/project/tox/
  - Izkata 4y ago
    This, absolutely. If you're running tests for building a package, here's an entire "tox.ini" file for running all your tests on multiple python versions: [tox] envlist = py27,py36,py38 [testenv] deps = pytest # Install extra packages for testing separate from what's in setup.py or whatever commands = pytest # Whatever test command your project uses That's really it for bare minimum. If you have pytest configuration, they can also go into tox.ini under a [pytest] section to keep them in the same place. Instead of installing tox globally you can even install it to a separate virtualenv, activate it, then run "tox". It figures out the right thing to do. If you're not building a package: [tox] envlist = py27,py36,py38 skipsdist = True # Don't attempt to build a package / pull from setup.py [testenv] deps = -r requirements.txt # Pull in project requirements pytest commands = pytest
- nicoburns 4y ago
  Not an expert in python, but there is pyenv. I personally use asdf, which is great because it manages versions for multiple language runtimes.
ssalka 4y ago
Python as a language is a joy to use (for small projects), however pip has soured my experience of Python so drastically that I actively avoid taking up Python projects out of knowledge - not fear, knowledge - that the setup/install process is going to be a humongous pain. In most cases I can get started more easily with Node.js/TypeScript. in b4 "use some other package manager / pipenv / virtualenv etc" - no. How about pip actually installs things as expected rather than making me jump through 1000 hoops. /rant
- ssalka 4y ago
  - ok_dad 4y ago
    Probably not down-voted for that, many Python devs agree that pip is basically garbage, down-voted because there are very simple solutions to Python's distribution problems that you refuse to acknowledge. Depending on your use case, there are various tools, but one easy set of tools that is good for anything from large production deployments down to an ML notebook environment is pyenv+poetry. Those two will, with very few commands, allow you to specify and install your environment for a project very easily and with similar ease as you might find when using nvm (similar tool to pyenv) and npm (similar tool to poetry) for a JS/TS/node/whatever project. You got down-voted for being willfully ignorant and proud of it, basically.
- ausbah 4y ago
  you can't complain if there are viable easy to use solutions
- bornfreddy 4y ago
  The issue is that there are many different ways to setup your environment. Every python dev has their own set of tools that mostly work, but sometimes break in some corner cases... and then the problems start, because searching for a solution will inevitably produce something that is almost, but not completely, applicable to your situation. I use pyenv+pipenv (@linux) and are mostly happy with it.
figbert 4y ago
If I'm understanding correctly, this basically just kicks the ball a little further down the road... You shouldn't use pip directly because you don't know which version is the one in your path. Ok: the same applies to the python command? Calling pip is version ambiguous, but so is calling python.
- remram 4y ago
  Yeah if your Linux distro comes with python, python2, python3.7, and python3.8... then you almost certainly have the matching pip, pip2, pip3.7, and pip3.8. If you activate a virtualenv, that will override Python and python3 but also pip and pip3. The only situations where I've encountered breakage is pydoc (because your virtualenv does not necessarily have its own pydoc, contrary to having pip) and calling pip from a Jupyter notebook: the current kernel's virtualenv is not necessarily activated (the solution is the %pip magic or `!{sys.executable} -m pip` since `!python` would have the same issue).
- pseudalopex 4y ago
  They used version numbers in the examples except upgrading pip on Windows. They even used the full path in the 1st example.
- bcbrown 4y ago
  It may be an ambiguous version, but it'll be the same version as the repl you get when you type `python`, and it'll be the same version that'll run your script when you type `python script.py`.
- dmurray 4y ago
  If there's an executable file named "python" in your current directory, typing "python" in your shell won't in general execute that file. You need to add the current directory to your PATH, or to run it explicitly with something like "./python". This is different from the behavior with "python -m modulename". So this security concern applies when you trust your shell and all the directories in your PATH, but you don't trust the contents of the current directory. That's not the norm, but it's quite a common situation to be in - you downloaded some files but don't intend to execute them. This is (used to be at least) different on Windows: typing "python" risks executing a file in the current directory called "python.exe", though maybe UAC saves you now.
shadycuz 4y ago
I haven't used pip directly in years. Lots have changed in the Python eco system and I feel like using pip (directly) for dependency management has it's writing on the wall. https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
kbumsik 4y ago
AFAIR I needed to do that when upgrading pip itself on Windows. `pip install --upgrade pip` may trigger error on Windows because the binary cannot be updated itself. `python -m pip install --upgrade pip` works instead.
- martopix 4y ago
  The article says that.
butwhywhyoh 4y ago
I mean, this is generally good advice, but this reads like an infomercial where they show someone struggling REALLY hard to boil water to make a pot of spaghetti, when we all know it's really not that hard. They try to make finding your pip executable sound difficult, and even more difficult would be to understand what interpreter its tied to. Except... > pip -V > # pip 19.0.3 from /usr/local/lib/python2.7/site-packages/pip (python 2.7) I'm all for evangelizing best practices, but let's at least be honest about it.
- psyklic 4y ago
  I think the point is that most people just immediately type `pip install ...`, without thinking about which interpreter it's tied to. It's fairly common for beginners to tell me they installed something using pip, then the Python interpreter can't find the package.
- bornfreddy 4y ago
  This. Also, the argument is a bit weird: - always use python -m pip - if in venv, it doesn't do any good, but do it anyway for consistency - you should also use venv So... Maybe just always use virtual env and then you can use just pip?
  - Hackbraten 4y ago
    While it’s good advice to use a venv for development work, there are a couple of Python packages that you want to have installed globally, such as pip, pipenv or poetry. So I don’t see a contradiction here after all.
CaliforniaKarl 4y ago
It has only recently sunk in for me that, with the latest versions, Pip is not a tool for maintaining an environment and list of dependencies (where you'd have a lockfile, for example). Instead, it is an interface to… 1. Download packages from PyPi (or a different repository that provides the same interface) 2. Read the pyproject.toml file to find the build backend to use, and then install that. 3. Call the build backend to actually do the installation. This can be Poetry, Setuptools, Flit, or something else. Pipenv is another such interface. PEP 517 (https://peps.python.org/pep-0517/ https://peps.python.org/pep-0517/) created pyproject.toml and defined the API that build backends follow. That API includes a way for the build backend to tell Pip (or Pipenv, etc.) what dependencies to install. For systems that have a lockfile (like Pipenv), that could be a list of packages with explicit versions. PIP 660 extended PEP 517, defining a standard way to have "editable installs". In other words, a way to support `pip install -e package`, or even `pip install -e .`. The above is all my understanding, which is not at all authoritative! It's worth checking out the list of packaging-related PEPs: https://peps.python.org/topic/packaging/ https://peps.python.org/topic/packaging/ It's worth reading PEPs 518 and 621. As an example, here's a Python package that uses Poetry: https://github.com/globus/globus-timer-cli https://github.com/globus/globus-timer-cli It works perfectly fine with Pip, because Pip sees (via `pyproject.toml`) that it needs to pull in Poetry, and call it to do the actual install. Here's an older repo of mine, where I've just started to learn about the transition: https://github.com/stanford-rc/mais-apis-python/blob/main/pyproject.toml https://github.com/stanford-rc/mais-apis-python/blob/main/py... In this case, I could delete `pyproject.toml` entirely, Pip would see the `setup.py` file, and understand to use Setuptools. This was me when I was just starting to learn about the transition. Finally, here's a newer repo of mine, where I've ditched setup.py (and _almost_ ditched setup.cfg) entirely: https://github.com/stanford-rc/globus-group-manager https://github.com/stanford-rc/globus-group-manager I'm still using Setuptools, but all of the metadata and requirements are included in the `pyproject.toml` file. It's definitely been a rocky transition, but it's really looking (to me, at least) like we're at (or near) the point where I can just use `pip install …` and it'll work, regardless of the build backend in use!
dtrodrigues 4y ago
There's currently discussion within pip to support calling pip directly: https://github.com/pypa/pip/issues/11243 https://github.com/pypa/pip/issues/11243
rthomas6 4y ago
alias pip="python -m pip"
- obert 4y ago
  …and doskey pip=python -m pip $*
ZeroCool2u 4y ago
I'm currently fighting with pip and and some particularly brutal dependicies related to Kerberos (Windows build tools are exhausting and such a pain in the ass to install) on two different projects. Number one on my Python wishlist is a significant overhaul to pip to make it more like Cargo.
TeresaFord 4y ago
4dregress 4y ago
I don’t understand why people seem to have so many problems with installing python packages? It could not be more simple.
- jamesjamesm 4y ago
  It's simple until it isn't
martopix 4y ago
My two cents: if you have to use `python -m pip` because it might be the wrong pip or something like that, if you have multiple versions of python and you don't know which one you're using, it means you have a mess in your system, and you're probably going to cry sooner or later regardless of whether you use `python -m`. If there is confusion, `which pip` can help. As for upgrading pip in Windows, that's a silly argument. When I want to upgrade pip, I will use `python -m`, no need to use it all the time. Yes, it's a pain to keep your Pythons in order. We can use pyenv and the like. I wish there was a better way and it's unfortunate, but at the moment it's what it is, and having to use `python -m` is only a sign that you don't have control over your own system.
JadeNB 4y ago
I know we editorialize titles, but the actual title is "why you shouldn't use `python -m pip`". Setting aside the strange capitalization, this still seems like a big change.

Use `Python -m Pip`

131 points by behnamoh 4y ago

valbaca 4y ago

Where this conversation is inevitably going to go: https://xkcd.com/1987/ https://xkcd.com/1987/

sidpatil 4y ago
Contrast with https://xkcd.com/353/ https://xkcd.com/353/.
- josefx 4y ago
  They broke that with python 3. print "Hello World!" no longer works and import antigravity now only links to the comic.
  - Liquid_Fire 4y ago
    > import antigravity now only links to the comic "now"? Hasn't it always?
deleted 4y ago
[deleted]
ActorNightly 4y ago
Note how that is the environment under Mac OS This issue does not exist under Linux.

whichquestion 4y ago

Pyenv[1] solves the multiple versions of python problem in my experience. You can install the version of python you want and then can set that version globally, per directory, and then use that versions pip or take it further and use a virtualenv/poetry shell. 1: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv

sph 4y ago
Not to be confused with pipenv!
romeoblade 4y ago
This! I use a combination pyenv, direnv and poetry for my projects.
aynawn 4y ago
asdf works as a universal tool for any cli app i.e. python, terraform, ruby, golang, etc https://github.com/asdf-vm/asdf https://github.com/asdf-vm/asdf asdf plugin add python asdf install python 3.9.1 asdf global python 3.9.1 asdf local python 3.9.1

revskill 4y ago

I have many friends, who's not working as a programmer, but treat programming as a side job only. And what i observe the common theme from them is, they don't mind software engineering practice. What made me confusing and i think, Python actually did a good job at bringing programming to mass audience. What's not going good, is, in Python, software engineering practice is not something to be a huge concern, so end-user hardly learn the same way about how to do thing "correctly". That made me sad though.

Thrymr 4y ago

(2019) Not that anything has fundamentally changed, there are just more other ways to install and manage python packages.

woodruffw 4y ago

This is great advice. To tack on: Python has had built-in virtualenv support since 3.3[1], meaning that you can do this: python -m venv env/ ...on any version of Python released in the last decade and get a reasonable virtual environment. To go one step further, you can also have `venv` automatically bring `pip` and `setuptools` to their latest versions: python -m venv --upgrade-deps env/ ...which you should almost always do, since the versions bundled with your Python distribution are likely to be behind the latest.

jonatron 4y ago
If you're on Debian/Ubuntu, you'd first need to install python3-virtualenv. It does say that though IIRC, if you try without it installed.
- deleted 4y ago
  [deleted]
- woodruffw 4y ago
  Almost! `python3-virtualenv` is virtualenv, the third-party tool that ultimately inspired `venv`. `python3-venv` is the Ubuntu/Debian breakout package for `python -m venv`. TL;DR: Install python3-venv, not python3-virtualenv. (Ubuntu and Debian's decision to break out core parts of the Python standard library has been an unending source of problems. PEP 668[1] is an in-progress effort to better categorize these kinds of distro changes and improve the user experience around them.) Edit: I should also qualify: virtualenv is still maintained and useful; venv is contains the subset of virtualenv's functionality that 99% of users use. [1]: https://peps.python.org/pep-0668/ https://peps.python.org/pep-0668/
CobrastanJorji 4y ago
Yeah, I strongly advise ALWAYS using a virtualenv for any Python project. It's pretty much always a good idea.
srvmshr 4y ago
As someone who uses environment & containers fairly regularly, to this day, I didn't know `virtualenv` & `venv` were actually two different packages. I always used these interchangeably. I saw the difference pointed out in the article.

gammalost 4y ago

Python is a fun language but the ecosystem around it is horrible. It's a shame. I just want to pip install like I would a package manager

newuser4321 4y ago
I think you mean the package management rather than the ecosystem. For all its failings, the python ecosystem (for machine learning / data science in particular) is unmatched, which is why python is so popular. As a "professional" python user (who got there in a roundabout way), I'd say the biggest problem with package management is all the conflicting advice and different ways to accomplish the same thing (there are other problems, but they are surmountable). Once you get a workflow down, it ends up being pretty easy to set up and manage an environment for a given project. Not that it's perfect, but I think a lot of the stereotypes about how bad it is come from how bad it appears as you try to converge on a setup that works for you.
- bloblaw 4y ago
  I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TIMTOWTDI). As a tongue-in-cheek reaction, Python espoused "TOOWTDI", or There's Only One Way To Do It :) reference: https://wiki.python.org/moin/TOOWTDI https://wiki.python.org/moin/TOOWTDI The problem is, when it comes to the Python package management ecosystem, there are SO MANY ways to do it. And they aren't equal and require a deep understanding of each tool to select the correct choice. It is a problem for Python, although I often regrettably see it dismissed as not a true issue because $latest_package_management_system fixes it. Python has allowed me to be profoundly productive in many ways, but this is a huge sore spot for the Python ecosystem.
  - xdfgh1112 4y ago
    I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?

cmcconomy 4y ago

I wish there was a way to lock the global python so you couldn't install packages to it by accident

dragonwriter 4y ago
On unixy systems, won't the system Python usually require root/sudo to install packages, whereas the ones in your environments will be owned by a less-privileged user? And on Windows, you may have a globally installed Python (though there is little reason to have one instead of the Py launcher), but even if you do it's not a system Python that system components are relying on.
- remram 4y ago
  Unfortunately the pip developers had the genius idea to add a "user install" mode which pip will default to in this situation. So if you're not root, you won't be able to mess up your system-global Python, but you can still mess up your user-global Python (deleting some ~/.local subfolders takes care of this).
  - mixmastamyk 4y ago
    Useful feature that should have existed from the beginning. If it did, the sudo mistake wouldn’t have happened.
__mharrison__ 4y ago
You can do this in UNIX systems by dropping a line like this in your ~/.bashrc . /home/matt/envs/menv/bin/activate
kstrauser 4y ago
I have: $ cat ~/.pip/pip.conf [global] require-virtualenv = true ...which makes pip refuse to install anything unless I'm in an activated virtualenv. That, plus running as a regular user that doesn't have write permission to /usr, goes a long way.

jwilk 4y ago

Beware that "python -m" is insecure in untrusted cwd: https://bugs.python.org/issue33053 https://bugs.python.org/issue33053 E.g.: $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -m pip --version _______ < pwned > ------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

cgriswald 4y ago
You can use -I to have the interpreter not include the directory in sys.path. (This also eschews site-packages and any environment variables.) $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -Im pip --version pip 22.0.2 ...
ericholscher 4y ago
Isn’t that true of command execution as well? I guess you could use a full path to the pip executable, but similar issue.
- jjnoakes 4y ago
  Do you mean there might be a 'pip' command in the local directory that might get called by accident, or do you mean the real python pip command might load a python module from the local directory by accident?
  - romanows 4y ago
    The second thing. Python includes the current working directory first in the module search path. It leads to the above issue and occasionally some a tricky debugging problem.
    - jwilk 4y ago
      If you run a script, such as "pip", sys.path[0] is the script's directory, not cwd: https://docs.python.org/3/using/cmdline.html https://docs.python.org/3/using/cmdline.html > If the script name refers directly to a Python file, the directory containing that file is added to the start of sys.path

bxparks 4y ago

What is the best way for an author of a Python package to develop and test on multiple versions of Python (e.g. 3.6, 3.7, 3.8, 3.9, 3.10), and be able to switch easily among the various Python versions? Every time I try to research this, I get lost in the chaos of Python packaging and environments. Currently I do a `pip3 install -e .`, which uses the default Python provided by the OS. Then I hope that my continuous integration matrix catches any problems in the other Python versions.

StopHammoTime 4y ago
asdf is good, and supports more than Python https://asdf-vm.com/ https://asdf-vm.com/
maest 4y ago
I think tox was designed to solve that problem. https://pypi.org/project/tox/ https://pypi.org/project/tox/
- Izkata 4y ago
  This, absolutely. If you're running tests for building a package, here's an entire "tox.ini" file for running all your tests on multiple python versions: [tox] envlist = py27,py36,py38 [testenv] deps = pytest # Install extra packages for testing separate from what's in setup.py or whatever commands = pytest # Whatever test command your project uses That's really it for bare minimum. If you have pytest configuration, they can also go into tox.ini under a [pytest] section to keep them in the same place. Instead of installing tox globally you can even install it to a separate virtualenv, activate it, then run "tox". It figures out the right thing to do. If you're not building a package: [tox] envlist = py27,py36,py38 skipsdist = True # Don't attempt to build a package / pull from setup.py [testenv] deps = -r requirements.txt # Pull in project requirements pytest commands = pytest
nicoburns 4y ago
Not an expert in python, but there is pyenv. I personally use asdf, which is great because it manages versions for multiple language runtimes.

ssalka 4y ago

Python as a language is a joy to use (for small projects), however pip has soured my experience of Python so drastically that I actively avoid taking up Python projects out of knowledge - not fear, knowledge - that the setup/install process is going to be a humongous pain. In most cases I can get started more easily with Node.js/TypeScript. in b4 "use some other package manager / pipenv / virtualenv etc" - no. How about pip actually installs things as expected rather than making me jump through 1000 hoops. /rant

ssalka 4y ago
- ok_dad 4y ago
  Probably not down-voted for that, many Python devs agree that pip is basically garbage, down-voted because there are very simple solutions to Python's distribution problems that you refuse to acknowledge. Depending on your use case, there are various tools, but one easy set of tools that is good for anything from large production deployments down to an ML notebook environment is pyenv+poetry. Those two will, with very few commands, allow you to specify and install your environment for a project very easily and with similar ease as you might find when using nvm (similar tool to pyenv) and npm (similar tool to poetry) for a JS/TS/node/whatever project. You got down-voted for being willfully ignorant and proud of it, basically.
ausbah 4y ago
you can't complain if there are viable easy to use solutions
bornfreddy 4y ago
The issue is that there are many different ways to setup your environment. Every python dev has their own set of tools that mostly work, but sometimes break in some corner cases... and then the problems start, because searching for a solution will inevitably produce something that is almost, but not completely, applicable to your situation. I use pyenv+pipenv (@linux) and are mostly happy with it.

figbert 4y ago

If I'm understanding correctly, this basically just kicks the ball a little further down the road... You shouldn't use pip directly because you don't know which version is the one in your path. Ok: the same applies to the python command? Calling pip is version ambiguous, but so is calling python.

remram 4y ago
Yeah if your Linux distro comes with python, python2, python3.7, and python3.8... then you almost certainly have the matching pip, pip2, pip3.7, and pip3.8. If you activate a virtualenv, that will override Python and python3 but also pip and pip3. The only situations where I've encountered breakage is pydoc (because your virtualenv does not necessarily have its own pydoc, contrary to having pip) and calling pip from a Jupyter notebook: the current kernel's virtualenv is not necessarily activated (the solution is the %pip magic or `!{sys.executable} -m pip` since `!python` would have the same issue).
pseudalopex 4y ago
They used version numbers in the examples except upgrading pip on Windows. They even used the full path in the 1st example.
bcbrown 4y ago
It may be an ambiguous version, but it'll be the same version as the repl you get when you type `python`, and it'll be the same version that'll run your script when you type `python script.py`.
dmurray 4y ago
If there's an executable file named "python" in your current directory, typing "python" in your shell won't in general execute that file. You need to add the current directory to your PATH, or to run it explicitly with something like "./python". This is different from the behavior with "python -m modulename". So this security concern applies when you trust your shell and all the directories in your PATH, but you don't trust the contents of the current directory. That's not the norm, but it's quite a common situation to be in - you downloaded some files but don't intend to execute them. This is (used to be at least) different on Windows: typing "python" risks executing a file in the current directory called "python.exe", though maybe UAC saves you now.

shadycuz 4y ago

I haven't used pip directly in years. Lots have changed in the Python eco system and I feel like using pip (directly) for dependency management has it's writing on the wall. https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...

kbumsik 4y ago

AFAIR I needed to do that when upgrading pip itself on Windows. `pip install --upgrade pip` may trigger error on Windows because the binary cannot be updated itself. `python -m pip install --upgrade pip` works instead.

martopix 4y ago
The article says that.

butwhywhyoh 4y ago

I mean, this is generally good advice, but this reads like an infomercial where they show someone struggling REALLY hard to boil water to make a pot of spaghetti, when we all know it's really not that hard. They try to make finding your pip executable sound difficult, and even more difficult would be to understand what interpreter its tied to. Except... > pip -V > # pip 19.0.3 from /usr/local/lib/python2.7/site-packages/pip (python 2.7) I'm all for evangelizing best practices, but let's at least be honest about it.

psyklic 4y ago
I think the point is that most people just immediately type `pip install ...`, without thinking about which interpreter it's tied to. It's fairly common for beginners to tell me they installed something using pip, then the Python interpreter can't find the package.
bornfreddy 4y ago
This. Also, the argument is a bit weird: - always use python -m pip - if in venv, it doesn't do any good, but do it anyway for consistency - you should also use venv So... Maybe just always use virtual env and then you can use just pip?
- Hackbraten 4y ago
  While it’s good advice to use a venv for development work, there are a couple of Python packages that you want to have installed globally, such as pip, pipenv or poetry. So I don’t see a contradiction here after all.

CaliforniaKarl 4y ago

It has only recently sunk in for me that, with the latest versions, Pip is not a tool for maintaining an environment and list of dependencies (where you'd have a lockfile, for example). Instead, it is an interface to… 1. Download packages from PyPi (or a different repository that provides the same interface) 2. Read the pyproject.toml file to find the build backend to use, and then install that. 3. Call the build backend to actually do the installation. This can be Poetry, Setuptools, Flit, or something else. Pipenv is another such interface. PEP 517 (https://peps.python.org/pep-0517/ https://peps.python.org/pep-0517/) created pyproject.toml and defined the API that build backends follow. That API includes a way for the build backend to tell Pip (or Pipenv, etc.) what dependencies to install. For systems that have a lockfile (like Pipenv), that could be a list of packages with explicit versions. PIP 660 extended PEP 517, defining a standard way to have "editable installs". In other words, a way to support `pip install -e package`, or even `pip install -e .`. The above is all my understanding, which is not at all authoritative! It's worth checking out the list of packaging-related PEPs: https://peps.python.org/topic/packaging/ https://peps.python.org/topic/packaging/ It's worth reading PEPs 518 and 621. As an example, here's a Python package that uses Poetry: https://github.com/globus/globus-timer-cli https://github.com/globus/globus-timer-cli It works perfectly fine with Pip, because Pip sees (via `pyproject.toml`) that it needs to pull in Poetry, and call it to do the actual install. Here's an older repo of mine, where I've just started to learn about the transition: https://github.com/stanford-rc/mais-apis-python/blob/main/pyproject.toml https://github.com/stanford-rc/mais-apis-python/blob/main/py... In this case, I could delete `pyproject.toml` entirely, Pip would see the `setup.py` file, and understand to use Setuptools. This was me when I was just starting to learn about the transition. Finally, here's a newer repo of mine, where I've ditched setup.py (and _almost_ ditched setup.cfg) entirely: https://github.com/stanford-rc/globus-group-manager https://github.com/stanford-rc/globus-group-manager I'm still using Setuptools, but all of the metadata and requirements are included in the `pyproject.toml` file. It's definitely been a rocky transition, but it's really looking (to me, at least) like we're at (or near) the point where I can just use `pip install …` and it'll work, regardless of the build backend in use!

dtrodrigues 4y ago

There's currently discussion within pip to support calling pip directly: https://github.com/pypa/pip/issues/11243 https://github.com/pypa/pip/issues/11243

rthomas6 4y ago

alias pip="python -m pip"

obert 4y ago
…and doskey pip=python -m pip $*

ZeroCool2u 4y ago

I'm currently fighting with pip and and some particularly brutal dependicies related to Kerberos (Windows build tools are exhausting and such a pain in the ass to install) on two different projects. Number one on my Python wishlist is a significant overhaul to pip to make it more like Cargo.

TeresaFord 4y ago

4dregress 4y ago

I don’t understand why people seem to have so many problems with installing python packages? It could not be more simple.

jamesjamesm 4y ago
It's simple until it isn't

martopix 4y ago

My two cents: if you have to use `python -m pip` because it might be the wrong pip or something like that, if you have multiple versions of python and you don't know which one you're using, it means you have a mess in your system, and you're probably going to cry sooner or later regardless of whether you use `python -m`. If there is confusion, `which pip` can help. As for upgrading pip in Windows, that's a silly argument. When I want to upgrade pip, I will use `python -m`, no need to use it all the time. Yes, it's a pain to keep your Pythons in order. We can use pyenv and the like. I wish there was a better way and it's unfortunate, but at the moment it's what it is, and having to use `python -m` is only a sign that you don't have control over your own system.

JadeNB 4y ago

I know we editorialize titles, but the actual title is "why you shouldn't use `python -m pip`". Setting aside the strange capitalization, this still seems like a big change.