13 ms·

> 10. What Changes When You Are Free > You will not be forced to work inside something known as a “virtual environment.” Oof, this terrible advice cancels out

> 10. What Changes When You Are Free > You will not be forced to work inside something known as a “virtual environment.” Oof, this terrible advice cancels out an otherwise reasonable post. Beginners who don't know what they're doing are the last people who should be `pip install -r requirements.txt`-ing into the system Python the way this article is recommending. That's not only going to make working on multiple projects nearly impossible (especially for the kind of beginning students who get recommended Anaconda, which are almost always the Data Science-y crowd using NumPy, Pandas, Scikit, etc, which are notoriously finicky with version conflicts), but it stands a good chance of breaking other Python-based system utilities in completely opaque ways. This sort of advice can fubar a naive user's entire workflow. I know virtualenvs suck to explain to people, but in my opinion it needs to be done before you ever tell them about `pip install`.

dumbo-octopus 2y ago
Yes. I recently had to debug my gf’s entire work computer setup getting fubar’d from installing a pip package at a user level that conflicted with the name of some package used internally by the 10k+/seat/year software the company devices have installed. Proximal cause? The installer “helpfully” failed over to installing at the user level when she didn’t have permissions to write to the place she told it to. Root cause of course is that million dollar fancy fancy enterprise software somehow being unable to give itself an isolated python package namespace, or provide any relevant errors when conflicts occurred. The python packaging system has to be used in case studies of worst design decisions of all time.
- adolph 2y ago
  > package used internally by the 10k+/seat/year software the company devices have installed Sounds like a skill issue that shouldn’t be seen in “10k+/seat/year software.” I wonder what other fails are in it.
  - dumbo-octopus 2y ago
    Well it constantly crashes and requires every shop to develop their own half-broken way of interfacing between it and other industry-standard software, but at least ~half the pixels you’ve seen in any modern movie or tv show have been touched by it at one point or another. So that’s interesting.
    - geysersam 2y ago
      Thats extremely fascinating. What's it used for?
      - deleted 2y ago
        [deleted]
      - dumbo-octopus 2y ago
        By the books Digital Compositing, but Editors also gravitate towards it for generalist work.
- AtlasBarfed 2y ago
  Python has dogshit package management that makes java look well thought out and sane.
  - echelon 2y ago
    PYTHON @&$!! The Python community needs to solve this ASAP. This almost weekly pain point has turned a language I used to love in college into one of my most despised languages. The fact that the ML community uses this broken platform is infuriating. Make a Python version 4 that focuses only on fixing the packaging. 100% per-project hermeticy. No global packages whatsoever. Solve just this issue and bring the entire ecosystem on board. Kill all the various virtualenvs, the anacondas, global packages. All of it. Learn from Rust/Cargo. That project does it mostly right (sans lack of namespaces and reproducible builds).
    - Eisenstein 2y ago
      It took 10 years to force a breaking migration from Python 2 to Python 3 in order to handle Unicode... I have resigned myself to repeatedly smashing my keyboard on the desk until it rains keys in blind rage induced frustration to let off steam and then calmly creating an issue in the appropriate repo instead of allowing even a modicum of hope that this will ever be fixed systemically.
    - mr_toad 2y ago
      Wouldn’t this end up like the XKCD standards cartoon. You’d have yet another Python package system and even more fragmentation. https://xkcd.com/927/ https://xkcd.com/927/
      - ethbr1 2y ago
        If Python users weren't comfortable living in that reality already, they wouldn't still be Python users.
    - superlopuh 2y ago
      As far as I'm concerned, this is the fix: https://github.com/astral-sh/uv https://github.com/astral-sh/uv
- LarsDu88 2y ago
  Sigh https://dublog.net/blog/so-many-python-package-managers/ https://dublog.net/blog/so-many-python-package-managers/
hkdfgdfg 2y ago
I have never broken my Python installation on Windows (official installer) despite recklessly installing anything. If I had broken it, I would have just uninstalled and reinstalled. While not familiar with macOS (topic of this article), I think the Mac installer works the same. On Debian of course you can cause great damage because the system Python is used for critical system functions. This is silly, I think the system python should be an isolated install in /usr/sbin. Better yet, move back to Perl for the system.
- rbanffy 2y ago
  You won’t break a machine by installing Python packages on an OS that never uses Python for anything.
  - analog31 2y ago
    This is a good point and something I hadn't thought about. I'm also a happy Windows user, and have never had a problem with Python installations. But like you say, Python isn't baked into the system. I do use one piece of commercial software that includes Python. You can see the installer saying "installing Python." I suppose it should fall on the OS and software vendors to not abuse the infrastructure... "to whom much is given, much is expected," but maybe too much to expect.
    - rbanffy 2y ago
      I remember that, in the past at least, one could completely destroy a Red Hat install by messing too much with the system Python.
      - analog31 2y ago
        I guess "Python is my operating system" has its unintended consequences. ;-)
- swiftcoder 2y ago
  To be clear, there's absolutely nothing broken about the system Python in the article. There is just a shell alias causing the anaconda version of Python to be launched instead when you type `python3` at the command prompt...
abdullahkhalids 2y ago
I teach using python. You should try explaining virtual environments to students, most of them with no programming experience, or any real notion of the state of a computer system. I do, because we recommend students use them. Every class consists of endless debugging of student systems. After that you might understand the OP's viewpoint.
- freeopinion 2y ago
  After that I wonder why people think Python is a good choice to teach as a first experience in programming. Virtual environments is actually my biggest gripe with Python. I think it is interesting to consider why Python is so popular despite such fiascos. The answers can be very informative but can also be giant red flags.
  - Al-Khwarizmi 2y ago
    Because every option sucks for teaching. C/C++? Full of footguns. Java/C#? Too complicated for beginners. Pascal? Outdated. Etc. I personally prefer C to teach first-year CS students but just as the lesser evil. A good first programming language is sorely lacking. (Note that I'm talking about the imperative programming paradigm. The debate on whether one should start with functional programming is outside the scope of this comment).
    - Spivak 2y ago
      It's weird you throw out Java because they are the teaching language. They're the stick everything else is measured against. CS101 is a sea of Eclipse. Having a programming environment where all the major structures are discoverable through menus is actually pretty useful.
      - dvdkon 2y ago
        Java is popular as a first language, but I think it's just due to its (bygone?) general popularity. I don't think a language that forces you into a one-class-per-file mindset from day one has any business being used for teaching; students should start from the basics ("commands"/function calls, if, while, for, ...) and then discover what more complex concepts do within that framework. Java's tooling is nice, but frankly Turbo Pascal (or something at its level) is enough for beginners.
orlp 2y ago
What I do on every dev machine is build Python from source and install it to ~/.localpython and put it in my path. Then I can pip install away care-free, and quickly run scripts / repls without having to activate an environment.
- eru 2y ago
  Why does it need to be built from source for that?
  - orlp 2y ago
    It doesn't, but it's typically the simplest (for me). It also means I get exactly the version I want compiled with the best optimizations for that machine.
    - eru 2y ago
      Thanks, that makes sense.
  - m0llusk 2y ago
    Typically building from source is done to include optional components such as with the ./configure --with-pydebug option. With many projects make doc is an option because so many prefer to skip that part of the build in favor of online documentation.
- sevensor 2y ago
  Building Python is so quick and easy I’ve built an interpreter per project in some circumstances. Sometimes more if I want to make sure it’s compatible with multiple Python versions.
BugsJustFindMe 2y ago
> I know virtualenvs suck to explain to people, but in my opinion it needs to be done This is because Python doesn't have versioned imports, which means you can't have multiple versions of the same package in the same environment, but I like to dream about a world where this isn't the case. If instead of import foo we had import foo@x.y.z#optional-checksum, the Python world would be massively improved. It seems like it would be such a simple change too.
- zbentley 2y ago
  Virtualenvs solve not having separate workspace-local package installations, not a lack of versioned imports. Versioned imports are not a good solution to separate installations of packages: code is harder to upgrade, cluttered, and encouraged to depend on specifics rather than contracts. There’s a reason every major language localizes their version pinning into a per-project dependencies file (which can be anywhere on the spectrum between “contracts only semver ranges” and “checksummed/vendored lockfile”). And that’s before considering the troublesome behaviors that emerge when you permit importing multiple versions of the same library in the same program, if you want that too.
  - eru 2y ago
    > And that’s before considering the troublesome behaviors that emerge when you permit importing multiple versions of the same library in the same program, if you want that too. Rust has versioned imports. So you can import different versions of the same module (at least in your transitive dependencies).
    - zbentley 2y ago
      Yep. And while that’s not something I’d call a mistake per se, it can be troublesome. What if a crate you depend on at different versions provides access to global state? What if a transitive embeds definitions from that crate into public contracts consumed by code using a different version? Rust offers mitigations or means of detection for those cases, but they still require thought and troubleshooting when they occur. Given Python’s lack of static typing and more-likely-to-be-nonexpert user base and usage patterns, I suspect that troublesomeness would not be a value-add for the Python platform.
nox101 2y ago
IMO what needs to happen is python needs to get rid of system installs and only work via venv. It should check the current folder for some .venv or something and auto-config itself to the current directory (or complain that you need to config with a messages like "python not configured for current folder. Run 'source bin/activate' to configure. Honestly thought that could be so much better. The entire problem of python an venv IMO comes from it not being the default.
- polemic 2y ago
  Python developers agree. See PEP 668 – Marking Python base environments as “externally managed” – https://peps.python.org/pep-0668/ https://peps.python.org/pep-0668/
  - brylie 2y ago
    More recent document https://packaging.python.org/en/latest/specifications/externally-managed-environments/#externally-managed-environments https://packaging.python.org/en/latest/specifications/extern...
    - ajb 2y ago
      Hmm that still recommends that distros allow admins to install to /usr/local, albeit in such a way that it at least can't break the OS. IMO the idea that a 'Linux admin' is better informed than a 'Linux user' is increasingly anachronistic. In most cases the admin is just the user running sudo. I'd suggest that such functionality should be enabled by installing some kind of OS package rather than being the default
  - jampekka 2y ago
    Yet another hack on hack. Venv was the wrong fix/hack and now the package management is cascading into even more madness.
  - lhfacx 2y ago
    This is insane. Apparently (at least on Debian) this can be circumvented by putting this in ~/.config/pip/pip.conf: [global] break-system-packages = true I would have preferred single-version-externally-managed to keep fond memories of setuptools alive. It becomes increasingly impossible to track down home directory pollution and config files in Python. Next step will be a Python registry on Linux. How about: regedit VENV=/home/sjw/venv42 KEYWORD=single-version-externally-managed DWORD=0xbadbee
promer 2y ago
I don't see how it could be correct to say that I'm recommending that students pip install into the system Python. On macOS, there hasn't been a system Python since 12.3 came out in Jan of 2022. And as far as I know, there was never a system Python on Windows.
- swiftcoder 2y ago
  > On macOS, there hasn't been a system Python since 12.3 came out in Jan of 2022 It may not be pre-installed, but if you install Xcode it will also install python3 in /usr/bin/ for you...
  - hwgern 2y ago
    Xcode users should probably be able to fix breakage and that /usr/bin/python is not used by MacOS for critical tasks.
  - promer 2y ago
    The assertion that seemed incorrect was that I was recommending an option that would install libraries to the system Python. Xcode does put a binary with the name python3 into `/usr/bin`. This is not a system Python, but set that aside. Can you explain how someone might pip install libraries to this instance of Python?
    - swiftcoder 2y ago
      Once Xcode's python is installed, `pip3 install ...` will install libraries to that python installation (assuming you haven't aliased pip3 to another installation, as it appears the anaconda installer did)
      - promer 2y ago
        The install by xcode of a python3 binary in `/usr/bin` does not come with its own copy of pip3. So either your command will fail to install or it will use a version of pip that is on PATH but was installed by some other version of python. So in either case you can't install a library into what you are mistakenly calling a system python. So the original assertion was not true and you have not been able to make up an ex post justification for it.
djaouen 2y ago
You are wrong: this was not a reasonable post.

dumbo-octopus 2y ago

Yes. I recently had to debug my gf’s entire work computer setup getting fubar’d from installing a pip package at a user level that conflicted with the name of some package used internally by the 10k+/seat/year software the company devices have installed. Proximal cause? The installer “helpfully” failed over to installing at the user level when she didn’t have permissions to write to the place she told it to. Root cause of course is that million dollar fancy fancy enterprise software somehow being unable to give itself an isolated python package namespace, or provide any relevant errors when conflicts occurred. The python packaging system has to be used in case studies of worst design decisions of all time.

adolph 2y ago
> package used internally by the 10k+/seat/year software the company devices have installed Sounds like a skill issue that shouldn’t be seen in “10k+/seat/year software.” I wonder what other fails are in it.
- dumbo-octopus 2y ago
  Well it constantly crashes and requires every shop to develop their own half-broken way of interfacing between it and other industry-standard software, but at least ~half the pixels you’ve seen in any modern movie or tv show have been touched by it at one point or another. So that’s interesting.
  - geysersam 2y ago
    Thats extremely fascinating. What's it used for?
    - deleted 2y ago
      [deleted]
    - dumbo-octopus 2y ago
      By the books Digital Compositing, but Editors also gravitate towards it for generalist work.
AtlasBarfed 2y ago
Python has dogshit package management that makes java look well thought out and sane.
- echelon 2y ago
  PYTHON @&$!! The Python community needs to solve this ASAP. This almost weekly pain point has turned a language I used to love in college into one of my most despised languages. The fact that the ML community uses this broken platform is infuriating. Make a Python version 4 that focuses only on fixing the packaging. 100% per-project hermeticy. No global packages whatsoever. Solve just this issue and bring the entire ecosystem on board. Kill all the various virtualenvs, the anacondas, global packages. All of it. Learn from Rust/Cargo. That project does it mostly right (sans lack of namespaces and reproducible builds).
  - Eisenstein 2y ago
    It took 10 years to force a breaking migration from Python 2 to Python 3 in order to handle Unicode... I have resigned myself to repeatedly smashing my keyboard on the desk until it rains keys in blind rage induced frustration to let off steam and then calmly creating an issue in the appropriate repo instead of allowing even a modicum of hope that this will ever be fixed systemically.
  - mr_toad 2y ago
    Wouldn’t this end up like the XKCD standards cartoon. You’d have yet another Python package system and even more fragmentation. https://xkcd.com/927/ https://xkcd.com/927/
    - ethbr1 2y ago
      If Python users weren't comfortable living in that reality already, they wouldn't still be Python users.
  - superlopuh 2y ago
    As far as I'm concerned, this is the fix: https://github.com/astral-sh/uv https://github.com/astral-sh/uv
LarsDu88 2y ago
Sigh https://dublog.net/blog/so-many-python-package-managers/ https://dublog.net/blog/so-many-python-package-managers/

hkdfgdfg 2y ago

I have never broken my Python installation on Windows (official installer) despite recklessly installing anything. If I had broken it, I would have just uninstalled and reinstalled. While not familiar with macOS (topic of this article), I think the Mac installer works the same. On Debian of course you can cause great damage because the system Python is used for critical system functions. This is silly, I think the system python should be an isolated install in /usr/sbin. Better yet, move back to Perl for the system.

rbanffy 2y ago
You won’t break a machine by installing Python packages on an OS that never uses Python for anything.
- analog31 2y ago
  This is a good point and something I hadn't thought about. I'm also a happy Windows user, and have never had a problem with Python installations. But like you say, Python isn't baked into the system. I do use one piece of commercial software that includes Python. You can see the installer saying "installing Python." I suppose it should fall on the OS and software vendors to not abuse the infrastructure... "to whom much is given, much is expected," but maybe too much to expect.
  - rbanffy 2y ago
    I remember that, in the past at least, one could completely destroy a Red Hat install by messing too much with the system Python.
    - analog31 2y ago
      I guess "Python is my operating system" has its unintended consequences. ;-)
swiftcoder 2y ago
To be clear, there's absolutely nothing broken about the system Python in the article. There is just a shell alias causing the anaconda version of Python to be launched instead when you type `python3` at the command prompt...

abdullahkhalids 2y ago

I teach using python. You should try explaining virtual environments to students, most of them with no programming experience, or any real notion of the state of a computer system. I do, because we recommend students use them. Every class consists of endless debugging of student systems. After that you might understand the OP's viewpoint.

freeopinion 2y ago
After that I wonder why people think Python is a good choice to teach as a first experience in programming. Virtual environments is actually my biggest gripe with Python. I think it is interesting to consider why Python is so popular despite such fiascos. The answers can be very informative but can also be giant red flags.
- Al-Khwarizmi 2y ago
  Because every option sucks for teaching. C/C++? Full of footguns. Java/C#? Too complicated for beginners. Pascal? Outdated. Etc. I personally prefer C to teach first-year CS students but just as the lesser evil. A good first programming language is sorely lacking. (Note that I'm talking about the imperative programming paradigm. The debate on whether one should start with functional programming is outside the scope of this comment).
  - Spivak 2y ago
    It's weird you throw out Java because they are the teaching language. They're the stick everything else is measured against. CS101 is a sea of Eclipse. Having a programming environment where all the major structures are discoverable through menus is actually pretty useful.
    - dvdkon 2y ago
      Java is popular as a first language, but I think it's just due to its (bygone?) general popularity. I don't think a language that forces you into a one-class-per-file mindset from day one has any business being used for teaching; students should start from the basics ("commands"/function calls, if, while, for, ...) and then discover what more complex concepts do within that framework. Java's tooling is nice, but frankly Turbo Pascal (or something at its level) is enough for beginners.

orlp 2y ago

What I do on every dev machine is build Python from source and install it to ~/.localpython and put it in my path. Then I can pip install away care-free, and quickly run scripts / repls without having to activate an environment.

eru 2y ago
Why does it need to be built from source for that?
- orlp 2y ago
  It doesn't, but it's typically the simplest (for me). It also means I get exactly the version I want compiled with the best optimizations for that machine.
  - eru 2y ago
    Thanks, that makes sense.
- m0llusk 2y ago
  Typically building from source is done to include optional components such as with the ./configure --with-pydebug option. With many projects make doc is an option because so many prefer to skip that part of the build in favor of online documentation.
sevensor 2y ago
Building Python is so quick and easy I’ve built an interpreter per project in some circumstances. Sometimes more if I want to make sure it’s compatible with multiple Python versions.

BugsJustFindMe 2y ago

> I know virtualenvs suck to explain to people, but in my opinion it needs to be done This is because Python doesn't have versioned imports, which means you can't have multiple versions of the same package in the same environment, but I like to dream about a world where this isn't the case. If instead of import foo we had import foo@x.y.z#optional-checksum, the Python world would be massively improved. It seems like it would be such a simple change too.

zbentley 2y ago
Virtualenvs solve not having separate workspace-local package installations, not a lack of versioned imports. Versioned imports are not a good solution to separate installations of packages: code is harder to upgrade, cluttered, and encouraged to depend on specifics rather than contracts. There’s a reason every major language localizes their version pinning into a per-project dependencies file (which can be anywhere on the spectrum between “contracts only semver ranges” and “checksummed/vendored lockfile”). And that’s before considering the troublesome behaviors that emerge when you permit importing multiple versions of the same library in the same program, if you want that too.
- eru 2y ago
  > And that’s before considering the troublesome behaviors that emerge when you permit importing multiple versions of the same library in the same program, if you want that too. Rust has versioned imports. So you can import different versions of the same module (at least in your transitive dependencies).
  - zbentley 2y ago
    Yep. And while that’s not something I’d call a mistake per se, it can be troublesome. What if a crate you depend on at different versions provides access to global state? What if a transitive embeds definitions from that crate into public contracts consumed by code using a different version? Rust offers mitigations or means of detection for those cases, but they still require thought and troubleshooting when they occur. Given Python’s lack of static typing and more-likely-to-be-nonexpert user base and usage patterns, I suspect that troublesomeness would not be a value-add for the Python platform.

nox101 2y ago

IMO what needs to happen is python needs to get rid of system installs and only work via venv. It should check the current folder for some .venv or something and auto-config itself to the current directory (or complain that you need to config with a messages like "python not configured for current folder. Run 'source bin/activate' to configure. Honestly thought that could be so much better. The entire problem of python an venv IMO comes from it not being the default.

polemic 2y ago
Python developers agree. See PEP 668 – Marking Python base environments as “externally managed” – https://peps.python.org/pep-0668/ https://peps.python.org/pep-0668/
- brylie 2y ago
  More recent document https://packaging.python.org/en/latest/specifications/externally-managed-environments/#externally-managed-environments https://packaging.python.org/en/latest/specifications/extern...
  - ajb 2y ago
    Hmm that still recommends that distros allow admins to install to /usr/local, albeit in such a way that it at least can't break the OS. IMO the idea that a 'Linux admin' is better informed than a 'Linux user' is increasingly anachronistic. In most cases the admin is just the user running sudo. I'd suggest that such functionality should be enabled by installing some kind of OS package rather than being the default
- jampekka 2y ago
  Yet another hack on hack. Venv was the wrong fix/hack and now the package management is cascading into even more madness.
- lhfacx 2y ago
  This is insane. Apparently (at least on Debian) this can be circumvented by putting this in ~/.config/pip/pip.conf: [global] break-system-packages = true I would have preferred single-version-externally-managed to keep fond memories of setuptools alive. It becomes increasingly impossible to track down home directory pollution and config files in Python. Next step will be a Python registry on Linux. How about: regedit VENV=/home/sjw/venv42 KEYWORD=single-version-externally-managed DWORD=0xbadbee

promer 2y ago

I don't see how it could be correct to say that I'm recommending that students pip install into the system Python. On macOS, there hasn't been a system Python since 12.3 came out in Jan of 2022. And as far as I know, there was never a system Python on Windows.

swiftcoder 2y ago
> On macOS, there hasn't been a system Python since 12.3 came out in Jan of 2022 It may not be pre-installed, but if you install Xcode it will also install python3 in /usr/bin/ for you...
- hwgern 2y ago
  Xcode users should probably be able to fix breakage and that /usr/bin/python is not used by MacOS for critical tasks.
- promer 2y ago
  The assertion that seemed incorrect was that I was recommending an option that would install libraries to the system Python. Xcode does put a binary with the name python3 into `/usr/bin`. This is not a system Python, but set that aside. Can you explain how someone might pip install libraries to this instance of Python?
  - swiftcoder 2y ago
    Once Xcode's python is installed, `pip3 install ...` will install libraries to that python installation (assuming you haven't aliased pip3 to another installation, as it appears the anaconda installer did)
    - promer 2y ago
      The install by xcode of a python3 binary in `/usr/bin` does not come with its own copy of pip3. So either your command will fail to install or it will use a version of pip that is on PATH but was installed by some other version of python. So in either case you can't install a library into what you are mistakenly calling a system python. So the original assertion was not true and you have not been able to make up an ex post justification for it.

djaouen 2y ago

You are wrong: this was not a reasonable post.