9 ms·

Python Environment

307 points by mgrayson 8y ago

mmcnl 8y ago
I use pipenv and couldn't be happier. It's dead simple and it works. It is a bit slow. The problem presented in this xkcd has been fixed.
gkya 8y ago
I guess what Python needs to do is to crown one solution as the official, and then start imposing on PyPA packages to convert before a deadline or exclude. Then they can move the non-compliant packages to a legacy archive, and build on the one solution that the community chose, say in a poll. Granted I've only ever interacted pip and virtualenv and though it was simple; but looking at this, things don't have to be this complex.
- pwang 8y ago
  The heart of the problem is that different people are doing different things with Python, and nobody wants to pay cognitive load for what they don't want to use. The packagers and package tool makers of the world inherit ALL of the technical debt from ALL of the upstream software devs. They're either the liver or the colon. An upstream dev decided to not document which compiler they use during dev time? That's now your problem. A compiler maker (GNU, MSFT, etc.) decides to change how they distribute the C runtime? Congrats, now it's your packaging system's job to know how to differentiate between Windows 7 and Windows 10 running particular versions of Visual Studio. Nobody wants a "single simple unified solution" for packaging more than the packaging tool makers and distro vendors. Believe me. But it's not going to happen until software devs become take more responsibility for what they build upstream, and how they build it.
  - oblio 8y ago
    Pypi and Python devs themselves could start doing the curation. The package just isn't accepted into the repo if it doesn't follow X, Y, Z guidelines. Python is big and important enough that it needs these kinds of quality assurances. Punting on them is as bad as compromising on language design decisions.
seanc 8y ago
No docker containers?
- s_kilk 8y ago
  That would just make the chart recursive.
- julienchastang 8y ago
  That's how I've solved this problem. Containers (i.e., clean-slate, defined media) give me something I can reason about and control.
deleted 8y ago
[deleted]
sudders 8y ago
It's funny because it's true
devy 8y ago
All jokes are only funny when they are resembling to the truth but they are not when it's far from it. So this maybe funny five years ago. But it doesn't any more since Docker[1]. docker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon. [1]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/
- fenwick67 8y ago
  Oh look, we found one more arrow to add to the chart!
  - devy 8y ago
    You mean one more solution to add to the chart to solve that mess?
    - cdancette 8y ago
      https://xkcd.com/927/ https://xkcd.com/927/
    - adtac 8y ago
      That's what every solution aims to be at the start.
      - your-nanny 8y ago
        One noodle to rule them all.
    - JustSomeNobody 8y ago
      Every solution has drawbacks. What are some for the solution you propose?
- icebraining 8y ago
  Sorry, can't run Docker on the Windows Server 2012 machines provided by the client. Or on the CentOS 6 machines provided by the other.
neuland 8y ago
This doesn't reflect the reality of what devs should be using: virtualenv or docker.
- djhworld 8y ago
  I guess if you view it from a newcomers perspective it can be quite daunting, especially on operating systems like OSX.
  - djsumdog 8y ago
    Yea macos is kinda a clusterfuck when it comes to dealing with Linux packages. Now you're fighting between system, brew and pip and none of it is fun. At least in Linux you're pretty much dealing with system python packages and user python packages.
    - neuland 8y ago
      Maybe this is my experience biasing things, but it doesn't seem that complicated to me even on Mac. - Install brew w/ the one-liner on https://brew.sh https://brew.sh - `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on - Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you. This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv. Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal. Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar). And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
- djsumdog 8y ago
  I use Docker and like it over virutalenvs, but even without it, if you're just playing around/sandboxing, pip3 --user is your friend. Don't use pip for system packages ever! Let your distros package manager handle that.
kissgyorgy 8y ago
Still missing from the picture: pipenv, virtualenv, virtualenvwrapper, pyenv
- hobofan 8y ago
  I think the comic should be more aptly be titled "Life before pipenv". Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
  - nothrabannosir 8y ago
    Not to be glib, but your comment ironically reminds me of this: https://xkcd.com/927/ https://xkcd.com/927/ No amount of adding things to the Python comic is going to simplify it.
    - cecilpl2 8y ago
      I think the intent here is to convince adherents of other tools to switch to pipenv. Having used most of the solutions in this thread, pipenv has been the simplest and the one that "just works" most often.
    - hobofan 8y ago
      Yeah, that one also popped into my mind when I wrote the comment. > No amount of adding things to the Python comic is going to simplify it. It doesn't simplify it, but it puts you in a whole new canvas where you don't have to worry about whats in the messed up one.
  - cecilpl2 8y ago
    Pipenv has been my solution as well. It handles venvs for all my projects, dev/prod configuration files, and auto-generation of requirements files.
  - jaxtellerSoA 8y ago
LyndsySimon 8y ago
It's almost - but not quite - this bad when using a "simple" local development environment. As others have said it's not nearly this bad when you're deploying to sane infrastructure, be it containerized or not.
grx 8y ago
But why.. every sane programmer uses Python 3 and pip in a virtualenvwrapper environment, cleanly separated for non-pip packages!
- quietbritishjim 8y ago
  I've never understood the purpose of virtualenvwrapper. I only use three commands for administering virtual environments: "virtualenv -p python3 ~/envs/foo" to make one, ". ~/envs/foo/activate" to activate it, and "rm -r ~/envs/foo" to delete it. That is really not something that needs further simplification. What am I missing?
  - BerislavLopac 8y ago
    Virtualenvwrapper (I hate the name, and as a fish user I actually use virtualfish instead) has as the main advantage in keeping all virtualenvs in one place (by default ~/.virtualenvs), so you don't have to worry about the actual location of your virtualenv. This allows for a number of useful features like activating, listing and deleting available environments with autocomplete, and creating a virtualenv is also quite simplified. But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
    - MereInterest 8y ago
      It always felt like virtualenvwrapper was moving things in the wrong direction. Why should the virtualenv be located in the home directory, instead of in the project directory? The whole point of virtualenv is so that you don't have environment dependencies, and each project can have its own environment.
      - actuallyalys 8y ago
        Treating the environment files as a build artifact does make more sense than having a directory of environments somewhere else. I'm not sure there are any practical advantages to doing it that way, though. I name my environments after the project, so it's more like the environment's folder is a secondary project folder than part of the environment. Edit: It looks like pipenv, which hopefully will replace virtualenvwrapper and the like, supports storing the environment in the project folder: https://docs.pipenv.org/advanced/#changing-where-pipenv-stores-virtualenvs https://docs.pipenv.org/advanced/#changing-where-pipenv-stor...
ericfrederich 8y ago
Try installing pipsi such that it uses python3 to install simple scripts like httpie or youtube-dl and you'll see how big of a mess it really is.
dsr_ 8y ago
With 8 comments so far, I count seven solutions presented.
- znpy 8y ago
  https://xkcd.com/927/ https://xkcd.com/927/
  - mfoy_ 8y ago
    The alt-text is especially funny, now that we're in the world of USB-C and Thunderbolt 3, etc... Dongles for days.
- WDCDev 8y ago
  I thought I was the only one that noticed. It's certainly makes it clear why this type of situation persists.
dmart 8y ago
There was a time where I would have strongly related to this. Thankfully, now I've started using pipenv whenever possible and it basically just works(TM). I hope it becomes the standard going forward.
- tomwphillips 8y ago
  Agreed. pipenv is wonderful.
- cecilpl2 8y ago
  Yes, pipenv is great and I am hopeful it gets adopted into the standard.
- TheCowboy 8y ago
  Seconding this recommendation of pipenv. https://github.com/pypa/pipenv https://github.com/pypa/pipenv It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management. It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
- yash1th 8y ago
  I believe pipenv still uses virtualenv instead of python's venv. Isn't that a drawback to use it?
  - blattimwind 8y ago
    Who would know?
    - yash1th 8y ago
      Well, if more and more packages in future move to python's offering venv. It would certainly be a problem that right now we are already facing
- marviel 8y ago
smilekzs 8y ago
It's funny because within the comment section there are already 2 different opinions on why this is not funny because $xyz is what every sane python dev should use.
Myrmornis 8y ago
Maybe this is obvious, but to informally install executables supplied by python packages, use a separate virtualenv for each one and place a symlink to the venv/bin/theexecutable in a location like ~/bin that you have on your $PATH.
- icebraining 8y ago
  pipsi is great for that, it takes care of creating the virtualenv, etc.
tomkinstinch 8y ago
Python environment headaches have been largely solved for me thanks to Miniconda[1]. Not only do the environments isolate dependencies, they can easily use different versions of Python, and can include arbitrary binary packages too. It helps that for my work in biology an extensive number of packages are available from the bioconda[2] channel (with many non-bio packages from conda-forge[3]). Environments can be described via environment files, allowing them to be transferred to collaborators, managed with source control, or included with publications to support reproducible science. If you like virtualenvs, you may want to give Miniconda a try. One current limitation is that environment files cannot specify a source channel for specific packages--they're installed from channels based on the global channel precedence. 1. https://conda.io/miniconda.html https://conda.io/miniconda.html 2. https://bioconda.github.io https://bioconda.github.io 3. https://conda-forge.org https://conda-forge.org
- icebraining 8y ago
  I used Conda recently and it was OK, but it's annoying that it can't install stuff directly from PyPI. Unless you're lucky, you still have to use Pip inside the Conda environment to install some packages.
  - makmanalp 8y ago
    A sparkle of hope is that the conda people recently announced in a github ticket that this would be supported as a first class citizen soon.
- jhbadger 8y ago
  But it's annoying how many packages like qiime make it basically a requirement to use Anaconda/miniconda. I really don't want to pollute my computer with multiple installations of even things like R just to run a package.
  - pwang 8y ago
    What if the right answer to the packaging nightmare that plagues languages with native extensions, is to actually have multiple isolated copies of the interpreter, each with well-known build chain metadata and sane, isolated locations for shared libraries?
rbanffy 8y ago
What's wrong with virtualenv/venv and pip?
- rimunroe 8y ago
  Those seem great, but it doesn't help much if you haven't been using them from the start and now find yourself with a very messy environment.
  - rbanffy 8y ago
    People should teach proper computer hygiene in school ;-)
- pwang 8y ago
  * The package formats (source or wheel) do not sufficiently capture assumptions that devs make at release time, in order to produce a usable installable artifact at install time. * Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system. * Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work. A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules. Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)
jwilk 8y ago
Do edges in this graph mean something, or were they just added randomly?
- titanomachy 8y ago
  Symlinks.
cecilpl2 8y ago
This thread is an instance of https://xkcd.com/927/ https://xkcd.com/927/
hood_syntax 8y ago
I think people are missing that he seems to be aware this is a poor way of doing things...
clarkmoody 8y ago
"This is not a place of honor" https://news.ycombinator.com/item?id=11851871 https://news.ycombinator.com/item?id=11851871 https://archive.is/MVVU1 https://archive.is/MVVU1 Explanation edit: Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."
- Depllo 8y ago
  Care to explain a little? The link in the discussion is a broken error page.
  - deleted 8y ago
    [deleted]
  - tofof 8y ago
    "This place is not a place of honor" is the first line of the most basic signage created by a Sandia National Lab project aiming to keep future explorers/archaeologists/looters/settlers unharmed by a nuclear waste dump. The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old. It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years. It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc. 1: https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_civilization https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c... claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000 BCE,[32] the first civilizations did not emerge until around 3000 BCE in Egypt[33] and Mesopotamia.[34]"
xnyanta 8y ago
Honestly, Python on OSX is the most confusing of all. The OS comes with a python runtime, which is different from the runtime you can download from python.org. You can then also install it from Homebrew or from MacPorts. I'm probably even missing something.
- iaml 8y ago
  When I first got mac, I've spent the whole evening trying to install python3+pipenv and get it working properly. Ended up nuking system's python 2.7 and installing it through homebrew.
- psychometry 8y ago
  Should I set up this project with conda, miniconda, virtualenv, docker, some combination of the above...? Fuck it, I'll just use R.
  - RayDonnelly 8y ago
    You can install R and some commonly used R packages with: conda install r-essentials
- nicolashahn 8y ago
  This was also frustrating me til recently. My imperfect solution is to use pyenv, which you can get through homebrew. Try it out.
- jimwalsh 8y ago
  Homebrew installing Python and installing virtualenv and virtualenvwrapper makes life easy. All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point. I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.
deleted 8y ago
[deleted]
davidfstr 8y ago
I love this comic. I was just going through an old server of mine. Python 2 and Python 3 dependencies, installed over years, in a bit of a mess. We may have virtualenv and the like today, but a lot of old stuff was never setup to use it. :p
scrollaway 8y ago
Man it's frustrating to see this comic and read the comments here, because it's all so true. Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps. Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers. To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).
- devy 8y ago
  > To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
  - mikepurvis 8y ago
    If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.
- jwilk 8y ago
  > Pipenv [...] has been officially blessed as "The One True Way" of installing stuff by the Python documentation. Oh? I don't see any mention of pipenv in the cpython git repo.
est 8y ago
Need add pypy, cython, etc.
wruza 8y ago
Did they forget site-packages vs dist-packages? It’s so tempting to just apt install everything until it’s too late.
- djsumdog 8y ago
  I forgot about that. That annoys me to no end, and that's purely due to the Debian/Ubuntu package maintainers right?
  - wruza 8y ago
    Sorry, idk. I just went that route and then was like well, maybe this time I should simply return and rethink the choice of a language. Didn’t do much credit to an engineer, but honestly, I didn’t feel myself a mess cleaner that day.
  - jwilk 8y ago
    What's wrong with dist-packages? Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
    - coldtea 8y ago
      Users that are programmers also have to package Python packages. Sometimes in .deb form for Ubuntu too... It's not just official Debian packages that someone needs to maintain...
moonbug 8y ago
Anaconda. That's all you need to know.
avian 8y ago
One more thing to add to the mess: PyBOMBS [1] A while back GNU Radio went and reinvented the whole package manager thing on their own. Since then I constantly have problems integrating anything GNU Radio-related with the rest of the Python ecosystem. Random Python 2 modules being imported by Python 3? You bet it's some random gr module in the PYTHONPATH somewhere. [1] https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-the-why/ https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-t...
outsideoflife 8y ago
I'm using pyvenv even though it has started warning me I should move. I won't change from it for exactly the reasons in this comic. When I do a new OS install I will try pipenv again...but it dind't work well for me the first time due to conflicting with something I had installed outside of a virtualenv!
yagyu 8y ago
The 'framework build and venv on osx' thing ruined my night yesterday so this post had great timing. Seriously, starting from a clean osx, how do you install python? I always liked using vex for venv management. But I would also like to have anaconda for some tasks..
- deleted 8y ago
  [deleted]
Analemma_ 8y ago
In an earlier xkcd blag post, Randall talked about messing up his Linux box so badly that someone trying to help him fix it said "your computer looks like a redshirt after a transporter accident". That analogy has stuck with me ever since and I use it most often to describe Python installations.
- jwilk 8y ago
  https://blog.xkcd.com/2009/04/03/what-happened-to-my-laptop/ https://blog.xkcd.com/2009/04/03/what-happened-to-my-laptop/
lcnmrn 8y ago
sudo pip3 install everything
dguo 8y ago
One of the reasons I try to use Docker for everything is because it's a cross-language solution. It does add some complexity and has its drawbacks, but for me, it's worth it. I'd rather have one global dependency (Docker) than several (virtualenv for Python, nvm for Node, rbenv for Ruby, etc.), which all have their own idiosyncrasies. I also like using Docker because it handles non-language resources like databases. Even installing one version of Postgres locally was painful. It's hard to imagine trying to deal with multiple instances and versions on the same machine (without something like Docker or Vagrant) if two projects use 9.x, and another uses 10.x.
- djsumdog 8y ago
  I use the stock Dockerhub python container with a requirements.txt. Similar to my Ruby code. It's a lot nicer than virtual envs for deployment. It's still difficult to debug stuff inside your container (unless you do a docker exec and then you're limited to the tools you installed) so it's useful to have the same environment outside of it. `pip3 --user install` has been my friend; and you can create new users instead of full virtulenvs if you really need to. Still, my python environment isn't as screwed up as this comic. System python is always only controlled by the OS package manager. My own stuff is installed with --user and things I need to deploy somewhere have a Dockerfile or I have a Jenkins workflow that create rpms/debs with fpm and push them to my repo and sign them.
sologoub 8y ago
This is so true and so sad that it’s actually funny! This issue is one of the things that drove me to utilize a dedicated VPS for each project. This keeps environments clean, ensures project/client isolation and allows me to utilize the exact resources needed, as opposed to worrying about the local machine. Once done or if the prod environment changes, I can blow away the VPS and rebuild as needed.
kingofpandora 8y ago
Maybe I just never do anything complicated with Python, but I never have any the problems captured in this comic. I install all my Python modules into the system's Python using my OS's package manager. I have both Python 2 and Python 3 installed but beyond having the same module installed in each of those locations, I have never found the need for multiple versions of the same module installed at the same time. On the rare occasion where my OS doesn't provide a package I need, I use pip install --user. If it turns out to be something I'll want for the long term, I just knock together a quick package that I can install/uninstall using standard OS packaging tools.
- coldtea 8y ago
  >I install all my Python modules into the system's Python using my OS's package manager. If that's the case, then it's as you said: you "just never do anything complicated with Python".
  - gkya 8y ago
    Most of the time the language package manager for a very popular language is an overkill. You can get all the packages you need for web development, AI or statistical/mathematical stuff in python from say debian packages and build arbitrarily complex applications. And you have the additional benefit of being able to easily package your application for many distributions that package stable versions of popular libraries for languages like python, perl, ruby, etc. Another additional benefit is that because most of the time you fetch packages precompiled, you don't need to run probably insecure compilations on your system (many popular packages for Python for example require you to compile C extensions). And the trade-off is quite palatable: you use the APIs hygenically, don't do things like leftpad, and sometimes forego some cutting edge features (and many dynamic languages enable you to easily backport them without modifying the package itself most of the time).
    - Chyzwar 8y ago
      not really the case if you work in a team or deploy your application.
mkopinsky 8y ago
One thing that amazes me about the modern PHP ecosystem is that when it comes to distributable PHP packages, there is only one package manager (composer) and one central package repo (packagist). When it comes to binaries it's still a bit of a mess (homebrew? pecl? yum?) but the native PHP code story is clearer than any other language ecosystem I've seen.
coldtea 8y ago
LOL, haven't check if this XKCD was posted today, but it was particularly timely as I've spend the day debugging a Python + Jenkins + autotools + setuptools + RPM spec (mock etc) setup...
BerislavLopac 8y ago
To add on to the mess, Poetry seems like a strong contestant for a well-designed Python dependency management solution: https://github.com/sdispater/poetry https://github.com/sdispater/poetry
wocram 8y ago
Am I the only one using Nix (the package manager) for this? It's lovely and works for other languages and binary package management as well. I encourage people to try it out, interactive usage looks something like `nix-shell -p python36Packages.pyaml -p python36Packages.aiohttp --run "python some-script-with-dependencies.py" `.
vram22 8y ago
Guido today: https://mobile.twitter.com/gvanrossum/status/991083246220853248 https://mobile.twitter.com/gvanrossum/status/991083246220853...

307 points by mgrayson 8y ago

I use pipenv and couldn't be happier. It's dead simple and it works. It is a bit slow. The problem presented in this xkcd has been fixed.

gkya 8y ago

I guess what Python needs to do is to crown one solution as the official, and then start imposing on PyPA packages to convert before a deadline or exclude. Then they can move the non-compliant packages to a legacy archive, and build on the one solution that the community chose, say in a poll. Granted I've only ever interacted pip and virtualenv and though it was simple; but looking at this, things don't have to be this complex.

pwang 8y ago
The heart of the problem is that different people are doing different things with Python, and nobody wants to pay cognitive load for what they don't want to use. The packagers and package tool makers of the world inherit ALL of the technical debt from ALL of the upstream software devs. They're either the liver or the colon. An upstream dev decided to not document which compiler they use during dev time? That's now your problem. A compiler maker (GNU, MSFT, etc.) decides to change how they distribute the C runtime? Congrats, now it's your packaging system's job to know how to differentiate between Windows 7 and Windows 10 running particular versions of Visual Studio. Nobody wants a "single simple unified solution" for packaging more than the packaging tool makers and distro vendors. Believe me. But it's not going to happen until software devs become take more responsibility for what they build upstream, and how they build it.
- oblio 8y ago
  Pypi and Python devs themselves could start doing the curation. The package just isn't accepted into the repo if it doesn't follow X, Y, Z guidelines. Python is big and important enough that it needs these kinds of quality assurances. Punting on them is as bad as compromising on language design decisions.

seanc 8y ago

No docker containers?

s_kilk 8y ago
That would just make the chart recursive.
julienchastang 8y ago
That's how I've solved this problem. Containers (i.e., clean-slate, defined media) give me something I can reason about and control.

deleted 8y ago

[deleted]

sudders 8y ago

It's funny because it's true

devy 8y ago

All jokes are only funny when they are resembling to the truth but they are not when it's far from it. So this maybe funny five years ago. But it doesn't any more since Docker[1]. docker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon. [1]: https://hub.docker.com/_/python/ https://hub.docker.com/_/python/

fenwick67 8y ago
Oh look, we found one more arrow to add to the chart!
- devy 8y ago
  You mean one more solution to add to the chart to solve that mess?
  - cdancette 8y ago
    https://xkcd.com/927/ https://xkcd.com/927/
  - adtac 8y ago
    That's what every solution aims to be at the start.
    - your-nanny 8y ago
      One noodle to rule them all.
  - JustSomeNobody 8y ago
    Every solution has drawbacks. What are some for the solution you propose?
icebraining 8y ago
Sorry, can't run Docker on the Windows Server 2012 machines provided by the client. Or on the CentOS 6 machines provided by the other.

neuland 8y ago

This doesn't reflect the reality of what devs should be using: virtualenv or docker.

djhworld 8y ago
I guess if you view it from a newcomers perspective it can be quite daunting, especially on operating systems like OSX.
- djsumdog 8y ago
  Yea macos is kinda a clusterfuck when it comes to dealing with Linux packages. Now you're fighting between system, brew and pip and none of it is fun. At least in Linux you're pretty much dealing with system python packages and user python packages.
  - neuland 8y ago
    Maybe this is my experience biasing things, but it doesn't seem that complicated to me even on Mac. - Install brew w/ the one-liner on https://brew.sh https://brew.sh - `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on - Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you. This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv. Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal. Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar). And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
djsumdog 8y ago
I use Docker and like it over virutalenvs, but even without it, if you're just playing around/sandboxing, pip3 --user is your friend. Don't use pip for system packages ever! Let your distros package manager handle that.

kissgyorgy 8y ago

Still missing from the picture: pipenv, virtualenv, virtualenvwrapper, pyenv

hobofan 8y ago
I think the comic should be more aptly be titled "Life before pipenv". Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
- nothrabannosir 8y ago
  Not to be glib, but your comment ironically reminds me of this: https://xkcd.com/927/ https://xkcd.com/927/ No amount of adding things to the Python comic is going to simplify it.
  - cecilpl2 8y ago
    I think the intent here is to convince adherents of other tools to switch to pipenv. Having used most of the solutions in this thread, pipenv has been the simplest and the one that "just works" most often.
  - hobofan 8y ago
    Yeah, that one also popped into my mind when I wrote the comment. > No amount of adding things to the Python comic is going to simplify it. It doesn't simplify it, but it puts you in a whole new canvas where you don't have to worry about whats in the messed up one.
- cecilpl2 8y ago
  Pipenv has been my solution as well. It handles venvs for all my projects, dev/prod configuration files, and auto-generation of requirements files.
- jaxtellerSoA 8y ago

LyndsySimon 8y ago

It's almost - but not quite - this bad when using a "simple" local development environment. As others have said it's not nearly this bad when you're deploying to sane infrastructure, be it containerized or not.

grx 8y ago

But why.. every sane programmer uses Python 3 and pip in a virtualenvwrapper environment, cleanly separated for non-pip packages!

quietbritishjim 8y ago
I've never understood the purpose of virtualenvwrapper. I only use three commands for administering virtual environments: "virtualenv -p python3 ~/envs/foo" to make one, ". ~/envs/foo/activate" to activate it, and "rm -r ~/envs/foo" to delete it. That is really not something that needs further simplification. What am I missing?
- BerislavLopac 8y ago
  Virtualenvwrapper (I hate the name, and as a fish user I actually use virtualfish instead) has as the main advantage in keeping all virtualenvs in one place (by default ~/.virtualenvs), so you don't have to worry about the actual location of your virtualenv. This allows for a number of useful features like activating, listing and deleting available environments with autocomplete, and creating a virtualenv is also quite simplified. But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
  - MereInterest 8y ago
    It always felt like virtualenvwrapper was moving things in the wrong direction. Why should the virtualenv be located in the home directory, instead of in the project directory? The whole point of virtualenv is so that you don't have environment dependencies, and each project can have its own environment.
    - actuallyalys 8y ago
      Treating the environment files as a build artifact does make more sense than having a directory of environments somewhere else. I'm not sure there are any practical advantages to doing it that way, though. I name my environments after the project, so it's more like the environment's folder is a secondary project folder than part of the environment. Edit: It looks like pipenv, which hopefully will replace virtualenvwrapper and the like, supports storing the environment in the project folder: https://docs.pipenv.org/advanced/#changing-where-pipenv-stores-virtualenvs https://docs.pipenv.org/advanced/#changing-where-pipenv-stor...

ericfrederich 8y ago

Try installing pipsi such that it uses python3 to install simple scripts like httpie or youtube-dl and you'll see how big of a mess it really is.

dsr_ 8y ago

With 8 comments so far, I count seven solutions presented.

znpy 8y ago
https://xkcd.com/927/ https://xkcd.com/927/
- mfoy_ 8y ago
  The alt-text is especially funny, now that we're in the world of USB-C and Thunderbolt 3, etc... Dongles for days.
WDCDev 8y ago
I thought I was the only one that noticed. It's certainly makes it clear why this type of situation persists.

dmart 8y ago

There was a time where I would have strongly related to this. Thankfully, now I've started using pipenv whenever possible and it basically just works(TM). I hope it becomes the standard going forward.

tomwphillips 8y ago
Agreed. pipenv is wonderful.
cecilpl2 8y ago
Yes, pipenv is great and I am hopeful it gets adopted into the standard.
TheCowboy 8y ago
Seconding this recommendation of pipenv. https://github.com/pypa/pipenv https://github.com/pypa/pipenv It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management. It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
yash1th 8y ago
I believe pipenv still uses virtualenv instead of python's venv. Isn't that a drawback to use it?
- blattimwind 8y ago
  Who would know?
  - yash1th 8y ago
    Well, if more and more packages in future move to python's offering venv. It would certainly be a problem that right now we are already facing
marviel 8y ago

smilekzs 8y ago

It's funny because within the comment section there are already 2 different opinions on why this is not funny because $xyz is what every sane python dev should use.

Myrmornis 8y ago

Maybe this is obvious, but to informally install executables supplied by python packages, use a separate virtualenv for each one and place a symlink to the venv/bin/theexecutable in a location like ~/bin that you have on your $PATH.

icebraining 8y ago
pipsi is great for that, it takes care of creating the virtualenv, etc.

tomkinstinch 8y ago

Python environment headaches have been largely solved for me thanks to Miniconda[1]. Not only do the environments isolate dependencies, they can easily use different versions of Python, and can include arbitrary binary packages too. It helps that for my work in biology an extensive number of packages are available from the bioconda[2] channel (with many non-bio packages from conda-forge[3]). Environments can be described via environment files, allowing them to be transferred to collaborators, managed with source control, or included with publications to support reproducible science. If you like virtualenvs, you may want to give Miniconda a try. One current limitation is that environment files cannot specify a source channel for specific packages--they're installed from channels based on the global channel precedence. 1. https://conda.io/miniconda.html https://conda.io/miniconda.html 2. https://bioconda.github.io https://bioconda.github.io 3. https://conda-forge.org https://conda-forge.org

icebraining 8y ago
I used Conda recently and it was OK, but it's annoying that it can't install stuff directly from PyPI. Unless you're lucky, you still have to use Pip inside the Conda environment to install some packages.
- makmanalp 8y ago
  A sparkle of hope is that the conda people recently announced in a github ticket that this would be supported as a first class citizen soon.
jhbadger 8y ago
But it's annoying how many packages like qiime make it basically a requirement to use Anaconda/miniconda. I really don't want to pollute my computer with multiple installations of even things like R just to run a package.
- pwang 8y ago
  What if the right answer to the packaging nightmare that plagues languages with native extensions, is to actually have multiple isolated copies of the interpreter, each with well-known build chain metadata and sane, isolated locations for shared libraries?

rbanffy 8y ago

What's wrong with virtualenv/venv and pip?

rimunroe 8y ago
Those seem great, but it doesn't help much if you haven't been using them from the start and now find yourself with a very messy environment.
- rbanffy 8y ago
  People should teach proper computer hygiene in school ;-)
pwang 8y ago
* The package formats (source or wheel) do not sufficiently capture assumptions that devs make at release time, in order to produce a usable installable artifact at install time. * Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system. * Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work. A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules. Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)

jwilk 8y ago

Do edges in this graph mean something, or were they just added randomly?

titanomachy 8y ago
Symlinks.

cecilpl2 8y ago

This thread is an instance of https://xkcd.com/927/ https://xkcd.com/927/

hood_syntax 8y ago

I think people are missing that he seems to be aware this is a poor way of doing things...

clarkmoody 8y ago

"This is not a place of honor" https://news.ycombinator.com/item?id=11851871 https://news.ycombinator.com/item?id=11851871 https://archive.is/MVVU1 https://archive.is/MVVU1 Explanation edit: Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."

Depllo 8y ago
Care to explain a little? The link in the discussion is a broken error page.
- deleted 8y ago
  [deleted]
- tofof 8y ago
  "This place is not a place of honor" is the first line of the most basic signage created by a Sandia National Lab project aiming to keep future explorers/archaeologists/looters/settlers unharmed by a nuclear waste dump. The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old. It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years. It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc. 1: https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_civilization https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c... claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000 BCE,[32] the first civilizations did not emerge until around 3000 BCE in Egypt[33] and Mesopotamia.[34]"

xnyanta 8y ago

Honestly, Python on OSX is the most confusing of all. The OS comes with a python runtime, which is different from the runtime you can download from python.org. You can then also install it from Homebrew or from MacPorts. I'm probably even missing something.

iaml 8y ago
When I first got mac, I've spent the whole evening trying to install python3+pipenv and get it working properly. Ended up nuking system's python 2.7 and installing it through homebrew.
psychometry 8y ago
Should I set up this project with conda, miniconda, virtualenv, docker, some combination of the above...? Fuck it, I'll just use R.
- RayDonnelly 8y ago
  You can install R and some commonly used R packages with: conda install r-essentials
nicolashahn 8y ago
This was also frustrating me til recently. My imperfect solution is to use pyenv, which you can get through homebrew. Try it out.
jimwalsh 8y ago
Homebrew installing Python and installing virtualenv and virtualenvwrapper makes life easy. All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point. I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.

deleted 8y ago

[deleted]

davidfstr 8y ago

I love this comic. I was just going through an old server of mine. Python 2 and Python 3 dependencies, installed over years, in a bit of a mess. We may have virtualenv and the like today, but a lot of old stuff was never setup to use it. :p

scrollaway 8y ago

Man it's frustrating to see this comic and read the comments here, because it's all so true. Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps. Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers. To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).

devy 8y ago
> To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
- mikepurvis 8y ago
  If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.
jwilk 8y ago
> Pipenv [...] has been officially blessed as "The One True Way" of installing stuff by the Python documentation. Oh? I don't see any mention of pipenv in the cpython git repo.

est 8y ago

Need add pypy, cython, etc.

wruza 8y ago

Did they forget site-packages vs dist-packages? It’s so tempting to just apt install everything until it’s too late.

djsumdog 8y ago
I forgot about that. That annoys me to no end, and that's purely due to the Debian/Ubuntu package maintainers right?
- wruza 8y ago
  Sorry, idk. I just went that route and then was like well, maybe this time I should simply return and rethink the choice of a language. Didn’t do much credit to an engineer, but honestly, I didn’t feel myself a mess cleaner that day.
- jwilk 8y ago
  What's wrong with dist-packages? Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
  - coldtea 8y ago
    Users that are programmers also have to package Python packages. Sometimes in .deb form for Ubuntu too... It's not just official Debian packages that someone needs to maintain...

moonbug 8y ago

Anaconda. That's all you need to know.

avian 8y ago

One more thing to add to the mess: PyBOMBS [1] A while back GNU Radio went and reinvented the whole package manager thing on their own. Since then I constantly have problems integrating anything GNU Radio-related with the rest of the Python ecosystem. Random Python 2 modules being imported by Python 3? You bet it's some random gr module in the PYTHONPATH somewhere. [1] https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-the-why/ https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-t...

outsideoflife 8y ago

I'm using pyvenv even though it has started warning me I should move. I won't change from it for exactly the reasons in this comic. When I do a new OS install I will try pipenv again...but it dind't work well for me the first time due to conflicting with something I had installed outside of a virtualenv!

yagyu 8y ago

The 'framework build and venv on osx' thing ruined my night yesterday so this post had great timing. Seriously, starting from a clean osx, how do you install python? I always liked using vex for venv management. But I would also like to have anaconda for some tasks..

deleted 8y ago
[deleted]

Analemma_ 8y ago

In an earlier xkcd blag post, Randall talked about messing up his Linux box so badly that someone trying to help him fix it said "your computer looks like a redshirt after a transporter accident". That analogy has stuck with me ever since and I use it most often to describe Python installations.

jwilk 8y ago
https://blog.xkcd.com/2009/04/03/what-happened-to-my-laptop/ https://blog.xkcd.com/2009/04/03/what-happened-to-my-laptop/

lcnmrn 8y ago

sudo pip3 install everything

dguo 8y ago

One of the reasons I try to use Docker for everything is because it's a cross-language solution. It does add some complexity and has its drawbacks, but for me, it's worth it. I'd rather have one global dependency (Docker) than several (virtualenv for Python, nvm for Node, rbenv for Ruby, etc.), which all have their own idiosyncrasies. I also like using Docker because it handles non-language resources like databases. Even installing one version of Postgres locally was painful. It's hard to imagine trying to deal with multiple instances and versions on the same machine (without something like Docker or Vagrant) if two projects use 9.x, and another uses 10.x.

djsumdog 8y ago
I use the stock Dockerhub python container with a requirements.txt. Similar to my Ruby code. It's a lot nicer than virtual envs for deployment. It's still difficult to debug stuff inside your container (unless you do a docker exec and then you're limited to the tools you installed) so it's useful to have the same environment outside of it. `pip3 --user install` has been my friend; and you can create new users instead of full virtulenvs if you really need to. Still, my python environment isn't as screwed up as this comic. System python is always only controlled by the OS package manager. My own stuff is installed with --user and things I need to deploy somewhere have a Dockerfile or I have a Jenkins workflow that create rpms/debs with fpm and push them to my repo and sign them.

sologoub 8y ago

This is so true and so sad that it’s actually funny! This issue is one of the things that drove me to utilize a dedicated VPS for each project. This keeps environments clean, ensures project/client isolation and allows me to utilize the exact resources needed, as opposed to worrying about the local machine. Once done or if the prod environment changes, I can blow away the VPS and rebuild as needed.

kingofpandora 8y ago

Maybe I just never do anything complicated with Python, but I never have any the problems captured in this comic. I install all my Python modules into the system's Python using my OS's package manager. I have both Python 2 and Python 3 installed but beyond having the same module installed in each of those locations, I have never found the need for multiple versions of the same module installed at the same time. On the rare occasion where my OS doesn't provide a package I need, I use pip install --user. If it turns out to be something I'll want for the long term, I just knock together a quick package that I can install/uninstall using standard OS packaging tools.

coldtea 8y ago
>I install all my Python modules into the system's Python using my OS's package manager. If that's the case, then it's as you said: you "just never do anything complicated with Python".
- gkya 8y ago
  Most of the time the language package manager for a very popular language is an overkill. You can get all the packages you need for web development, AI or statistical/mathematical stuff in python from say debian packages and build arbitrarily complex applications. And you have the additional benefit of being able to easily package your application for many distributions that package stable versions of popular libraries for languages like python, perl, ruby, etc. Another additional benefit is that because most of the time you fetch packages precompiled, you don't need to run probably insecure compilations on your system (many popular packages for Python for example require you to compile C extensions). And the trade-off is quite palatable: you use the APIs hygenically, don't do things like leftpad, and sometimes forego some cutting edge features (and many dynamic languages enable you to easily backport them without modifying the package itself most of the time).
  - Chyzwar 8y ago
    not really the case if you work in a team or deploy your application.

mkopinsky 8y ago

One thing that amazes me about the modern PHP ecosystem is that when it comes to distributable PHP packages, there is only one package manager (composer) and one central package repo (packagist). When it comes to binaries it's still a bit of a mess (homebrew? pecl? yum?) but the native PHP code story is clearer than any other language ecosystem I've seen.

coldtea 8y ago

LOL, haven't check if this XKCD was posted today, but it was particularly timely as I've spend the day debugging a Python + Jenkins + autotools + setuptools + RPM spec (mock etc) setup...

BerislavLopac 8y ago

To add on to the mess, Poetry seems like a strong contestant for a well-designed Python dependency management solution: https://github.com/sdispater/poetry https://github.com/sdispater/poetry

wocram 8y ago

Am I the only one using Nix (the package manager) for this? It's lovely and works for other languages and binary package management as well. I encourage people to try it out, interactive usage looks something like `nix-shell -p python36Packages.pyaml -p python36Packages.aiohttp --run "python some-script-with-dependencies.py" `.

vram22 8y ago

Guido today: https://mobile.twitter.com/gvanrossum/status/991083246220853248 https://mobile.twitter.com/gvanrossum/status/991083246220853...