18 ms·

Why not tell people to “simply” use pyenv, poetry or anaconda

89 points by agos 3y ago

jeroenhd 3y ago
I like venvs and hate dealing with the system wide impact of anaconda (and the weird side projects you need for it to resolve dependencies within a single human lifetime) but if I see something that uses it, I just suck it up. It's annoying that my specific platform isn't supported, but unless I'm paying for some kind of support contract, I don't expect the devs to make things easy for me. They make what they need, and what apparently tons of other people also need, so I'm the one who will need to spend a few minutes getting everything sorted out. If you're on Windows you're going to have trouble with some packages. If you're on Linux, you're going to have trouble with others. MacOS and BSD aren't any different. From what I can tell, LTS Linux (Ubuntu, Fedora) is the most compatible, luckily. Yeah, I'd prefer to install everything in a venv. However, if I write software that works through venvs, someone with some obscure Python implementation is bound to report that the dependencies don't install in their system because of differences. They didn't choose that system just to be annoying, and neither did I. All of these solutions are valid answers to someone's problem. > Every time she digs, there is one more thing to learn, and for each thing, hundreds of ways to fail. Such is the life of someone trying to find workarounds for an upstream problem. "It works on this platform, not yours, but you can simulate that platform by..." implies that you're going to need to learn another platform or don't use the project until it's got support for your technology stack. There's plenty of Python that doesn't run on most Windows machines just like there are plenty of VBA macros that don't work on LibreOffice. Whether you solve that through workarounds, conversion systems, or installing a different platform you need to learn (in a VM or on bare metal), stuff is going to get messy. As for "just use venvs": to use venvs for various projects, you need to install cargo, clang/g++/msvc, sometimes Fortran, or even assemblers. All of these fail in spectacular fashion. Binary packages aren't a solution either, they'll just randomly fail to load (glibc mismatch, musl, or hell, "go binary loaded on Alpine by Python" just seems to fail for obscure Linux linker reasons). There is no easy answer that doesn't require some kind of additional tooling if you're trying to use big, unwieldy packages. This is the case for every programming language!
jwr 3y ago
I don't normally use Python. When I see something that I'd like to install, and the docs say "written in Python", my heart sinks. I know I'll have to find out how to install multiple dependencies in various packaging systems, all of which will be incompatible (but surely much better) than my system package manager. I find all of this ironic given that the main gripe people used to have against Common Lisp was "but how do I make a self-contained binary?". Somehow this was never required of Python.
- bombolo 3y ago
  For me if I can do apt install is good. No care which language it's written in.
  - rgoulter 3y ago
    Sure, it's convenient if there's already a system package; in which case, yes, the language is irrelevant. However, if you want to run software which isn't available as a system package, then you'll need to know how to build/run the software.
    - roflyear 3y ago
      Isn't that true of many languages?
      - arp242 3y ago
        Kind of, but some languages are definitely easier to build than others (on average).
      - rgoulter 3y ago
        Surely "to run the program from source, you need to know how to build/run it" applies to every language. But I'm more comfortable working through problems I might encounter in languages I've worked with, than in languages I'm not familiar with. I'd say Python is notorious, since you're probably going to hear about at least six or seven ways people are able to run things; so if you're not familiar with it, you're likely to do something wrong. -- e.g. elsewhere in this thread, you've pointed out that stuff should be run from a venv; that's something you have to know.
heywhatupboys 3y ago
Python's package ecosystem is probably the worst in the world
- artemonster 3y ago
  Can you elaborate what this system lacks or what are the key properties that make it „the worst“? I know its a mess, but I cant pinpoint the reason why.
  - kristofferR 3y ago
    That so many things doesn't run on the latest stable version is one of them. Python 3.11 was published in all the way back in October, and it's still not stable and well supported.
  - linkdd 3y ago
    Is the fact that there are multiple package managers for the same package fornat/spec the reason why people say it's a mess? Genuine question: How is PyPI different from NPM, Maven, crates.io, etc..? Why "Python packaging is a mess" and not the others?
  - throwawaaarrgh 3y ago
    The non-hierarchical, not-reused, reinvented package convention. The most popular package repository being incredibly hard to find one package you want among a sea of crap. The lack of a single standard solution to the most common problems. Frequent backwards compatibility issues. It's definitely not the worst. But it's hilarious how bad it is considering how popular it is and how long it's been around
- j16sdiz 3y ago
  try ruby.
  - Alifatisk 3y ago
    What's wrong with bundler / gem?
WillAdams 3y ago
That people find this comic: https://xkcd.com/1987/ https://xkcd.com/1987/ to be quite reasonable is why. It kills me that the Python folks don't have a resource like to CTAN and an established naming convention so that things (mostly) work when installed, and when things don't help is just a "minimum (not) working example" away on tex.stackexchange.com
ncphil 3y ago
Anyone who rarely uses python, or who uses it all the time, should take the time to read this article and its companions in the series. Good, solid, advice that will make life easier. Just read it and try out the suggestions made. I was amazed at how close it came to my own workflow developed over many years since migrating from perl to python.
kkirsche 3y ago
I’m not convinced by the argument as it sounds like it’s tone implies that it’s written to people who aren’t aware of who their audience is. Sure, not all of these tools are right for all audiences, but that doesn’t mean that they aren’t good suggestions for wide swathes of the ecosystem. I think it’s important instead to think about your largest user groups and provide workflow ideas for those groups. For example, with Python we may have (this is a simplified list): * Web Development * Data Science * Embedded * Compilation / Transpilation (e.g., PyO3) Of these groups, I’d expect the tools mentioned to be reasonable for two or three of the groups, but less so in groups like data science where there is a higher concentration of non-development first specialties. There are always outliers, but I think choosing your target audience is important. On a team, I wouldn’t necessarily want to teach a person a random workflow if the team all uses one of these types of tools.
kristofferR 3y ago
So what is actually the solution? Let's say I want to run Stable Diffusion WEBUI (Python 3.10) and Whisper (Python 3.11). Last I tried to run both at the same time it was hell - when I downgraded Python for Stable Diffusion as per the instructions, Whisper stopped working.
- 0x008 3y ago
  you can use pyenv to have multiple parallel python versions installed and you can choose per virtual environment which version you want to use.
  - kristofferR 3y ago
    Pyenv doesn't seem to work on Windows, is the fork recommended? Or what about Pyenv on WSL2, will Cuda work well?
    - eurekin 3y ago
      CUDA works great in a WSL2 vm. Highly recommended. I found that just sound doesn't work (any soundcard operation isn't implemented in the WSL)
- deleted 3y ago
  [deleted]
- zokier 3y ago
  To me the solution is just use system python. If some packages do not work, then patch them if easy, or just avoid using if not. Docker as last-resort solution.
- matthiaswh 3y ago
  You want something that manages different versions of Python itself. pyenv is a popular option for doing this, but I'm partial to rtx which is largely language agnostic and also handles Python virtual environments in a very elegant way. https://github.com/jdxcode/rtx https://github.com/jdxcode/rtx
- 3y ago
ipsum2 3y ago
Use conda to set up new environments and switch between them, and use pip to install modules. It's really not as complicated as people make it seem. I haven't had any issues with this set up across Linux and MacOS, doing web / ML / data development for the last 3 years. It also avoids the pitfalls mentioned in the blog post.
- gpderetta 3y ago
  I'll provide a witty reply to this comment as soon as conda finishes resolving the dependencies for my projects. Any time now...
  - suslik 3y ago
    Use mamba as a drop-in replacement, it's rather fast.
  - Garcia98 3y ago
    Have you ever tried mamba[0]? It's much, much faster than conda for me. 0: https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba
  - montebicyclelo 3y ago
    There is a faster solver for conda, that you can enable. [1] [1] A Faster Solver for Conda: Libmamba - https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community https://www.anaconda.com/blog/a-faster-conda-for-a-growing-c...
  - peteradio 3y ago
    You are complaining about built-in coffee/smoke breaks? Whats wrong with you man!?
  - ipsum2 3y ago
    This shouldn't happen if you use pip to install packages, not conda.
- YetAnotherNick 3y ago
clarionbell 3y ago
I prefer python package management to whatever is happening in the node.js world.
- agos 3y ago
  what is happening in the node.js world? people typing npm install and then being done is somehow worse than the Python situation?
  - fragmede 3y ago
    Node.js installs the libraries into the `node_modules` subfolder in the working directory of the project. Python installs the libraries in to a system dir (unless you're using venv or conda or whatever). This means that dependency mismatches are resolved on a per-project level with node.js but are on a system-wide basis for python. If `npm install` is failing, it's not unreasonable to just run `rm -rf node_modules` and then try again. Worst that happens is that project is still broken. Meanwhile, if I nuke `/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/*` on my mac to try and resolve a problem, I've broken everything else that uses python 3.11 on my system. The failure scenario with Python is when two or more projects need conflicting version of numpy or scipy or whatever. Then you start fighting with pyenv/poetry/venv/conda/docker to get a different environment to run it in. As noted, it works, eventually, but it's just more work. More work is just more work and we as developers hate doing busy work like that.
imgabe 3y ago
For me none of the things that are suppose to "relieve packaging pain" have ever done anything but cause more pain. Plain virtual environments with venv work fine. If you need a different version of python, download the source, build it and install with make altinstall, then use it to make your venv.
- shipmaker 3y ago
  Thank you for your post, I was looking for something like altinstall, I wasn't aware of it. I've used dnf to install python3.8 side by side with the default python3.6 on Almalinux and they seem to work fine, but I keep wondering if I haven't broken something.
  - Chiron1991 3y ago
    As long as you're installing/updating Python via your distro's package manager you won't break anything. Most horror stories of borked system Pythons are caused by people uninstalling/updating packages from the system Python via pip or fiddling with their PATH. altinstall is only useful if you have to compile Python from source to get it onto your system.
    - shipmaker 3y ago
      Thanks!
BiteCode_dev 3y ago
Damn, the reddit shuts down really is bringing a lot more attention to my blog. I've got links on the front pages posted twice. Well, AMA, I suppose.
rcfox 3y ago
Title nitpick: Starting with "Why not..." sounds like a suggestion. I'd recommend "Why you should not..."
brylie 3y ago
I recently switched back to venv and pip with pip-tools with dependencies configured via pyproject.toml. I have a basic Makefile containing convenience command aliases with inter-dependencies. While not perfect, the venv/pip approach has simplified my development process, allowed conventional deployments to PaaS, and made things easier for new developers. I will gladly dispense with pip-tools and Makefile if Python had/has a more capable standard library CLI.
- crabbone 3y ago
  > pip-tools ... pyproject.toml ... Makefile This is a plate of cow dang with glass shards on steroids. I wouldn't even know where to begin to explain how bad this is... I mean, you already had an abomination in the form of pip-tools and pyproject.toml, and then you decided to pepper this trash with Make? Like, come on... you are with a straight face suggesting people treat toothache with a chainsaw...
linuxftw 3y ago
I think this article is rather long-winded and not super informative. The previous post [1] lays out what one should actually do, and I agree with most of it, at least from a developer perspective. 1: https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...
- BiteCode_dev 3y ago
  This post is the justification of the one you link to. It's not meant to be practical, but rather an answer to the same comments I get over and over again when I suggest the method you just posted. Even with it though, you can see in the thread people don't read it and completely ignore its rational :)
  - linuxftw 3y ago
    I understand what you're trying to achieve here, but I think the content isn't very well organized. The people that have the chops to understand most of what you wrote won't need to read it, and those that don't won't come away with any clearer understanding. If you can trim the content to about 30% length, I think it would be better.
    - BiteCode_dev 3y ago
      Yes, it's too long, I should have made a series of 3 posts instead, each focused on one particular idea. One particular point "devs lie", really deserve its own article. Because we do lie a lot, and it's like a taboo to talk about it.
      - linuxftw 3y ago
        I don't think I'd characterize some of those things as lying. Setting up a development environment is no different than setting up other kinds of software: you obtain the software, and follow instructions to get it up and running. There's no substitute for this, unless you're setting up an environment for someone else (eg, a student), in which case, they don't need to know about many of the details.
mikepurvis 3y ago
I’m a fairly recent convert to poetry, but I definitely acknowledge the bootstrapping problem with it— it seems to work best if you have some non-Python means of acquiring poetry itself, whether that's a PPA, nixpkgs, or something else that's already "at hand".
crabbone 3y ago
> Among other things, it advised not to use homebrew, pyenv, anaconda, poetry, and other tools. Why are there so many so low quality programmers jumping on this problem? Why cannot they just step aside and leave this to people who actually understand what they are doing? OP has no clue what they are writing about. Anaconda is a different distribution of Python with a bunch of it's own packages, tools, processes... It definitely doesn't belong in the same group with homebrew, pyenv or Poetry. What are Python users supposed to do if they get a project exclusively distributed as an Anaconda package? The real list of things you shouldn't use to install Python packages is anything that's governed by PyPA -- it's all garbage. Unfortunately, this is impractical. The situation is in a way very similar to Linux and proprietary drivers. If you want Linux to run on your laptop, you will most likely end up needing proprietary drivers. And that's bad. And you have to compromise. So... setuptools? -- bad, shouldn't be used, but, realistically, you will end up using it anyways. pip? -- even worse, but it's very hard to avoid. In situations where you create a library, you'll probably have to also at some point face the need to provide it in Anaconda distribution, and you'll have to use conda too... which is also bad, but there's nothing you can do about it, and even if projects like mamba exist, it's still not going to help you because you must test your library with the tools the users are going to use on it, and it's a choice between not supporting a bunch of users and having to use crappy tools.
- secondcoming 3y ago
  Isn't setuptools deprecated for creating apt packages of python code?
  - crabbone 3y ago
    The deprecation of setuptools is a joke. There's no replacement. What PyPA does is that they produce ever more bullshit wrappers for setuptools pretending they are doing something new. I think that by "apt packages" you mean DEB packages. As someone who has to package a lot of Python both for RHEL and for Debian-like... well... so far and in the observable future, setuptools isn't going anywhere. On the other hand, I could easily do without it, and sometimes do. But it's not because PyPA produced any replacement to it. It's because I simply grow frustrated with how bad of a tool setuptools is, and I just write my own. The claims PyPA makes about setuptools are because nobody wants to support that project anymore. It's awfully written at every level, from the minutia of formatting and up to the high-level design decisions. The problem is that everything else that PyPA came up with so far is a joke. The only reason I keep paying attention to it is because I'm being paid to fix problems created by using it. I'm also going to tell you a secret... well, of sorts. DEB, and RPM too are also pathetically bad... since you seem to have some experience with DEB -- to assemble this stuff you typically have to use a bunch of Make macros. Excruciatingly painful to debug. Virtually no documentation and very few people who could help you if you have a problem. And, when it comes to Python packaging, things also changed over time. Not so long ago I saw some write-up about how to package Python including virtual environment in the package. In truth, if I were a distro maintainer, and someone came to me with a package like that, I'd told them to try elsewhere. It's a sign of disrespect to the efforts of the distro maintainers to assemble a working and coherent system. But, guess who does stuff like that? -- big companies, like Amazon or Microsoft. Instead of spending effort to figure out how to play nice with the distro's packages, they distribute a huge blob that's not humanly possible to audit.
novaRom 3y ago
Always use containers! Any big project should be sandboxed, having completely isolated environment, dependencies, and you have all installation steps well described and reproducible. Containers in Linux are lightweight citizens and using them is absolutely necessary to avoid any dependency hell and broken or trashed system. Plus, you can simply distribute them.
- jackdh 3y ago
  From the article... >> You should really use docker > I think you missed the whole point.
- dagw 3y ago
  Always use containers! Even for short scripts I write? A unique container for every script? How do I go about calling one script in one container from a script in a different container. Containers in Linux are lightweight Much of the world isn't running Linux. Deploying a single project you have developed, targeting a single OS and CPU arch, onto a server that you have complete control over is the easiest case and not really where most problems show up. The hard problem is being able to email a script I wrote to a colleague and have it run on their machine.
- bryancoxwell 3y ago
  I think you’re right. I also think that’s a big problem for python as a language. For a lot of use cases just choosing a different language from the jump that allows you to easily compile a standalone executable can make life so, so much easier.
computerfriend 3y ago
For what it's worth, python -m venv venv source venv/bin/activate pip install -r requirements.txt has never let me down and doesn't abstract away anything.
- BiteCode_dev 3y ago
  That's pretty much the procedure the article link to in the intro, but with some additional subtleties on how to source and run python.
- pera 3y ago
  That might work for a subset of projects, but try that with anything that does for instance numerical analysis stuff, like machine learning. There is a reason why Conda is a requirement for so many Python projects.
- roel_v 3y ago
  In your example, one of the abstractions is 'what Python does it use'. Because I'm dealing with this very issue right now, distributing a simple Python tool to a Windows end user, and they're on heavily managed government machines and it turns out that their 'python' command invokes a python.exe somewhere on a network drive and for some reason the '-m venv venv' fails, which then causes all the rest to fail as well of course. Now I don't know yet if this network thing is really the root cause, I've just received some error logs that showed this this morning, so I'm on another round of trying to remote debug this, via email, with someone who is (while patient and willing) not a Python programmer. Python packaging is such a ridiculous PITA, it's laughable if it wasn't so sad. I tried nuitka as suggested upthread, and send my customer a single binary to try, let's see if it works.
  - deleted 3y ago
    [deleted]
  - aragilar 3y ago
    It's probably Python 2, which won't have venv.
- 3y ago
lkrubner 3y ago
I recall the negative reaction I got, on Hacker News and on Twitter, back in 2018, when I made the point that the Python community was one of the main groups driving the adoption of Docker: "Package management in Python is so hopelessly broken that the Python community is eager to use Docker everywhere, as the only way to save the situation." In response, people laughed: "No one is using Docker for package management! Python has an abundance of good tools for that!" And yet it remains true, if package and dependency management in Python was seamless, then Docker would have had fewer advocates back in the early days. And one of the criticisms of Docker remains that it is sometimes used as a Band-Aid for flaws in some computer programming language, flaws that need a more fundamental solution.
- BiteCode_dev 3y ago
  To be honest, you still should use a venv in a docker image.
  - crabbone 3y ago
    Insert the "if those people (behind the window with a poster) could read" meme. Seriously, if Python developers could produce a coherent thought, we wouldn't be here. They aren't bothered by a contradiction you pointed out. They still push for Docker adoption, even though it's not helping them, because they "want to believe".
- macNchz 3y ago
  This is likely true for me–I’m a fan of Docker and a longtime Python developer. Docker rose in popularity at around the same time that many applications were in between Python 2 and 3, and putting apps in containers was a bit of a magic wand to wave away various issues, some of which still exist and some that were worse during that transition. I still like Docker regardless of Python–I wouldn’t want to go back to development using a vm-per-app, or dealing with mismatched db/cache/queue/system package versions between projects or across different developers machines.
- deepsquirrelnet 3y ago
  I’ve been writing python code for about 19 years, and I remember yours or similar arguments about docker. I completely agree. Docker has more or less saved python from its own packaging issues, especially in the cloud. On local machines, I really found the most comical show of python distribution wreckage in RedHat/CentOS distributions. Back in 2015 or so, yum was tied to the system version of python. Heaven help you if you didn’t realize it and went mucking around on it (like me).
cpfohl 3y ago
This problem is not unique to Python, it’s just the proliferation of incompatible tools that makes Python quite so annoying to use. I’ve found that none of this is really an issue if you provide a container as a dev environment. Before someone accuses me of “yet another standard”ing this, it’s my standard across all my programming environments. It’s not a silver bullet, especially with Apple silicon Docker can be a massive pain in the butt, but it works for me. Inside the container I can be guaranteed that only a single Python version is installed (correctly) and packages are installed in a standard way from a package file. Adding a package can be scripted so that it gets executed inside the container and adds it to you packages file. The other massive benefit is that your local machine needs no locally installed tools other than docker (or podman, or whatever). this works for me and my team it’s not perfect, but it is a complex problem.
- crabbone 3y ago
  It's not "this" problem. It's many problems together. Some of them, while maybe not unique to Python are completely preventable. Where the right thing to do was obvious, but Python core dev decided to cut corners, or do nothing etc. So, we need to be fair and put the blame where it belongs. Python packaging format didn't just "happen", it was designed... by a bunch of half-wits, but that's exactly the problem! Python import system didn't "just happen", it was designed, by mostly the same kind of people -- and, again, that's the problem! If anyone wanted to put a good effort and thinking into designing packaging format and the import system, they'd realized that they need to deal with multiple versions of the same package somehow, they would've realized that they need to distinguish between binary distribution and source / documentation / data packages, that they need a reliable tool for installing those packages. But nothing of the kind was done. Every "development" in this area was incrementally adding more technical debt. Is technical debt a new thing? -- surely not. Is it preventable? -- well, not so much, but being cognizant of it one could establish certain practices to mitigate the adverse effects. And this, this particular thing, has never been done in Python. Is this a common failure? -- well, again yes, but this doesn't vindicate Python core developers, they are just as guilty as a bunch of other people who suck at programming.
TheAceOfHearts 3y ago
For a language purported to be user friendly, Python sure does have an extremely unfriendly package management ecosystem. I'm sure a lot of it comes down to familiarity though. With nodejs I'll use asdf to switch and manage runtime versions, and npm to install packages per project. Dealing with transpilers and bundlers is terrible experience though. The best programming language that I know to get everything right with the out of box experience is Rust. It has a tool that allows me to switch Rust compiler versions and cargo Just Works.
elzbardico 3y ago
Half the problems in this long article can be avoided by not using Windows. I wouldn't try developing a .NET Forms application with Sublime Text on Gentoo, would I? Open Source is done by people solving their own problems and by their own volition, sharing their solutions with the community. Some of those projects are useful enough for people to become commercial or semi-commercial endeavors. But yet, resources are limited, and compatibility with Windows will only be there if more people care about it than they care for other features. Most Linux distributions have no licensing costs, are highly compatible with most basically the same hardware Windows runs on, and even more, in very inexpensive hardware like raspberries pi where Windows doesn't even run. So, given that resources for development are not infinite, I think that we should be glad that Linux and other Unix-like environments are the baseline target for compatibility for python packages instead of Windows. Also, most Python software that is of interest to non-programmers like Jupyter Notebooks, PyTorch and Keras is still of a technical nature. It is not like people are installing mail clients or Word Processors written in Python with pip install. Even if those projects are being used by non-professional programmers themselves, the user must learn some programming to use them. It is reasonable that someone wanting to use such sophisticated packages will have to devote some time to learn how to properly install them. Again, someone could argue this is unacceptable, but this comes from the reality that making this easier requires resources, programmer time, testing and access to environments for said testing. Most open source project don't have the luxury of access to vast farms of devices with hundreds of different combinations of hardware, devices and software as a company like Microsoft does. But on the other hand, they are not asking you thousands of dollars for a Microsoft Subscription, neither they receive a tax the moment you bought your machine with their operating system installed.
andybak 3y ago
> "There are usually two kinds of coders giving advice. A fresh one that has no idea how complex things really are yet. Or an experienced one, that has forgotten it." This line is gold.
montebicyclelo 3y ago
The disadvantage of this approach is that you can't easily switch major Python versions per project. I use miniconda [1], and I'll create a conda env per project, with the newest version of Python that is compatible with the dependencies. Also, conda doesn't just install Python packages, it can also install cuda drivers (e.g. for GPU accelerated pytorch/tensorflow [2]), it can also install ffmpeg, etc. The complaint in the article is true, that you do generally need to do a mixture of `conda install` and `pip install` depending on whether the package is in a conda package repository or not; but I've not found this too bad in practice - I look up whether the package is in a conda repo, and install from pip if it isn't. Also it would probably be better for conda to be automatically activated for a given project / folder, rather than needing manual activation via `conda activate <name>`, but you get used to it. [1] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html [2] https://pytorch.org/ https://pytorch.org/ <- conda is listed in pytorch installation guide
- crabbone 3y ago
  1. Anaconda tooling and PyPI tooling are just different things, you cannot substitute one for the other, so whether you had any success with Anaconda or not is really irrelevant to OPs problem. 2. How much more easier to you want this to be: python3.8 -m venv .venv38 python3.11 -m venv .venv311 This, at least for me, creates two environments with two different minor versions of Python. NB. I think, you meant "minor" version -- why would you want to switch between Python 2.X and Python 3.X today? Or did you actually meant that?
  - montebicyclelo 3y ago
    1. Not sure what you mean? "is irrelevant to OPs problem" -> op advises to not use conda as a solution for Python packaging 2. This is easier than what you propose, because conda will install the python version for the env on the fly (including the patch version): conda create -n myproject python=3.9.4 conda activate myproject NB. if we're being pedantic, conda and anaconda are different Yep, should be minor, but I chose to use the same language as the article [1] [1] https://www.bitecode.dev/i/110377053/dont-install-the-latest-major-version-of-python https://www.bitecode.dev/i/110377053/dont-install-the-latest...
gtirloni 3y ago
What worries me is that any solution to this problem could a Python 2->3 decade-long effort again.
jjgreen 3y ago
Very true. One (quite narrow) use-case does have a simple solution: a) your install targets are only Debian derivatives, b) you can survive with only the Python packages provided by the OS, then don't use pip at all, install everything via apt and have a long lie-in.
Xenoamorphous 3y ago
And then there’s the issue of a lot of ML packages having a native component… I spent half a day yesterday trying to build a project in an M1 Mac. Not even using an x86 Linux Docker image helped as tensorflow complained it was compiled for AVX-512 and the CPU didn’t support it.
egberts1 3y ago
Evilution of virtual Python build environments: setuptools distutils pyenv poetry anaconda pip install --user pkg-name python -m venv venv source venv/bin/activate pip install -r requirements.txt python3.8 -m venv .venv38 python3.11 -m venv .venv311 conda create -n myproject python=3.9.4 conda activate myproject pytorch meanwhile, in C and C++, we have `make`.
- bheadmaster 3y ago
  > meanwhile, in C and C++, we have `make`. And `cmake`, and `conan`, and `autotools`, and `scons`, and `bazel`, and...
  - akvadrako 3y ago
    You left out `meson`, which is what I would say is gaining marketshare fastest in the Linux world. And it's written in python, so even C doesn't escape the python packaging ecosystem.
- SAI_Peregrinus 3y ago
  C & C++ also have Autotools, Ninja, CMake, Meson, Bazel, and several more obscure build systems.
TYPE_FASTER 3y ago
I've had success using pyenv on Mac and Linux. If I had to use Windows, I'd probably try pyenv-win [0]. I have run into issues trying to use packages that were available for one platform but not another, due to native code, etc. Most of the time I could find a pure Python alternative, but not always. This can lead to using containers, which adds complexity, which is a drawback because one of the advantages of Python to me is the simplicity (assuming you have something like pyenv). I've used Poetry in the past, but it added enough complexity/overhead that I probably won't again. I've had some success putting a line in a README for an internal tool that other devs can use to pip install from a Git repo. Again, assuming you have pyenv or the like, starting from a clean venv and pip install from Git seems to be pretty straightforward. [0] - https://github.com/pyenv-win/pyenv-win https://github.com/pyenv-win/pyenv-win
gpderetta 3y ago
The advice (python3.x -m venv + python -m pip install ) is good, but I think it is incomplete. What's the best practice for making the venv reproducible? pip freeze + pip install -r? What about upgrading or deleting a package? Also, what if I don't want to install pure development packages (black, mypy, pylint, doc generation packages etc.) in the deployment venv?
- JetSpiegel 3y ago
  pip-tools can create lockfiles, similar to other language package managers.
braza 3y ago
Even using python for almost a decade sometimes I feel the entire debacle around packaging and development environment it’s too sophisticated for a user base that grows every day with more beginners.

89 points by agos 3y ago

I like venvs and hate dealing with the system wide impact of anaconda (and the weird side projects you need for it to resolve dependencies within a single human lifetime) but if I see something that uses it, I just suck it up. It's annoying that my specific platform isn't supported, but unless I'm paying for some kind of support contract, I don't expect the devs to make things easy for me. They make what they need, and what apparently tons of other people also need, so I'm the one who will need to spend a few minutes getting everything sorted out. If you're on Windows you're going to have trouble with some packages. If you're on Linux, you're going to have trouble with others. MacOS and BSD aren't any different. From what I can tell, LTS Linux (Ubuntu, Fedora) is the most compatible, luckily. Yeah, I'd prefer to install everything in a venv. However, if I write software that works through venvs, someone with some obscure Python implementation is bound to report that the dependencies don't install in their system because of differences. They didn't choose that system just to be annoying, and neither did I. All of these solutions are valid answers to someone's problem. > Every time she digs, there is one more thing to learn, and for each thing, hundreds of ways to fail. Such is the life of someone trying to find workarounds for an upstream problem. "It works on this platform, not yours, but you can simulate that platform by..." implies that you're going to need to learn another platform or don't use the project until it's got support for your technology stack. There's plenty of Python that doesn't run on most Windows machines just like there are plenty of VBA macros that don't work on LibreOffice. Whether you solve that through workarounds, conversion systems, or installing a different platform you need to learn (in a VM or on bare metal), stuff is going to get messy. As for "just use venvs": to use venvs for various projects, you need to install cargo, clang/g++/msvc, sometimes Fortran, or even assemblers. All of these fail in spectacular fashion. Binary packages aren't a solution either, they'll just randomly fail to load (glibc mismatch, musl, or hell, "go binary loaded on Alpine by Python" just seems to fail for obscure Linux linker reasons). There is no easy answer that doesn't require some kind of additional tooling if you're trying to use big, unwieldy packages. This is the case for every programming language!

jwr 3y ago

I don't normally use Python. When I see something that I'd like to install, and the docs say "written in Python", my heart sinks. I know I'll have to find out how to install multiple dependencies in various packaging systems, all of which will be incompatible (but surely much better) than my system package manager. I find all of this ironic given that the main gripe people used to have against Common Lisp was "but how do I make a self-contained binary?". Somehow this was never required of Python.

bombolo 3y ago
For me if I can do apt install is good. No care which language it's written in.
- rgoulter 3y ago
  Sure, it's convenient if there's already a system package; in which case, yes, the language is irrelevant. However, if you want to run software which isn't available as a system package, then you'll need to know how to build/run the software.
  - roflyear 3y ago
    Isn't that true of many languages?
    - arp242 3y ago
      Kind of, but some languages are definitely easier to build than others (on average).
    - rgoulter 3y ago
      Surely "to run the program from source, you need to know how to build/run it" applies to every language. But I'm more comfortable working through problems I might encounter in languages I've worked with, than in languages I'm not familiar with. I'd say Python is notorious, since you're probably going to hear about at least six or seven ways people are able to run things; so if you're not familiar with it, you're likely to do something wrong. -- e.g. elsewhere in this thread, you've pointed out that stuff should be run from a venv; that's something you have to know.

heywhatupboys 3y ago

Python's package ecosystem is probably the worst in the world

artemonster 3y ago
Can you elaborate what this system lacks or what are the key properties that make it „the worst“? I know its a mess, but I cant pinpoint the reason why.
- kristofferR 3y ago
  That so many things doesn't run on the latest stable version is one of them. Python 3.11 was published in all the way back in October, and it's still not stable and well supported.
- linkdd 3y ago
  Is the fact that there are multiple package managers for the same package fornat/spec the reason why people say it's a mess? Genuine question: How is PyPI different from NPM, Maven, crates.io, etc..? Why "Python packaging is a mess" and not the others?
- throwawaaarrgh 3y ago
  The non-hierarchical, not-reused, reinvented package convention. The most popular package repository being incredibly hard to find one package you want among a sea of crap. The lack of a single standard solution to the most common problems. Frequent backwards compatibility issues. It's definitely not the worst. But it's hilarious how bad it is considering how popular it is and how long it's been around
j16sdiz 3y ago
try ruby.
- Alifatisk 3y ago
  What's wrong with bundler / gem?

WillAdams 3y ago

That people find this comic: https://xkcd.com/1987/ https://xkcd.com/1987/ to be quite reasonable is why. It kills me that the Python folks don't have a resource like to CTAN and an established naming convention so that things (mostly) work when installed, and when things don't help is just a "minimum (not) working example" away on tex.stackexchange.com

ncphil 3y ago

Anyone who rarely uses python, or who uses it all the time, should take the time to read this article and its companions in the series. Good, solid, advice that will make life easier. Just read it and try out the suggestions made. I was amazed at how close it came to my own workflow developed over many years since migrating from perl to python.

kkirsche 3y ago

I’m not convinced by the argument as it sounds like it’s tone implies that it’s written to people who aren’t aware of who their audience is. Sure, not all of these tools are right for all audiences, but that doesn’t mean that they aren’t good suggestions for wide swathes of the ecosystem. I think it’s important instead to think about your largest user groups and provide workflow ideas for those groups. For example, with Python we may have (this is a simplified list): * Web Development * Data Science * Embedded * Compilation / Transpilation (e.g., PyO3) Of these groups, I’d expect the tools mentioned to be reasonable for two or three of the groups, but less so in groups like data science where there is a higher concentration of non-development first specialties. There are always outliers, but I think choosing your target audience is important. On a team, I wouldn’t necessarily want to teach a person a random workflow if the team all uses one of these types of tools.

kristofferR 3y ago

So what is actually the solution? Let's say I want to run Stable Diffusion WEBUI (Python 3.10) and Whisper (Python 3.11). Last I tried to run both at the same time it was hell - when I downgraded Python for Stable Diffusion as per the instructions, Whisper stopped working.

0x008 3y ago
you can use pyenv to have multiple parallel python versions installed and you can choose per virtual environment which version you want to use.
- kristofferR 3y ago
  Pyenv doesn't seem to work on Windows, is the fork recommended? Or what about Pyenv on WSL2, will Cuda work well?
  - eurekin 3y ago
    CUDA works great in a WSL2 vm. Highly recommended. I found that just sound doesn't work (any soundcard operation isn't implemented in the WSL)
deleted 3y ago
[deleted]
zokier 3y ago
To me the solution is just use system python. If some packages do not work, then patch them if easy, or just avoid using if not. Docker as last-resort solution.
matthiaswh 3y ago
You want something that manages different versions of Python itself. pyenv is a popular option for doing this, but I'm partial to rtx which is largely language agnostic and also handles Python virtual environments in a very elegant way. https://github.com/jdxcode/rtx https://github.com/jdxcode/rtx
3y ago

ipsum2 3y ago

Use conda to set up new environments and switch between them, and use pip to install modules. It's really not as complicated as people make it seem. I haven't had any issues with this set up across Linux and MacOS, doing web / ML / data development for the last 3 years. It also avoids the pitfalls mentioned in the blog post.

gpderetta 3y ago
I'll provide a witty reply to this comment as soon as conda finishes resolving the dependencies for my projects. Any time now...
- suslik 3y ago
  Use mamba as a drop-in replacement, it's rather fast.
- Garcia98 3y ago
  Have you ever tried mamba[0]? It's much, much faster than conda for me. 0: https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba
- montebicyclelo 3y ago
  There is a faster solver for conda, that you can enable. [1] [1] A Faster Solver for Conda: Libmamba - https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community https://www.anaconda.com/blog/a-faster-conda-for-a-growing-c...
- peteradio 3y ago
  You are complaining about built-in coffee/smoke breaks? Whats wrong with you man!?
- ipsum2 3y ago
  This shouldn't happen if you use pip to install packages, not conda.
YetAnotherNick 3y ago

clarionbell 3y ago

I prefer python package management to whatever is happening in the node.js world.

agos 3y ago
what is happening in the node.js world? people typing npm install and then being done is somehow worse than the Python situation?
- fragmede 3y ago
  Node.js installs the libraries into the `node_modules` subfolder in the working directory of the project. Python installs the libraries in to a system dir (unless you're using venv or conda or whatever). This means that dependency mismatches are resolved on a per-project level with node.js but are on a system-wide basis for python. If `npm install` is failing, it's not unreasonable to just run `rm -rf node_modules` and then try again. Worst that happens is that project is still broken. Meanwhile, if I nuke `/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/*` on my mac to try and resolve a problem, I've broken everything else that uses python 3.11 on my system. The failure scenario with Python is when two or more projects need conflicting version of numpy or scipy or whatever. Then you start fighting with pyenv/poetry/venv/conda/docker to get a different environment to run it in. As noted, it works, eventually, but it's just more work. More work is just more work and we as developers hate doing busy work like that.

imgabe 3y ago

For me none of the things that are suppose to "relieve packaging pain" have ever done anything but cause more pain. Plain virtual environments with venv work fine. If you need a different version of python, download the source, build it and install with make altinstall, then use it to make your venv.

shipmaker 3y ago
Thank you for your post, I was looking for something like altinstall, I wasn't aware of it. I've used dnf to install python3.8 side by side with the default python3.6 on Almalinux and they seem to work fine, but I keep wondering if I haven't broken something.
- Chiron1991 3y ago
  As long as you're installing/updating Python via your distro's package manager you won't break anything. Most horror stories of borked system Pythons are caused by people uninstalling/updating packages from the system Python via pip or fiddling with their PATH. altinstall is only useful if you have to compile Python from source to get it onto your system.
  - shipmaker 3y ago
    Thanks!

BiteCode_dev 3y ago

Damn, the reddit shuts down really is bringing a lot more attention to my blog. I've got links on the front pages posted twice. Well, AMA, I suppose.

rcfox 3y ago

Title nitpick: Starting with "Why not..." sounds like a suggestion. I'd recommend "Why you should not..."

brylie 3y ago

I recently switched back to venv and pip with pip-tools with dependencies configured via pyproject.toml. I have a basic Makefile containing convenience command aliases with inter-dependencies. While not perfect, the venv/pip approach has simplified my development process, allowed conventional deployments to PaaS, and made things easier for new developers. I will gladly dispense with pip-tools and Makefile if Python had/has a more capable standard library CLI.

crabbone 3y ago
> pip-tools ... pyproject.toml ... Makefile This is a plate of cow dang with glass shards on steroids. I wouldn't even know where to begin to explain how bad this is... I mean, you already had an abomination in the form of pip-tools and pyproject.toml, and then you decided to pepper this trash with Make? Like, come on... you are with a straight face suggesting people treat toothache with a chainsaw...

linuxftw 3y ago

I think this article is rather long-winded and not super informative. The previous post [1] lays out what one should actually do, and I agree with most of it, at least from a developer perspective. 1: https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...

BiteCode_dev 3y ago
This post is the justification of the one you link to. It's not meant to be practical, but rather an answer to the same comments I get over and over again when I suggest the method you just posted. Even with it though, you can see in the thread people don't read it and completely ignore its rational :)
- linuxftw 3y ago
  I understand what you're trying to achieve here, but I think the content isn't very well organized. The people that have the chops to understand most of what you wrote won't need to read it, and those that don't won't come away with any clearer understanding. If you can trim the content to about 30% length, I think it would be better.
  - BiteCode_dev 3y ago
    Yes, it's too long, I should have made a series of 3 posts instead, each focused on one particular idea. One particular point "devs lie", really deserve its own article. Because we do lie a lot, and it's like a taboo to talk about it.
    - linuxftw 3y ago
      I don't think I'd characterize some of those things as lying. Setting up a development environment is no different than setting up other kinds of software: you obtain the software, and follow instructions to get it up and running. There's no substitute for this, unless you're setting up an environment for someone else (eg, a student), in which case, they don't need to know about many of the details.

mikepurvis 3y ago

I’m a fairly recent convert to poetry, but I definitely acknowledge the bootstrapping problem with it— it seems to work best if you have some non-Python means of acquiring poetry itself, whether that's a PPA, nixpkgs, or something else that's already "at hand".

crabbone 3y ago

> Among other things, it advised not to use homebrew, pyenv, anaconda, poetry, and other tools. Why are there so many so low quality programmers jumping on this problem? Why cannot they just step aside and leave this to people who actually understand what they are doing? OP has no clue what they are writing about. Anaconda is a different distribution of Python with a bunch of it's own packages, tools, processes... It definitely doesn't belong in the same group with homebrew, pyenv or Poetry. What are Python users supposed to do if they get a project exclusively distributed as an Anaconda package? The real list of things you shouldn't use to install Python packages is anything that's governed by PyPA -- it's all garbage. Unfortunately, this is impractical. The situation is in a way very similar to Linux and proprietary drivers. If you want Linux to run on your laptop, you will most likely end up needing proprietary drivers. And that's bad. And you have to compromise. So... setuptools? -- bad, shouldn't be used, but, realistically, you will end up using it anyways. pip? -- even worse, but it's very hard to avoid. In situations where you create a library, you'll probably have to also at some point face the need to provide it in Anaconda distribution, and you'll have to use conda too... which is also bad, but there's nothing you can do about it, and even if projects like mamba exist, it's still not going to help you because you must test your library with the tools the users are going to use on it, and it's a choice between not supporting a bunch of users and having to use crappy tools.

secondcoming 3y ago
Isn't setuptools deprecated for creating apt packages of python code?
- crabbone 3y ago
  The deprecation of setuptools is a joke. There's no replacement. What PyPA does is that they produce ever more bullshit wrappers for setuptools pretending they are doing something new. I think that by "apt packages" you mean DEB packages. As someone who has to package a lot of Python both for RHEL and for Debian-like... well... so far and in the observable future, setuptools isn't going anywhere. On the other hand, I could easily do without it, and sometimes do. But it's not because PyPA produced any replacement to it. It's because I simply grow frustrated with how bad of a tool setuptools is, and I just write my own. The claims PyPA makes about setuptools are because nobody wants to support that project anymore. It's awfully written at every level, from the minutia of formatting and up to the high-level design decisions. The problem is that everything else that PyPA came up with so far is a joke. The only reason I keep paying attention to it is because I'm being paid to fix problems created by using it. I'm also going to tell you a secret... well, of sorts. DEB, and RPM too are also pathetically bad... since you seem to have some experience with DEB -- to assemble this stuff you typically have to use a bunch of Make macros. Excruciatingly painful to debug. Virtually no documentation and very few people who could help you if you have a problem. And, when it comes to Python packaging, things also changed over time. Not so long ago I saw some write-up about how to package Python including virtual environment in the package. In truth, if I were a distro maintainer, and someone came to me with a package like that, I'd told them to try elsewhere. It's a sign of disrespect to the efforts of the distro maintainers to assemble a working and coherent system. But, guess who does stuff like that? -- big companies, like Amazon or Microsoft. Instead of spending effort to figure out how to play nice with the distro's packages, they distribute a huge blob that's not humanly possible to audit.

novaRom 3y ago

Always use containers! Any big project should be sandboxed, having completely isolated environment, dependencies, and you have all installation steps well described and reproducible. Containers in Linux are lightweight citizens and using them is absolutely necessary to avoid any dependency hell and broken or trashed system. Plus, you can simply distribute them.

jackdh 3y ago
From the article... >> You should really use docker > I think you missed the whole point.
dagw 3y ago
Always use containers! Even for short scripts I write? A unique container for every script? How do I go about calling one script in one container from a script in a different container. Containers in Linux are lightweight Much of the world isn't running Linux. Deploying a single project you have developed, targeting a single OS and CPU arch, onto a server that you have complete control over is the easiest case and not really where most problems show up. The hard problem is being able to email a script I wrote to a colleague and have it run on their machine.
bryancoxwell 3y ago
I think you’re right. I also think that’s a big problem for python as a language. For a lot of use cases just choosing a different language from the jump that allows you to easily compile a standalone executable can make life so, so much easier.

computerfriend 3y ago

For what it's worth, python -m venv venv source venv/bin/activate pip install -r requirements.txt has never let me down and doesn't abstract away anything.

BiteCode_dev 3y ago
That's pretty much the procedure the article link to in the intro, but with some additional subtleties on how to source and run python.
pera 3y ago
That might work for a subset of projects, but try that with anything that does for instance numerical analysis stuff, like machine learning. There is a reason why Conda is a requirement for so many Python projects.
roel_v 3y ago
In your example, one of the abstractions is 'what Python does it use'. Because I'm dealing with this very issue right now, distributing a simple Python tool to a Windows end user, and they're on heavily managed government machines and it turns out that their 'python' command invokes a python.exe somewhere on a network drive and for some reason the '-m venv venv' fails, which then causes all the rest to fail as well of course. Now I don't know yet if this network thing is really the root cause, I've just received some error logs that showed this this morning, so I'm on another round of trying to remote debug this, via email, with someone who is (while patient and willing) not a Python programmer. Python packaging is such a ridiculous PITA, it's laughable if it wasn't so sad. I tried nuitka as suggested upthread, and send my customer a single binary to try, let's see if it works.
- deleted 3y ago
  [deleted]
- aragilar 3y ago
  It's probably Python 2, which won't have venv.
3y ago

lkrubner 3y ago

I recall the negative reaction I got, on Hacker News and on Twitter, back in 2018, when I made the point that the Python community was one of the main groups driving the adoption of Docker: "Package management in Python is so hopelessly broken that the Python community is eager to use Docker everywhere, as the only way to save the situation." In response, people laughed: "No one is using Docker for package management! Python has an abundance of good tools for that!" And yet it remains true, if package and dependency management in Python was seamless, then Docker would have had fewer advocates back in the early days. And one of the criticisms of Docker remains that it is sometimes used as a Band-Aid for flaws in some computer programming language, flaws that need a more fundamental solution.

BiteCode_dev 3y ago
To be honest, you still should use a venv in a docker image.
- crabbone 3y ago
  Insert the "if those people (behind the window with a poster) could read" meme. Seriously, if Python developers could produce a coherent thought, we wouldn't be here. They aren't bothered by a contradiction you pointed out. They still push for Docker adoption, even though it's not helping them, because they "want to believe".
macNchz 3y ago
This is likely true for me–I’m a fan of Docker and a longtime Python developer. Docker rose in popularity at around the same time that many applications were in between Python 2 and 3, and putting apps in containers was a bit of a magic wand to wave away various issues, some of which still exist and some that were worse during that transition. I still like Docker regardless of Python–I wouldn’t want to go back to development using a vm-per-app, or dealing with mismatched db/cache/queue/system package versions between projects or across different developers machines.
deepsquirrelnet 3y ago
I’ve been writing python code for about 19 years, and I remember yours or similar arguments about docker. I completely agree. Docker has more or less saved python from its own packaging issues, especially in the cloud. On local machines, I really found the most comical show of python distribution wreckage in RedHat/CentOS distributions. Back in 2015 or so, yum was tied to the system version of python. Heaven help you if you didn’t realize it and went mucking around on it (like me).

cpfohl 3y ago

This problem is not unique to Python, it’s just the proliferation of incompatible tools that makes Python quite so annoying to use. I’ve found that none of this is really an issue if you provide a container as a dev environment. Before someone accuses me of “yet another standard”ing this, it’s my standard across all my programming environments. It’s not a silver bullet, especially with Apple silicon Docker can be a massive pain in the butt, but it works for me. Inside the container I can be guaranteed that only a single Python version is installed (correctly) and packages are installed in a standard way from a package file. Adding a package can be scripted so that it gets executed inside the container and adds it to you packages file. The other massive benefit is that your local machine needs no locally installed tools other than docker (or podman, or whatever). this works for me and my team it’s not perfect, but it is a complex problem.

crabbone 3y ago
It's not "this" problem. It's many problems together. Some of them, while maybe not unique to Python are completely preventable. Where the right thing to do was obvious, but Python core dev decided to cut corners, or do nothing etc. So, we need to be fair and put the blame where it belongs. Python packaging format didn't just "happen", it was designed... by a bunch of half-wits, but that's exactly the problem! Python import system didn't "just happen", it was designed, by mostly the same kind of people -- and, again, that's the problem! If anyone wanted to put a good effort and thinking into designing packaging format and the import system, they'd realized that they need to deal with multiple versions of the same package somehow, they would've realized that they need to distinguish between binary distribution and source / documentation / data packages, that they need a reliable tool for installing those packages. But nothing of the kind was done. Every "development" in this area was incrementally adding more technical debt. Is technical debt a new thing? -- surely not. Is it preventable? -- well, not so much, but being cognizant of it one could establish certain practices to mitigate the adverse effects. And this, this particular thing, has never been done in Python. Is this a common failure? -- well, again yes, but this doesn't vindicate Python core developers, they are just as guilty as a bunch of other people who suck at programming.

TheAceOfHearts 3y ago

For a language purported to be user friendly, Python sure does have an extremely unfriendly package management ecosystem. I'm sure a lot of it comes down to familiarity though. With nodejs I'll use asdf to switch and manage runtime versions, and npm to install packages per project. Dealing with transpilers and bundlers is terrible experience though. The best programming language that I know to get everything right with the out of box experience is Rust. It has a tool that allows me to switch Rust compiler versions and cargo Just Works.

elzbardico 3y ago

Half the problems in this long article can be avoided by not using Windows. I wouldn't try developing a .NET Forms application with Sublime Text on Gentoo, would I? Open Source is done by people solving their own problems and by their own volition, sharing their solutions with the community. Some of those projects are useful enough for people to become commercial or semi-commercial endeavors. But yet, resources are limited, and compatibility with Windows will only be there if more people care about it than they care for other features. Most Linux distributions have no licensing costs, are highly compatible with most basically the same hardware Windows runs on, and even more, in very inexpensive hardware like raspberries pi where Windows doesn't even run. So, given that resources for development are not infinite, I think that we should be glad that Linux and other Unix-like environments are the baseline target for compatibility for python packages instead of Windows. Also, most Python software that is of interest to non-programmers like Jupyter Notebooks, PyTorch and Keras is still of a technical nature. It is not like people are installing mail clients or Word Processors written in Python with pip install. Even if those projects are being used by non-professional programmers themselves, the user must learn some programming to use them. It is reasonable that someone wanting to use such sophisticated packages will have to devote some time to learn how to properly install them. Again, someone could argue this is unacceptable, but this comes from the reality that making this easier requires resources, programmer time, testing and access to environments for said testing. Most open source project don't have the luxury of access to vast farms of devices with hundreds of different combinations of hardware, devices and software as a company like Microsoft does. But on the other hand, they are not asking you thousands of dollars for a Microsoft Subscription, neither they receive a tax the moment you bought your machine with their operating system installed.

andybak 3y ago

> "There are usually two kinds of coders giving advice. A fresh one that has no idea how complex things really are yet. Or an experienced one, that has forgotten it." This line is gold.

montebicyclelo 3y ago

The disadvantage of this approach is that you can't easily switch major Python versions per project. I use miniconda [1], and I'll create a conda env per project, with the newest version of Python that is compatible with the dependencies. Also, conda doesn't just install Python packages, it can also install cuda drivers (e.g. for GPU accelerated pytorch/tensorflow [2]), it can also install ffmpeg, etc. The complaint in the article is true, that you do generally need to do a mixture of `conda install` and `pip install` depending on whether the package is in a conda package repository or not; but I've not found this too bad in practice - I look up whether the package is in a conda repo, and install from pip if it isn't. Also it would probably be better for conda to be automatically activated for a given project / folder, rather than needing manual activation via `conda activate <name>`, but you get used to it. [1] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html [2] https://pytorch.org/ https://pytorch.org/ <- conda is listed in pytorch installation guide

crabbone 3y ago
1. Anaconda tooling and PyPI tooling are just different things, you cannot substitute one for the other, so whether you had any success with Anaconda or not is really irrelevant to OPs problem. 2. How much more easier to you want this to be: python3.8 -m venv .venv38 python3.11 -m venv .venv311 This, at least for me, creates two environments with two different minor versions of Python. NB. I think, you meant "minor" version -- why would you want to switch between Python 2.X and Python 3.X today? Or did you actually meant that?
- montebicyclelo 3y ago
  1. Not sure what you mean? "is irrelevant to OPs problem" -> op advises to not use conda as a solution for Python packaging 2. This is easier than what you propose, because conda will install the python version for the env on the fly (including the patch version): conda create -n myproject python=3.9.4 conda activate myproject NB. if we're being pedantic, conda and anaconda are different Yep, should be minor, but I chose to use the same language as the article [1] [1] https://www.bitecode.dev/i/110377053/dont-install-the-latest-major-version-of-python https://www.bitecode.dev/i/110377053/dont-install-the-latest...

gtirloni 3y ago

What worries me is that any solution to this problem could a Python 2->3 decade-long effort again.

jjgreen 3y ago

Very true. One (quite narrow) use-case does have a simple solution: a) your install targets are only Debian derivatives, b) you can survive with only the Python packages provided by the OS, then don't use pip at all, install everything via apt and have a long lie-in.

Xenoamorphous 3y ago

And then there’s the issue of a lot of ML packages having a native component… I spent half a day yesterday trying to build a project in an M1 Mac. Not even using an x86 Linux Docker image helped as tensorflow complained it was compiled for AVX-512 and the CPU didn’t support it.

egberts1 3y ago

Evilution of virtual Python build environments: setuptools distutils pyenv poetry anaconda pip install --user pkg-name python -m venv venv source venv/bin/activate pip install -r requirements.txt python3.8 -m venv .venv38 python3.11 -m venv .venv311 conda create -n myproject python=3.9.4 conda activate myproject pytorch meanwhile, in C and C++, we have `make`.

bheadmaster 3y ago
> meanwhile, in C and C++, we have `make`. And `cmake`, and `conan`, and `autotools`, and `scons`, and `bazel`, and...
- akvadrako 3y ago
  You left out `meson`, which is what I would say is gaining marketshare fastest in the Linux world. And it's written in python, so even C doesn't escape the python packaging ecosystem.
SAI_Peregrinus 3y ago
C & C++ also have Autotools, Ninja, CMake, Meson, Bazel, and several more obscure build systems.

TYPE_FASTER 3y ago

I've had success using pyenv on Mac and Linux. If I had to use Windows, I'd probably try pyenv-win [0]. I have run into issues trying to use packages that were available for one platform but not another, due to native code, etc. Most of the time I could find a pure Python alternative, but not always. This can lead to using containers, which adds complexity, which is a drawback because one of the advantages of Python to me is the simplicity (assuming you have something like pyenv). I've used Poetry in the past, but it added enough complexity/overhead that I probably won't again. I've had some success putting a line in a README for an internal tool that other devs can use to pip install from a Git repo. Again, assuming you have pyenv or the like, starting from a clean venv and pip install from Git seems to be pretty straightforward. [0] - https://github.com/pyenv-win/pyenv-win https://github.com/pyenv-win/pyenv-win

gpderetta 3y ago

The advice (python3.x -m venv + python -m pip install ) is good, but I think it is incomplete. What's the best practice for making the venv reproducible? pip freeze + pip install -r? What about upgrading or deleting a package? Also, what if I don't want to install pure development packages (black, mypy, pylint, doc generation packages etc.) in the deployment venv?

JetSpiegel 3y ago
pip-tools can create lockfiles, similar to other language package managers.

braza 3y ago

Even using python for almost a decade sometimes I feel the entire debacle around packaging and development environment it’s too sophisticated for a user base that grows every day with more beginners.