42 ms·

Why not tell people to “simply” use pyenv, poetry or anaconda

183 points by sametmax 4y ago

throwawaaarrgh 4y ago
Newbs add abstractions to avoid complexity. Veterans avoid complexity by removing abstractions.
- deleted 4y ago
  [deleted]
- regularjack 4y ago
  My personal experience is the complete opposite of this.
- nonethewiser 4y ago
  Back to assembly I guess
  - bick_nyers 4y ago
    That is way too high of an abstraction. Use breadboards.
- deleted 4y ago
  [deleted]
dsr_ 4y ago
In general, don't tell people to "simply" or "just" use anything unless you're willing to provide the precise config that they need or otherwise hand-hold them through the starting phase. Nothing in computing is "simply".
- livelielife 4y ago
  except that side of computing for which users are really "input" to be turned into "output" which is the industry for which users are really the product. user's data in, profits out.
- inconceivable 4y ago
  "trivial" also fits into this category. i've been keyboard jockeying for 20 years and every time i see that word i groan a little, because it's both a shitty flex and probably untrue.
  - tetha 4y ago
    My documentation skills have improved when I started to be critical about "obvious", or "trivial". And now I routinely end up writing "obviously", stop, and end up with 3 pages of clarification.
  - m463 4y ago
    Actually the best use of "trivial" is in the negative, like... "Wellllll, that's non-trivial."
- kjkjadksj 4y ago
  Check then for conda, which only asks you to copy and paste a shell command they provide for you and eventually say “yes.”
- mparnisari 4y ago
  THIIIIS i hate it when people say "oh just do this"!! "JUST" implies it will take me 3 seconds, where in 90% of the cases it takes me 3 hours!!
- nonbirithm 4y ago
  I agree with you, but the difference between being able to tell someone "if you need to install some dependencies in Rust, just use Cargo" and show them the documentation, and whatever the equivalent documentation is in Python are worlds apart. I think the author's argument would be weaker if he focused on a language other than Python that doesn't have a hellscape of a package ecosystem. And I remember custom build scripts with Cargo being a major pain at the time I used it, but the Cargo developers were able to sidestep major fundamental problems because they knew what didn't work with packaging in the decades before and managed to think about the design carefully enough from the start. Rust is different because the choice of package manager is "simply" Cargo, not necessarily that Cargo itself is trivial to use.
- theptip 4y ago
  I think this applies in some cases, but disagree here. “Just use poetry and pyenv” is my line, and I stand by it because these tools are well documented, and work well for most use-cases. It’s contrasted with a bunch of other more complex options, in tedious yak shave conversations. Often in threads where many people are complaining about how confusing the multitude of options are. It’s not confusing if you short-circuit the conversation; if you need advice, don’t go down the rabbit hole, just use poetry and pyenv on MacOS, you will make your life easier than the alternatives. There are plenty of docs that will give the hand-holding that you are looking for, so nice you have made the decision. If you know what you are doing there are arguments for other options, and data scientists have different tool chains. But I think we make it harder for new engineers by having five different ways of doing it, each with people arguing that actually their way is better.
2h 4y ago
100% agree. I use a programming language to get stuff done. and if the day ever comes that someone wants me to show them how I do what I do, I dont want to start that conversation with a sigh "well...", I want to start it with a "OK cool..." and all these Python "tools on top of tools" make me sad. Personally I like Go. If someone wants to build my stuff, then I just say go here http://go.dev/dl http://go.dev/dl and download Go, then set location to where the code is, and enter "go build". thats it. All languages should be that easy.
- paulddraper 4y ago
  Wait did you just call Go versioning and dependency management "easy"??
  - 2h 4y ago
    I dont have trouble with it. Link to my code is in my bio, wheres yours? Unless you actually write Go code, I dont think you have a leg to stand on here.
  - cure 4y ago
    Compared to the dumpster fire aka Python packaging, Go versioning is easy and predictable indeed.
    - silverwind 4y ago
      Until you hit the obscure rules around v2+ go modules.
      - 2h 4y ago
        I have been coding in Go for years, even professionally sometimes, and I have never had to use "v2" in any of my code. granted my stuff is not popular, but its really just a way for popular repos to handle big changes. Once I got past v1.9.9, I just changed to v1.10.0.
        4y ago
jakewins 4y ago
Are there any efforts akin to deno for python? A “burn all the packaging down and start over” path? It’s so thoroughly broken, every day a dev on some team gets their poetry env entangled with some system installed python, or numpy suddenly decides all the CI builds will now compile it from scratch on every build, or.. Today it was segfaults on poetry version X on the M1 Mac’s, that went away in version Y but of course version Y broke pandas for the windows devs..
- LanternLight83 4y ago
  Guix (and maybe nix?) insists on being the "one package manager to rule them all", so Python packages and Rust crates are all re-packaged (sometimes with automated importers). Well I do still occasionally need to package something myself (or cheat pip/requirements.txt it), it definitely covers this "burn it down" philosophy and keeps environments isolated and reproducible.
- bayesian_horse 4y ago
  I have never used poetry... Mostly I just use conda or plain python envs. Never had those problems like you mention. When using anaconda in the stable channel you'll get a straight-forward "conservative" distribution of all the data science/numerical packages (and more). That's why people often say "just use anaconda", it really is quite simple as long as you don't mess up your stable environments with exotic packages you just want to try out. And it works well on Windows. No idea about Mac and don't care. Python doesn't need "something like deno". All programming languages need package management.
  - nonethewiser 4y ago
    How many people work on these projects managed by conda? > And it works well on Windows. No idea about Mac and don't care. Package management is a cross platform problem. You may not need it but that doesn’t make the current solution good.
    - bayesian_horse 4y ago
bayesian_horse 4y ago
In my personal experience, I'll take Python's packaging hell over nuget or npm any day. And it's often less about the package manager and more about the ecosystem: Can you find what you need? Is what you need stable enough? Does it break every few months with new versions of the runtime, either because of actual incompatibility (npm/node) or versioning shenanigans (nuget)?
- inferiorhuman 4y ago
  Let's back up a moment. npm at least works on *BSD. Anaconda is just as toxic as Electron in that regard.
  - bayesian_horse 4y ago
    Given the target audience of Anaconda, I can see how BSD compatibility doesn't matter to them. I guess it would take some major lifting because much of what they do is wrangling the compiling environment. Supporting Windows is hard enough. That's not "being toxic", you just can't be everything to all people.
    - inferiorhuman 4y ago
      Nah it's toxic. It's not just that Anaconda won't provide binaries, it's that you can't even build a project that uses Anaconda on not-Linux/Windows. In terms of why you shouldn't tell someone to simply use Anaconda, that's pretty high up there. As tedious as javascript package management can be, python has consistently given me more trouble.
      - bayesian_horse 4y ago
        I think that's an "Am I the Asshole" situation, and you are getting it wrong. BSD is niche, especially in scientific computing. Saying anaconda is worthless because its maintainers don't expend a ton of effort on making your particular niche more convenient is the actual toxicity...
NuSkooler 4y ago
Why not "simply" admit that Python and it's ecosystem while broad, are an absolute clusterfuck?
- whalesalad 4y ago
  Because that’s not at all the case and your rhetoric is what perpetuates the false image.
  - stathibus 4y ago
    Yeah it totally is, and people who pretend it's not are the problem. Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. And I'm awfully tired of folks in the "python community" blaming the victims of the mess they made.
    - nonethewiser 4y ago
      > Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. This approach seems insane but correct.
      - BiteCode_dev 4y ago
        No, use the procedure described in the previous article. It will be less hard than using docker.
  - JohnFen 4y ago
    As a user, it kinda is. My recent example: last week, I was installing a new program that uses some python scripts during its operation. I could not get those scripts to run. I knew it was because of the usual python problem of the scripts not matching the version of python that was executing them, but figuring out how to fix that took me a full day. Just to make Python work. Not even as a dev. Wearing my user hat, this is a clusterfuck. There is no other language that I am exposed to that presents this sort of problem, and this is a very common issue with Python. This is why I start to get sweaty any time that I'm using software that involves Python. It turns into a crapshoot and half the time, it's going to cost me a lot of time and stress. > perpetuates the false image. You can believe it's a false image if you like, but there are a whole lot of end user experiences that indicate it's very real.
jooz 4y ago
Ive heard that 'venv' are very problematic, but honestly, Ive never had a problem. And I used them daily. I understand that it can not be enough on some cases... that don't concern me. I would recommend to 'python -m venv' and thats all.
- bobx11 4y ago
  This is also my setup. It has the added benefit of being already included on every python install already so there is nothing extra to use.
- hobs 4y ago
  Same no problem, different solution - I use pycharm extensively and it manages my venvs 99% of the time with no issues at all, the only time it took me a minute of head scratching was realizing I needed to install a new system python to make a venv with it, but that would be clear if you were doing it via the shell approach you are using as well.
- Groxx 4y ago
  I've heard a lot (a lot) of complaints that are wildly misattributed to venv (like "version X of Y broke my project"), but I've essentially never heard of issues with venv itself. Aside from needing to know to use it. Which is certainly a problem. But python blessing a single venv-system might be worse in the long run...?
- abrichr 4y ago
  Agreed, never had a problem with this approach. The only limitation I've encountered is when moving the environment or renaming one of the parent directories. In which case it's easy to create a new one: # optional: freeze the environment if you don't already have a requirements.txt source .venv/bin/activate pip freeze > requirements.txt deactivate # remove the old environment rm -rf .venv # create a new one python3.10 -m venv .venv # activate it source .venv/bin/activate # install the requirements pip install -r requirements.txt
  - nicoburns 4y ago
doublepg23 4y ago
I think I mostly wrapped my head around pyenv and used Anaconda the other day. It was quite the pain, to setup and then it seemingly mangled my fish and bash configs causing a noticeable delay every start up. Not something I was hoping for just for hacking around on some AI project. Disclaimer: I was using Fedora which has Python 3.11, using Fish which is clearly non-standard and I’m a sysadmin not a Python dev.
- nightfly 4y ago
  You don't have to activate venvs, you can just refer to the paths to pip and python inside of them as needed
  - ElectricalUnion 4y ago
    Second this, just run python/whatever-binary-you-need inside a venv: `${my-venv-path}/bin/python` `${my-venv-path}/bin/${whatever-binary-you-need}` `%my-venv-path%\Scripts\%whatever-binary-you-need%` (because Windows...)
rektide 4y ago
Having encountered poetry recently for the first time, it was "simply" hell. I just wanted to use a single file python project, https://github.com/rumpelsepp/oscclip https://github.com/rumpelsepp/oscclip I spent about three hours trying to figure out how to setup python keyrings to work, to let me just get started using poetry. On a system I was ssh'ed I to. Gnome-keyring-daemom was up. I spent a while adding random pam rules suggested by archwiki in to inject more gnome-daemon stuff in my envs. Random gnome-keyring-unlock scripts, which quickly start talking about Clevis and tpm and fido 2-factor. Wading through hundreds of responses about Seahorse, a gui tool unsuitable for ssh. Many long miserable sad stories. In the end I stumbled upon someone who suggested just nulling out & turning off keyring with some config to make it have a null provider. After this the poetry project just worked. The tiny handful of deps this project has were already installed on my system, but poetry was also a task runner, instrumental for the usage of this single-file script. There's been so many years of churn in the python world of tools. A fractal nesting doll of virtual-env, mkvirtualenv, & various offshoots. I hope some day there is a mature reasonable option folks generally find agreeable. Poetry eventually worked for me, but what a miserable gauntlet I had to walk, and the cries of so many who'd walked the path & utterly failed echoed out at me at every step.
- whalesalad 4y ago
  Yikes. One of my favorite features from pip is how easily you can install from a git repo, or even the absolute url to the master.zip
- mixmastamyk 4y ago
  Yeah, that's the problem with all these "helpful" posts. You don't need poetry for that. My single CLI packages still use a setup.py that I haven't touched in five years and was simple enough to write.
- deleted 4y ago
  [deleted]
- AndyKluger 4y ago
  FWIW, for the case of installing Python packages as runnable tools (as opposed to importable libraries), pipx is solid. For example, you can install oscclip to a temp dir with a dedicated venv and run it immediately: $ pipx run --spec 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help Or install it more permanently: $ pipx install 'oscclip @ git+https://github.com/rumpelsepp/oscclip' My own Zsh frontend for managing Python venvs and deps, zpy, makes use of pip-tools to accomplish the same, with its pipx clone, pipz: $ pipz runpkg 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help $ pipz install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'
trey-jones 4y ago
> You should really use docker >> I think you missed the point. Maybe I did, but I've been using Docker as version management for pretty much every technology I employ for five or six years. Prior to that I sparsely used things like rbenv and virtualenv and I actually thought it was super dangerous and unreliable. Maybe it's gotten better in recent years, and certainly people who write python and ruby every day are going to know more about this than I do. I don't install anything on my computer if I can just use Docker for it. OK, I do have go:latest, but I use docker images for various projects that might be on any version of go from 1.8 to 1.20. Your website still runs on PHP5.3? I can help you (I won't, but I could totally run it locally!). Reasons I like docker better: 1. Any scripts or configs can explicitly refer to the version number. No guessing or assuming. 2. Our whole team uses the same version. 3. Only one dependency: docker. Granted I'm more of a sysadmin than a developer and I'm sure that biases apply.
- whstl 4y ago
  Suggesting Docker in the context of Python dependencies is still missing the point, because you still need proper dependency management in case you have to rebuild your image for some reason. If you have a Docker image that builds with "pip install whatever" and this library gets updated with a breaking change, you won't be able to rebuild the image without changing the dependency or the code itself, for example.
melody_calling 4y ago
If you're just using python as a local scripting language, and not pushing production code, the other option is to simply not bother with any of this. When there's a new python version I'm interested in, I install it via Homebrew and update my zshrc to clobber everything else via $PATH. All my scripts and tools are broken? Just reinstall the packages globally. Whatever. Since the big 3.x transition, it's pretty rare for forwards-compatibility to break (IME), and if something does, I can just try running prior python3x binaries until I find the last version that worked. It's hideous, but honestly the least stressful way I've found to date.
- kjkjadksj 4y ago
  Or you can just do everything in conda and maintain compatibility forever.
tipsytoad 4y ago
Aside: I usually use direnv to activate the venv (or poetry) when entering a dictionary https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19fba6aeb0 https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19f...
davb 4y ago
Python runtime deployment is a major pain point for us (CS department at a university). On the most tightly managed lab machines, which are all in lockstep on a fixed configuration (latest Ubuntu LTS with a updated image pushed annually), we can provide a consistent Python setup (e.g. Python 3.10 and a fixed set of C-based modules like psycopg2). However, our staff and PhD desktops and laptops are more diverse - with the OS often only being upgraded when the distro is going out of support, they could be running n, n-1 or n-2. That, most likely, means three different Python versions. We could use pyenv to let people install their own preferred version. Installing with pyenv requires building from source (slow on some of our oldest machines). This also means installing the Python build deps, which is fine for our departmental machines but not possible on the HPC cluster (operated by a different business unit) or the Physics shared servers. It's also less than ideal for our servers where students deploy their projects (where we want to minimise the packages installed, like the build-essentials meta package). It's also a massive stumbling block for less experienced students with their own laptops which could be running any distro, of any age. Many CS101 or engineering/business/humanities students taking a programming class, who have never programmed before, would really struggle. So, classes might tend towards teaching lowest common denominator Python (i.e. the oldest conceivable version a student might have installed on their machine). Sure, we have in-person and remote lab machines students can use - but it's not always convenient (especially for the data science / ML students running Jupyter notebooks with their own GPU). There are workarounds, but they all have serious downsides. Compared with Node.js and Go, where users can just download the appropriate package and unzip/untar the runtime or compiler version of their choice, deploying the Python runtime has enormous friction (especially for less experienced users). This has the bonus of simplifying deployments elsewhere in our infrastructure (CI/CD, containers, etc). And while we all complain about node_modules, Python venvs not being trivially relocatable is another huge frustration. We've used Anaconda, but that comes with its own issues (most recently, finding that it ships with its own gio/gvfs binaries and libraries which fail to mount our CIFS DFS shares - causing confusion for users running `gio mount` from within a conda environment).
tibbon 4y ago
Python as a language I find pretty nice. What I don't find is their environment and packaging system compared to something like Rust. "There should be one-- and preferably only one --obvious way to do it.", unless it is how to setup your environment. I only use Python every few months, and it is always a struggle. In comparison, "cargo build" works 98% of the time just after "git checkout"
- belval 4y ago
  Yes, but Python really is in a tough spot when it comes to dependencies because a lot of it are compiled in a lot of different languages using libraries that may or may not be available on your OS. That's the real issue. If you install python packages honestly anything works, it's when you have the ML stack of PyTorch (needs cuda, cudnn, needs to be compiled to match your OS version) + custom CUDA operators (also need to be compiled) with something like a mariadb connection (needs the mariadb OS library). conda solves it by packaging EVERYTHING, giving you atrocious 30GB environments, pip doesn't solve it at all and none of the challengers really have much to offer (in my opinion).
  - blactuary 4y ago
    That's Anaconda that packages everything, not conda itself. No one should really use Anaconda imo, instead use Miniconda which only installs the bare minimum needed for conda to work in a base environment, and then you create environments for each project.
aldanor 4y ago
Unfortunately or not, in some fields conda is the only sane choice because it can manage non-Python binary dependencies that Python packages may depend on. Some of those dependencies may be huge C libraries that are a pain to build, like HDF5, so if you're not using conda you'll be relying on your OS's package manager to serve your particular venv's needs - we all know what usually happens next.
- zackees 4y ago
  [dead]
not_enoch_wise 4y ago
One does not simply tell people to use pyenv, poetry or anaconda...
deleted 4y ago
[deleted]
DemocracyFTW2 4y ago
IMO to fix these issues the first thing to do is to write an alternative to Python's `import`, more like a function call that works more similar to NodeJS' `require()`. That should start life in userland and only later become part of the language. How to transition a bazillion packages though I do not know.
- BiteCode_dev 4y ago
  I have an article planned solely on imports. It saddens me to say, but imports are indeed a complicated topic on Python. The PYTHONPATH is not something people know about, and once they do, it's not intuitive. The weird handling of namespaces or relative paths doesn't help.
BiteCode_dev 4y ago
Author here. I'm late to the party but AMA :)
- deleted 4y ago
  [deleted]
- oconnor663 4y ago
  How do you see this advice changing in 3-5 years?
  - BiteCode_dev 4y ago
    This procedure was valid 5 years ago, and all the alternatives have changed a lot in the same time period. So if I had to chose one way for the next decade, I would bet on this. But I will not pretend we can be sure about anything on this matter. In fact, my hope is that we can improve the process tremendously by creating an unified way of bootstrapping Python for all OS. If this ever happens, the procedure would become mostly obsolete. At the end of the week I will write about this, and the various experiments the community have done so far in that regard.
- vqbd 4y ago
  There's a version of pyenv for windows called pyenv-win. The cli installs as `pyenv` and the commands are the same. So not game over imo. Also, [even if it did], it installs binaries and doesn't compile.
  - BiteCode_dev 4y ago
    You are not telling them to use pyenv, you are telling them to use a totally different tool that happen to have a similar name and provide a similar interface. You have already lost.
tpoacher 4y ago
I often find the reason for all this hell is, ironically, an effort to help people who dont know "how computers work", by offering "useful automations". And then these automations clash and fail because of the complexity involved. If you "simply" (yes yes I know) download the python version you want directly and compile/install in a local folder, and use that with venv as a way to manage individual project dependencies, all problems go away.
- jakewins 4y ago
  Until you need to distribute that project as a library, or you’ve shipped it as an app and now you need all users to upgrade library X or add library Y. Python packaging is fine for small local dev; problems arise in distribution, rollouts of upgrades and ensuring your apps work in the zoo of local setups your users have
kjkjadksj 4y ago
These comments in this thread were a bit surprising to me. People really like to make things hard on themselves not doing a little do diligence. Yes, I still say to simply use conda. It spells out exactly what is getting installed in the environment, and uses a separate python installation for each env than the system. If you don’t trust it just type which python. I never get these headaches people seem to have, since conda is easy and well documented and supported.
zajio1am 4y ago
Perhaps people should accept that these are more developer tools (like git) and not end-user distribution tools and just use regular distribution packages (rpm / deb) for that.
pmarreck 4y ago
"simply" use Nix, which solves this problem for every language.
- n8henrie 4y ago
  I wish this was a "simply" answer! Been getting more into nix lately and really enjoying it, but converting python to nix has been a challenge. It seems like the existing and recommended suggestions are either deprecated / unmaintained or still alpha at best (pip2nix, mach-nix, dream2nix). https://discourse.nixos.org/t/basic-flake-run-existing-pytho https://discourse.nixos.org/t/basic-flake-run-existing-pytho...
  - pmarreck 4y ago
    Well one way of going about it is a hybrid solution. So you set up your python dev directory with a nix file, but then you use pyenv or whatever to encapsulate everything else inside it using python tooling. So it's not strictly Nix-all-the-way-down, but at least as far as being in the directory is concerned, the "right" python version and libraries are available.
INTPenis 4y ago
I don't use any of those, I use direnv with standard python3 -m venv module.
zaptheimpaler 4y ago
Python packaging is sooo fun it has given me permanent brain damage and PTSD. Now I have a docker devbox with all my language toolchains and fun unix tools installed. I could install 3 different versions of cuda, 8 different pyenv pythons all sharing parts (but not all) of each others modules with torch compiled for a 4th different version of cuda that is NOT installed, then replace the core system python with a duck. pipx has somehow installed a version of borg backup that depends on a secret hidden 9th python. Then I will simply `docker rm` and `docker compose up -d` and I'm back. Yesterday I ran a random academic paper ML model in python in 5 minutes on my docker machine. HAHAHAHAHAÀÄÄÄÄÄ i am invincible!!!!!!
- jamesralph8555 4y ago
  Docker is the only way to do ML. One thing I didn’t think to do before for a while was make separate docker files for each project so all of the deps are installed automatically and that has since helped tremendously.
Havoc 4y ago
The variety of solutions sure isn't helping. I'm just sticking all dev projects into a separate LXC and calling it a day. Don't want to deal with all the various separation models the various languages and package managers cooked up
CMCDragonkai 4y ago
I dealt with this years ago by only using Nix to do python Dev. Worked great for the entire ML stack. Had to do a bunch of packaging for nixpkgs early on though.
akasakahakada 4y ago
Any of these is complex as hell. I wish everything is as simple as downloading exe file on my Windows Desktop and double click, done.
phendrenad2 4y ago
The problem is, dependency management isn't a solved problem. I was nodding my head in agreement at the part where the author mentions that pyenv compiles Python from source when it installs. If you try to figure out the reason behind this, it becomes obvious that the only one that makes any sense at all is that distributing binaries, in a secure way, is hard for open-source projects to manage. After all, bandwidth costs money, someone has to pay for the build server, someone has to pay for the data transfer bandwidth. And, do you trust that person to not be a rogue actor? It's easier to pull down the source, on the end user's machine, cross your fingers for good luck, and compile it. We also haven't figured out how to incentivize maintaining backward-compatibility. 99.9% of the time when some library updates, and stops working with language version X, it's using some hot new feature of the language. Usually just because the library author thought it would be cool or more elegant. The entire software world needs to update now, because someone left-padded us for elegance.

183 points by sametmax 4y ago

throwawaaarrgh 4y ago

Newbs add abstractions to avoid complexity. Veterans avoid complexity by removing abstractions.

deleted 4y ago
[deleted]
regularjack 4y ago
My personal experience is the complete opposite of this.
nonethewiser 4y ago
Back to assembly I guess
- bick_nyers 4y ago
  That is way too high of an abstraction. Use breadboards.
deleted 4y ago
[deleted]

dsr_ 4y ago

In general, don't tell people to "simply" or "just" use anything unless you're willing to provide the precise config that they need or otherwise hand-hold them through the starting phase. Nothing in computing is "simply".

livelielife 4y ago
except that side of computing for which users are really "input" to be turned into "output" which is the industry for which users are really the product. user's data in, profits out.
inconceivable 4y ago
"trivial" also fits into this category. i've been keyboard jockeying for 20 years and every time i see that word i groan a little, because it's both a shitty flex and probably untrue.
- tetha 4y ago
  My documentation skills have improved when I started to be critical about "obvious", or "trivial". And now I routinely end up writing "obviously", stop, and end up with 3 pages of clarification.
- m463 4y ago
  Actually the best use of "trivial" is in the negative, like... "Wellllll, that's non-trivial."
kjkjadksj 4y ago
Check then for conda, which only asks you to copy and paste a shell command they provide for you and eventually say “yes.”
mparnisari 4y ago
THIIIIS i hate it when people say "oh just do this"!! "JUST" implies it will take me 3 seconds, where in 90% of the cases it takes me 3 hours!!
nonbirithm 4y ago
I agree with you, but the difference between being able to tell someone "if you need to install some dependencies in Rust, just use Cargo" and show them the documentation, and whatever the equivalent documentation is in Python are worlds apart. I think the author's argument would be weaker if he focused on a language other than Python that doesn't have a hellscape of a package ecosystem. And I remember custom build scripts with Cargo being a major pain at the time I used it, but the Cargo developers were able to sidestep major fundamental problems because they knew what didn't work with packaging in the decades before and managed to think about the design carefully enough from the start. Rust is different because the choice of package manager is "simply" Cargo, not necessarily that Cargo itself is trivial to use.
theptip 4y ago
I think this applies in some cases, but disagree here. “Just use poetry and pyenv” is my line, and I stand by it because these tools are well documented, and work well for most use-cases. It’s contrasted with a bunch of other more complex options, in tedious yak shave conversations. Often in threads where many people are complaining about how confusing the multitude of options are. It’s not confusing if you short-circuit the conversation; if you need advice, don’t go down the rabbit hole, just use poetry and pyenv on MacOS, you will make your life easier than the alternatives. There are plenty of docs that will give the hand-holding that you are looking for, so nice you have made the decision. If you know what you are doing there are arguments for other options, and data scientists have different tool chains. But I think we make it harder for new engineers by having five different ways of doing it, each with people arguing that actually their way is better.

2h 4y ago

100% agree. I use a programming language to get stuff done. and if the day ever comes that someone wants me to show them how I do what I do, I dont want to start that conversation with a sigh "well...", I want to start it with a "OK cool..." and all these Python "tools on top of tools" make me sad. Personally I like Go. If someone wants to build my stuff, then I just say go here http://go.dev/dl http://go.dev/dl and download Go, then set location to where the code is, and enter "go build". thats it. All languages should be that easy.

paulddraper 4y ago
Wait did you just call Go versioning and dependency management "easy"??
- 2h 4y ago
  I dont have trouble with it. Link to my code is in my bio, wheres yours? Unless you actually write Go code, I dont think you have a leg to stand on here.
- cure 4y ago
  Compared to the dumpster fire aka Python packaging, Go versioning is easy and predictable indeed.
  - silverwind 4y ago
    Until you hit the obscure rules around v2+ go modules.
    - 2h 4y ago
      I have been coding in Go for years, even professionally sometimes, and I have never had to use "v2" in any of my code. granted my stuff is not popular, but its really just a way for popular repos to handle big changes. Once I got past v1.9.9, I just changed to v1.10.0.
      - 4y ago

jakewins 4y ago

Are there any efforts akin to deno for python? A “burn all the packaging down and start over” path? It’s so thoroughly broken, every day a dev on some team gets their poetry env entangled with some system installed python, or numpy suddenly decides all the CI builds will now compile it from scratch on every build, or.. Today it was segfaults on poetry version X on the M1 Mac’s, that went away in version Y but of course version Y broke pandas for the windows devs..

LanternLight83 4y ago
Guix (and maybe nix?) insists on being the "one package manager to rule them all", so Python packages and Rust crates are all re-packaged (sometimes with automated importers). Well I do still occasionally need to package something myself (or cheat pip/requirements.txt it), it definitely covers this "burn it down" philosophy and keeps environments isolated and reproducible.
bayesian_horse 4y ago
I have never used poetry... Mostly I just use conda or plain python envs. Never had those problems like you mention. When using anaconda in the stable channel you'll get a straight-forward "conservative" distribution of all the data science/numerical packages (and more). That's why people often say "just use anaconda", it really is quite simple as long as you don't mess up your stable environments with exotic packages you just want to try out. And it works well on Windows. No idea about Mac and don't care. Python doesn't need "something like deno". All programming languages need package management.
- nonethewiser 4y ago
  How many people work on these projects managed by conda? > And it works well on Windows. No idea about Mac and don't care. Package management is a cross platform problem. You may not need it but that doesn’t make the current solution good.
  - bayesian_horse 4y ago

bayesian_horse 4y ago

In my personal experience, I'll take Python's packaging hell over nuget or npm any day. And it's often less about the package manager and more about the ecosystem: Can you find what you need? Is what you need stable enough? Does it break every few months with new versions of the runtime, either because of actual incompatibility (npm/node) or versioning shenanigans (nuget)?

inferiorhuman 4y ago
Let's back up a moment. npm at least works on *BSD. Anaconda is just as toxic as Electron in that regard.
- bayesian_horse 4y ago
  Given the target audience of Anaconda, I can see how BSD compatibility doesn't matter to them. I guess it would take some major lifting because much of what they do is wrangling the compiling environment. Supporting Windows is hard enough. That's not "being toxic", you just can't be everything to all people.
  - inferiorhuman 4y ago
    Nah it's toxic. It's not just that Anaconda won't provide binaries, it's that you can't even build a project that uses Anaconda on not-Linux/Windows. In terms of why you shouldn't tell someone to simply use Anaconda, that's pretty high up there. As tedious as javascript package management can be, python has consistently given me more trouble.
    - bayesian_horse 4y ago
      I think that's an "Am I the Asshole" situation, and you are getting it wrong. BSD is niche, especially in scientific computing. Saying anaconda is worthless because its maintainers don't expend a ton of effort on making your particular niche more convenient is the actual toxicity...

NuSkooler 4y ago

Why not "simply" admit that Python and it's ecosystem while broad, are an absolute clusterfuck?

whalesalad 4y ago
Because that’s not at all the case and your rhetoric is what perpetuates the false image.
- stathibus 4y ago
  Yeah it totally is, and people who pretend it's not are the problem. Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. And I'm awfully tired of folks in the "python community" blaming the victims of the mess they made.
  - nonethewiser 4y ago
    > Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. This approach seems insane but correct.
    - BiteCode_dev 4y ago
      No, use the procedure described in the previous article. It will be less hard than using docker.
- JohnFen 4y ago
  As a user, it kinda is. My recent example: last week, I was installing a new program that uses some python scripts during its operation. I could not get those scripts to run. I knew it was because of the usual python problem of the scripts not matching the version of python that was executing them, but figuring out how to fix that took me a full day. Just to make Python work. Not even as a dev. Wearing my user hat, this is a clusterfuck. There is no other language that I am exposed to that presents this sort of problem, and this is a very common issue with Python. This is why I start to get sweaty any time that I'm using software that involves Python. It turns into a crapshoot and half the time, it's going to cost me a lot of time and stress. > perpetuates the false image. You can believe it's a false image if you like, but there are a whole lot of end user experiences that indicate it's very real.

jooz 4y ago

Ive heard that 'venv' are very problematic, but honestly, Ive never had a problem. And I used them daily. I understand that it can not be enough on some cases... that don't concern me. I would recommend to 'python -m venv' and thats all.

bobx11 4y ago
This is also my setup. It has the added benefit of being already included on every python install already so there is nothing extra to use.
hobs 4y ago
Same no problem, different solution - I use pycharm extensively and it manages my venvs 99% of the time with no issues at all, the only time it took me a minute of head scratching was realizing I needed to install a new system python to make a venv with it, but that would be clear if you were doing it via the shell approach you are using as well.
Groxx 4y ago
I've heard a lot (a lot) of complaints that are wildly misattributed to venv (like "version X of Y broke my project"), but I've essentially never heard of issues with venv itself. Aside from needing to know to use it. Which is certainly a problem. But python blessing a single venv-system might be worse in the long run...?
abrichr 4y ago
Agreed, never had a problem with this approach. The only limitation I've encountered is when moving the environment or renaming one of the parent directories. In which case it's easy to create a new one: # optional: freeze the environment if you don't already have a requirements.txt source .venv/bin/activate pip freeze > requirements.txt deactivate # remove the old environment rm -rf .venv # create a new one python3.10 -m venv .venv # activate it source .venv/bin/activate # install the requirements pip install -r requirements.txt
- nicoburns 4y ago

doublepg23 4y ago

I think I mostly wrapped my head around pyenv and used Anaconda the other day. It was quite the pain, to setup and then it seemingly mangled my fish and bash configs causing a noticeable delay every start up. Not something I was hoping for just for hacking around on some AI project. Disclaimer: I was using Fedora which has Python 3.11, using Fish which is clearly non-standard and I’m a sysadmin not a Python dev.

nightfly 4y ago
You don't have to activate venvs, you can just refer to the paths to pip and python inside of them as needed
- ElectricalUnion 4y ago
  Second this, just run python/whatever-binary-you-need inside a venv: `${my-venv-path}/bin/python` `${my-venv-path}/bin/${whatever-binary-you-need}` `%my-venv-path%\Scripts\%whatever-binary-you-need%` (because Windows...)

rektide 4y ago

Having encountered poetry recently for the first time, it was "simply" hell. I just wanted to use a single file python project, https://github.com/rumpelsepp/oscclip https://github.com/rumpelsepp/oscclip I spent about three hours trying to figure out how to setup python keyrings to work, to let me just get started using poetry. On a system I was ssh'ed I to. Gnome-keyring-daemom was up. I spent a while adding random pam rules suggested by archwiki in to inject more gnome-daemon stuff in my envs. Random gnome-keyring-unlock scripts, which quickly start talking about Clevis and tpm and fido 2-factor. Wading through hundreds of responses about Seahorse, a gui tool unsuitable for ssh. Many long miserable sad stories. In the end I stumbled upon someone who suggested just nulling out & turning off keyring with some config to make it have a null provider. After this the poetry project just worked. The tiny handful of deps this project has were already installed on my system, but poetry was also a task runner, instrumental for the usage of this single-file script. There's been so many years of churn in the python world of tools. A fractal nesting doll of virtual-env, mkvirtualenv, & various offshoots. I hope some day there is a mature reasonable option folks generally find agreeable. Poetry eventually worked for me, but what a miserable gauntlet I had to walk, and the cries of so many who'd walked the path & utterly failed echoed out at me at every step.

whalesalad 4y ago
Yikes. One of my favorite features from pip is how easily you can install from a git repo, or even the absolute url to the master.zip
mixmastamyk 4y ago
Yeah, that's the problem with all these "helpful" posts. You don't need poetry for that. My single CLI packages still use a setup.py that I haven't touched in five years and was simple enough to write.
deleted 4y ago
[deleted]
AndyKluger 4y ago
FWIW, for the case of installing Python packages as runnable tools (as opposed to importable libraries), pipx is solid. For example, you can install oscclip to a temp dir with a dedicated venv and run it immediately: $ pipx run --spec 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help Or install it more permanently: $ pipx install 'oscclip @ git+https://github.com/rumpelsepp/oscclip' My own Zsh frontend for managing Python venvs and deps, zpy, makes use of pip-tools to accomplish the same, with its pipx clone, pipz: $ pipz runpkg 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help $ pipz install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'

trey-jones 4y ago

> You should really use docker >> I think you missed the point. Maybe I did, but I've been using Docker as version management for pretty much every technology I employ for five or six years. Prior to that I sparsely used things like rbenv and virtualenv and I actually thought it was super dangerous and unreliable. Maybe it's gotten better in recent years, and certainly people who write python and ruby every day are going to know more about this than I do. I don't install anything on my computer if I can just use Docker for it. OK, I do have go:latest, but I use docker images for various projects that might be on any version of go from 1.8 to 1.20. Your website still runs on PHP5.3? I can help you (I won't, but I could totally run it locally!). Reasons I like docker better: 1. Any scripts or configs can explicitly refer to the version number. No guessing or assuming. 2. Our whole team uses the same version. 3. Only one dependency: docker. Granted I'm more of a sysadmin than a developer and I'm sure that biases apply.

whstl 4y ago
Suggesting Docker in the context of Python dependencies is still missing the point, because you still need proper dependency management in case you have to rebuild your image for some reason. If you have a Docker image that builds with "pip install whatever" and this library gets updated with a breaking change, you won't be able to rebuild the image without changing the dependency or the code itself, for example.

melody_calling 4y ago

If you're just using python as a local scripting language, and not pushing production code, the other option is to simply not bother with any of this. When there's a new python version I'm interested in, I install it via Homebrew and update my zshrc to clobber everything else via $PATH. All my scripts and tools are broken? Just reinstall the packages globally. Whatever. Since the big 3.x transition, it's pretty rare for forwards-compatibility to break (IME), and if something does, I can just try running prior python3x binaries until I find the last version that worked. It's hideous, but honestly the least stressful way I've found to date.

kjkjadksj 4y ago
Or you can just do everything in conda and maintain compatibility forever.

tipsytoad 4y ago

Aside: I usually use direnv to activate the venv (or poetry) when entering a dictionary https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19fba6aeb0 https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19f...

davb 4y ago

Python runtime deployment is a major pain point for us (CS department at a university). On the most tightly managed lab machines, which are all in lockstep on a fixed configuration (latest Ubuntu LTS with a updated image pushed annually), we can provide a consistent Python setup (e.g. Python 3.10 and a fixed set of C-based modules like psycopg2). However, our staff and PhD desktops and laptops are more diverse - with the OS often only being upgraded when the distro is going out of support, they could be running n, n-1 or n-2. That, most likely, means three different Python versions. We could use pyenv to let people install their own preferred version. Installing with pyenv requires building from source (slow on some of our oldest machines). This also means installing the Python build deps, which is fine for our departmental machines but not possible on the HPC cluster (operated by a different business unit) or the Physics shared servers. It's also less than ideal for our servers where students deploy their projects (where we want to minimise the packages installed, like the build-essentials meta package). It's also a massive stumbling block for less experienced students with their own laptops which could be running any distro, of any age. Many CS101 or engineering/business/humanities students taking a programming class, who have never programmed before, would really struggle. So, classes might tend towards teaching lowest common denominator Python (i.e. the oldest conceivable version a student might have installed on their machine). Sure, we have in-person and remote lab machines students can use - but it's not always convenient (especially for the data science / ML students running Jupyter notebooks with their own GPU). There are workarounds, but they all have serious downsides. Compared with Node.js and Go, where users can just download the appropriate package and unzip/untar the runtime or compiler version of their choice, deploying the Python runtime has enormous friction (especially for less experienced users). This has the bonus of simplifying deployments elsewhere in our infrastructure (CI/CD, containers, etc). And while we all complain about node_modules, Python venvs not being trivially relocatable is another huge frustration. We've used Anaconda, but that comes with its own issues (most recently, finding that it ships with its own gio/gvfs binaries and libraries which fail to mount our CIFS DFS shares - causing confusion for users running `gio mount` from within a conda environment).

tibbon 4y ago

Python as a language I find pretty nice. What I don't find is their environment and packaging system compared to something like Rust. "There should be one-- and preferably only one --obvious way to do it.", unless it is how to setup your environment. I only use Python every few months, and it is always a struggle. In comparison, "cargo build" works 98% of the time just after "git checkout"

belval 4y ago
Yes, but Python really is in a tough spot when it comes to dependencies because a lot of it are compiled in a lot of different languages using libraries that may or may not be available on your OS. That's the real issue. If you install python packages honestly anything works, it's when you have the ML stack of PyTorch (needs cuda, cudnn, needs to be compiled to match your OS version) + custom CUDA operators (also need to be compiled) with something like a mariadb connection (needs the mariadb OS library). conda solves it by packaging EVERYTHING, giving you atrocious 30GB environments, pip doesn't solve it at all and none of the challengers really have much to offer (in my opinion).
- blactuary 4y ago
  That's Anaconda that packages everything, not conda itself. No one should really use Anaconda imo, instead use Miniconda which only installs the bare minimum needed for conda to work in a base environment, and then you create environments for each project.

aldanor 4y ago

Unfortunately or not, in some fields conda is the only sane choice because it can manage non-Python binary dependencies that Python packages may depend on. Some of those dependencies may be huge C libraries that are a pain to build, like HDF5, so if you're not using conda you'll be relying on your OS's package manager to serve your particular venv's needs - we all know what usually happens next.

zackees 4y ago
[dead]

not_enoch_wise 4y ago

One does not simply tell people to use pyenv, poetry or anaconda...

deleted 4y ago

[deleted]

DemocracyFTW2 4y ago

IMO to fix these issues the first thing to do is to write an alternative to Python's `import`, more like a function call that works more similar to NodeJS' `require()`. That should start life in userland and only later become part of the language. How to transition a bazillion packages though I do not know.

BiteCode_dev 4y ago
I have an article planned solely on imports. It saddens me to say, but imports are indeed a complicated topic on Python. The PYTHONPATH is not something people know about, and once they do, it's not intuitive. The weird handling of namespaces or relative paths doesn't help.

BiteCode_dev 4y ago

Author here. I'm late to the party but AMA :)

deleted 4y ago
[deleted]
oconnor663 4y ago
How do you see this advice changing in 3-5 years?
- BiteCode_dev 4y ago
  This procedure was valid 5 years ago, and all the alternatives have changed a lot in the same time period. So if I had to chose one way for the next decade, I would bet on this. But I will not pretend we can be sure about anything on this matter. In fact, my hope is that we can improve the process tremendously by creating an unified way of bootstrapping Python for all OS. If this ever happens, the procedure would become mostly obsolete. At the end of the week I will write about this, and the various experiments the community have done so far in that regard.
vqbd 4y ago
There's a version of pyenv for windows called pyenv-win. The cli installs as `pyenv` and the commands are the same. So not game over imo. Also, [even if it did], it installs binaries and doesn't compile.
- BiteCode_dev 4y ago
  You are not telling them to use pyenv, you are telling them to use a totally different tool that happen to have a similar name and provide a similar interface. You have already lost.

tpoacher 4y ago

I often find the reason for all this hell is, ironically, an effort to help people who dont know "how computers work", by offering "useful automations". And then these automations clash and fail because of the complexity involved. If you "simply" (yes yes I know) download the python version you want directly and compile/install in a local folder, and use that with venv as a way to manage individual project dependencies, all problems go away.

jakewins 4y ago
Until you need to distribute that project as a library, or you’ve shipped it as an app and now you need all users to upgrade library X or add library Y. Python packaging is fine for small local dev; problems arise in distribution, rollouts of upgrades and ensuring your apps work in the zoo of local setups your users have

kjkjadksj 4y ago

These comments in this thread were a bit surprising to me. People really like to make things hard on themselves not doing a little do diligence. Yes, I still say to simply use conda. It spells out exactly what is getting installed in the environment, and uses a separate python installation for each env than the system. If you don’t trust it just type which python. I never get these headaches people seem to have, since conda is easy and well documented and supported.

zajio1am 4y ago

Perhaps people should accept that these are more developer tools (like git) and not end-user distribution tools and just use regular distribution packages (rpm / deb) for that.

pmarreck 4y ago

"simply" use Nix, which solves this problem for every language.

n8henrie 4y ago
I wish this was a "simply" answer! Been getting more into nix lately and really enjoying it, but converting python to nix has been a challenge. It seems like the existing and recommended suggestions are either deprecated / unmaintained or still alpha at best (pip2nix, mach-nix, dream2nix). https://discourse.nixos.org/t/basic-flake-run-existing-pytho https://discourse.nixos.org/t/basic-flake-run-existing-pytho...
- pmarreck 4y ago
  Well one way of going about it is a hybrid solution. So you set up your python dev directory with a nix file, but then you use pyenv or whatever to encapsulate everything else inside it using python tooling. So it's not strictly Nix-all-the-way-down, but at least as far as being in the directory is concerned, the "right" python version and libraries are available.

INTPenis 4y ago

I don't use any of those, I use direnv with standard python3 -m venv module.

zaptheimpaler 4y ago

Python packaging is sooo fun it has given me permanent brain damage and PTSD. Now I have a docker devbox with all my language toolchains and fun unix tools installed. I could install 3 different versions of cuda, 8 different pyenv pythons all sharing parts (but not all) of each others modules with torch compiled for a 4th different version of cuda that is NOT installed, then replace the core system python with a duck. pipx has somehow installed a version of borg backup that depends on a secret hidden 9th python. Then I will simply `docker rm` and `docker compose up -d` and I'm back. Yesterday I ran a random academic paper ML model in python in 5 minutes on my docker machine. HAHAHAHAHAÀÄÄÄÄÄ i am invincible!!!!!!

jamesralph8555 4y ago
Docker is the only way to do ML. One thing I didn’t think to do before for a while was make separate docker files for each project so all of the deps are installed automatically and that has since helped tremendously.

Havoc 4y ago

The variety of solutions sure isn't helping. I'm just sticking all dev projects into a separate LXC and calling it a day. Don't want to deal with all the various separation models the various languages and package managers cooked up

CMCDragonkai 4y ago

I dealt with this years ago by only using Nix to do python Dev. Worked great for the entire ML stack. Had to do a bunch of packaging for nixpkgs early on though.

akasakahakada 4y ago

Any of these is complex as hell. I wish everything is as simple as downloading exe file on my Windows Desktop and double click, done.

phendrenad2 4y ago

The problem is, dependency management isn't a solved problem. I was nodding my head in agreement at the part where the author mentions that pyenv compiles Python from source when it installs. If you try to figure out the reason behind this, it becomes obvious that the only one that makes any sense at all is that distributing binaries, in a secure way, is hard for open-source projects to manage. After all, bandwidth costs money, someone has to pay for the build server, someone has to pay for the data transfer bandwidth. And, do you trust that person to not be a rogue actor? It's easier to pull down the source, on the end user's machine, cross your fingers for good luck, and compile it. We also haven't figured out how to incentivize maintaining backward-compatibility. 99.9% of the time when some library updates, and stops working with language version X, it's using some hot new feature of the language. Usually just because the library author thought it would be cool or more elegant. The entire software world needs to update now, because someone left-padded us for elegance.