22 ms·

Pyenv – lets you easily switch between multiple versions of Python

• jphoward 3y ago
  Is anyone is the AI/ML area finding success with anything other than conda, where installation of CUDA/CUDnn is required? Although I often have to pip install a lot of packages, I find conda's nvidia/pytorch/conda-forge channels are still by far the easiest way to get a deep learning stack up and running, and so I just stick with conda environments. I've tried poetry in the past but getting the NVidia deep learning stack up and running was really tough.
  • halflings 3y ago
    uv has been really awesome as a replacement for pip: https://github.com/astral-sh/uv https://github.com/astral-sh/uv So fast it finally made virtual environments usable for me. But it's not (yet) a full replacement for conda, e.g. it won't install things outside of Python packages
    • northzen 3y ago
      How about prefix then? https://prefix.dev/blog/uv_in_pixi https://prefix.dev/blog/uv_in_pixi
  • thangngoc89 3y ago
    I use poetry and direnv. Coming from node/npm, it feels natural for me to just do this. I have really no troubles of installing Pytorch with poetry
    • maleldil 3y ago
      How are you installing Pytorch with CUDA with Poetry? I stopped using Poetry because it wouldn't automatically get the CUDA version; instead, it would install the CPU version. I migrated to PDM, which does the right thing.
      • claytonjy 3y ago
        was this before torch 2.0? With the very notable exceptions of a few mispackaged versions, torch now includes all the relevant Nvidia libs, and I haven't seen it grab the CPU version on a GPU box yet, though I'm not sure what it looks for. A notable open issue in poetry is we can't currently specify one dependency on torch, and have it grab CPU version on some systems and GPU on others. Does PDM solve that?
        • maleldil 3y ago
          I don't think PDM solves that directly. What I do is have different lock files for different platforms (e.g. Linux/CUDA and macOS/Metal), but pyproject.toml lists only "torch".
      • thangngoc89 3y ago
        Before CUDA 12.0 you have to specify a field in pyproject.toml like this [tool.poetry.dependencies] python = ">=3.10,<3.12.0" torch = {version = "^2.0.1+cu118", source = "torch118"} torchvision = {version = "^0.15.2+cu118", source = "torch118"} [[tool.poetry.source]] name = "torch118" url = "https://download.pytorch.org/whl/cu118" priority = "explicit" However, since CUDA 12.0 and Pytorch 2.1.0, just install like normal poetry add torch torchvision
        • maleldil 3y ago
          I stand corrected. I was familiar with the first option, which coupled the dependencies with the platform, whereas I wanted a CUDA version on Linux and a Metal version on macOS. However, this works perfectly with Poetry 1.8 and Pytorch 2.2. I suppose the only problem is what PDM also does, where the lock file is platform-dependent. I'm not sure whether Poetry allows you to select a specific lock file, however.
  • rmnclmnt 3y ago
    For anything related to CUDA/CuDNN, use one of NVIDIA base Docker images. Then whether you use Conda / Pip / Poetry / Pipenv does not matter much. Not at all a Conda fan myself and avoid it like the plague
    • Y_Y 3y ago
      What's surprising to me is that this isn't better known. The only reliable solution I've found is to go with the pytorch or deepstream images from NGC. Conda is probably a good idea for noobs who need Cuda installed for them on windows, but otherwise I find it an endless source of finicky issues, especially for unsavvy ML scientists who are looking for a silver bullet for package management. This link shows which package versions come in which Docker tag and is invaluable: https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html https://docs.nvidia.com/deeplearning/frameworks/support-matr...
      • rmnclmnt 3y ago
        10 years ago, « Data Science » work past the experimental stage was performed by SWE with a knack for applied maths. So investing in tooling to do things properly was a given. Nowadays, most DS people only want to do ML at the experimental stage only and get lost when things get on the engineering side of things. But for their defense, nowadays the bare minimum skills require to do programming, containerization, CI/CD, etc. More experienced and swiss army knife SWE/MLE have to educate the willing. It was already the same 10 years ago with MATLAB dudes not wanting to get dirty with C/C++/ASM SIMD. The history repeats itself, only at a faster pace
  • godelski 3y ago
    Yes. I simply do python -m pip install torch torchvision and it works. It used to not, but it's been fine for me for about a year now. There's a very good chance I've installed cuda on my system before this though. And usually cudnn and some other packages because this is part of my standard install. And then I also never run into the issue where a package is looking for nvcc.
  • michaelmior 3y ago
    I use pipenv and I've found it to be much more usable than conda. For my use cases, it's generally faster and I've run into fewer dependency issues.
  • misnome 3y ago
    I love poetry but have found it pretty hard once you move off of anything that doesn’t manage to get wheels on pypi. We make extensive use of conda/mamba to solve this, and are pretty happy with it, especially with conda-forge.
  • samus 3y ago
    Pyenv just worked for me. I am actually using Fedora Silverblue and have GCC and the CUDA SDK available only inside a toolbox container. Therefore, I have to enter that toolbox to install things like FlashAttention.
  • claytonjy 3y ago
    I have successfully transitioned an ML/AI team of seasoned researchers away from conda and to poetry. Some also use pyenv, I suspect a lot don't bother but may get bitten eventually. It's definitely a learning curve, but it turns out every conda user has been bit by the irreproducible tendencies of conda quite often. Nobody uses the conda env file, they just start an env and pip install things into it. They don't realize the base env has stuff, too, and conda envs are hierarchical rather than isolated. I know it's possible to use conda in an isolated and reproducible way, but have yet to meet someone that does so. So it hasn't been hard to pitch poetry to these folks, and while many complain about the learning curve they appreciate the outcomes. We're a pytorch shop, and torch mostly just works with pip or poetry these days, as long as you skip the versions the torch maintainers mispackaged. We rarely need anything higher-level that only conda could install. We really like having more than two dependency groups as this allows us to keep research and production in the same repository. main, dev, research. Then researchers contribute to the core library of a project and keep research and production using the same code for running and evaluating models.
  • abkfenris 3y ago
    Have you tried https://pixi.sh/ https://pixi.sh/ ? It brings Cargo/NPM/Poetry like commands and lock files to the Conda ecosystem, and now can manage and lock PyPI dependencies alongside by using uv under the hood. I haven't been using anything CUDA, but the scientific geospatial stack is often a similar mess to install, and it's been handling it really well.
• blowski 3y ago
  OP - I'm curious why you've submitted this today, you seem experienced enough to guess this is not the first time you've used it. Has there been a major change or something happened in the community I need to know about? Or are you just spreading the love for a great piece of open source software?
  • dallyo 3y ago
    Also a bit confused as to why this is currently the #1 post.
    • BiteCode_dev 3y ago
      Because it's not a solved problem, pyenv is niche and people discover it once again and think that they finally solved their pain. They will try it, and most of them will pay the price for it. It's a cycle.
      • airstrike 3y ago
        Cue ominous Reaper sounds
        • slater 3y ago
          username checks out
  • punnerud 3y ago
    I used Python a lot, but missed pyenv. Used it for some weeks now and was thinking this is something I would like to know about earlier (thats normally when I post it to HN).
  • xeromal 3y ago
    There are always new developers coming onboard that might have missed this handy tool. It's worth firing up the conversation on occasion.
  • recursive 3y ago
• BiteCode_dev 3y ago
  I have to warn again users that think they have found the silver bullet that pyenv comes with a big caveat: it compiles python on your machine. The number of possible modes of failure in this situation is huge. See also: "Why not tell people to "simply" use pyenv, poetry or anaconda" https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup. Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.
  • bmitc 3y ago
    > it compiles python on your machine What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.
    • bheadmaster 3y ago
      > What is the alternative? Using per-distro official installation channels. E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
    • Leynos 3y ago
      Rye downloads pre-built binaries. I'll leave further discussion of the reliability and provenance of said binaries to someone else.
      • BiteCode_dev 3y ago
        Binaries are quite reliable and from a serious project, but rye is young and uses shim. Still, it's part of those new generation of tooling that bringing hope for the next few years.
    • kzrdude 3y ago
      These builds are an alternative: https://github.com/indygreg/python-build-standalone https://github.com/indygreg/python-build-standalone Those are what Rye and hatch use. Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
• godelski 3y ago
  Homebrew really bugs me. The devs want you to use the most current version of Python. Except the requirement is set at the package level and there's no enforcement. So you just keep getting python versions installed. I don't get why you just don't set a minimum version like every other manager. Or if you want to do this crazy thing, don't rely on package maintainers to enforce it. I just ends up in major bloat
  • rcarmo 3y ago
    This is misleading. The older versions get cleaned out as brew does the upgrades (if not, then check your brew settings). My take on brew's Python is that it is there to make sure there is a uniform runtime for everything with a Python dependency, but I am not required to use it.
    • mikemcquaid 3y ago
      As. Homebrew maintainer: confirmed. You can also adjust your settings to pin versions or avoid cleanup, too. ‘brew pyenv-sync’ may be useful around this area, too.
      • sgarland 3y ago
        If I pin a version of Python, isn't that going to wreck any tooling that depends on it? Unless you're saying have multiple versions of Python installed. This is practically the only remaining annoyance I have with the Python ecosystem (relative imports aside). I use some tools, like Glances [0] whose formula relies on a much newer version (3.12) than the actual package requires (3.8) [1]. So when there's a Python update, all of those update as well. I thought I'd fixed this with pipx, but in a way that's worse, because the venvs it builds depend on a specific version of Python existing, which doesn't work well with brew always wanting to upgrade it. I want a stable, system-level Python that I don't touch, don't add packages to, and which only exists as a dependency for anything that needs it. If an update would break a package I have installed (due to Python library deprecation, etc.), it should warn me before updating. Otherwise, I don't care, as long as any symlinks are taken care of. Separately, I want a stable, user-level Python that I can do whatever I want to. Nothing updates it automatically. I can accomplish this by compiling Python and using `make altinstall`, but if there's a better way, I'd love to hear about it. [0]: https://github.com/Homebrew/homebrew-core/blob/20e744191e74d563406018dca1050d202bedabcb/Formula/g/glances.rb https://github.com/Homebrew/homebrew-core/blob/20e744191e74d... [1]: https://github.com/nicolargo/glances https://github.com/nicolargo/glances
• NikkiA 3y ago
  I thought the recommended approach these days was to use the venv package built in to python3 with `python3 -mvenv ...`
  • Hasnep 3y ago
    You're right, but venv requires an existing Python install to create a venv. Pyenv is for installing multiple different versions of Python to create those virtual environments.
  • bheadmaster 3y ago
    It is, but the problem with `venv` is that it can only use the same version of Python that you're using. E.g. if your Python installation is 3.8, your `venv` will also use 3.8. Pyenv allows you to use whatever version of Python you want.
    • alwayslikethis 3y ago
      Most systems allow you to keep multiple python installations, no? I think the best way is to specify the python when you do this: python3.11 -m venv env On most Linux distros you probably don't want to change your system python to whatever, because there are a lot of system programs that may not be compatible with a version other than what is on the system by default. Then once you activate it you can make sure your python
    • 0x008 3y ago
      it's not a bug, it's a feature. just install the different python versions by downloading the binaries or through your package manager and run python3.<whatever_version_you_want> -m venv and the virtual environment will automatically, forever, use that python version when you activate it.
• deleted 3y ago
  [deleted]
• crasshit 3y ago
  [dead]
• _dain_ 3y ago
  I used to use this a lot when working on a Windows machine. Worked pretty well. But nowadays Nix solves the same problem in a fully general way, for all software rather than a single language. You can have whatever versions of whatever you want, side by side, without interfering with each other.
  • pyrophane 3y ago
    What is your solution here? Are you using a project-specific flake? Do you still install packages with pip?
    • _dain_ 3y ago
      I was more talking about version management of the Python interpreter itself. So far all the packages I need are already in nixpkgs so I haven't needed to stray off that path yet.
• TomSwirly 3y ago
  This is a truly, truly terrible idea. It adds several failure modes, some subtle so you can go a long way in a state of error, just so beginners can type `python` instead of e.g. `python3.10`. Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol. If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv. Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
  • fragmede 3y ago
    though, with openinterpreter, you can just ask it to fix your python and it'll help you out of a weird situation that you as an newbie don't understand. Still, node's node_modules implementation isn't a bad one, at the expense of disk space, you don't run into the same problems as you do python.
  • wodenokoto 3y ago
    But how do I get a .venv with a new/different version of Python in your setup? And how do I add some kind of Python version requirement in my project?
    • hnfong 3y ago
      python3.xx -m venv .
      • happymellon 3y ago
        And pyenv gives you python 3.xx rather than 3.xy...
  • rcarmo 3y ago
    This is not a terrible idea. It is a tool that makes it possible for people to either retrofit a newer version of Python onto an ageing OS or to develop for a specific older version of Python on a newer OS. If you don't need the features the tool uses, you shouldn't use it--I don't see why you're comparing it to virtualenvs, it provides completely different runtimes with a different engine and a standard library, and the ability to switch between those by just cd'ing to a folder with a .python-version file in them.
• lgxz 3y ago
  I think the only problem of pyenv is that you can't upgrade an old virtualenv to a newer Python version.
  • Hamuko 3y ago
    Isn't that more of a limitation with virtualenv? I've been using virtualfish as my venv manager in the fish shell and it has a `vf upgrade` command for upgrading virtualenvs to newer versions, either by updating in-place or rebuilding the entire virtualenv. So far has worked nicely with my pyenv Python installs.
    • rcarmo 3y ago
      Yeah. I had a shell snippet someplace that was essentially pip freeze | remove virtualenv | make virtualenv | pip install. It's not that hard.
  • rewgs 3y ago
    That has nothing to do with pyenv. Also, this really isn't a problem -- just blast out of the old venv, make a new one with a new version, and you're good.
• danieljanes 3y ago
  We use Pyenv successfully for developing the Flower open-source project. We use a few simple Bash scripts to manage virtual environments with different Python versions via pyenv and the pyenv-virtualenv plugin. The main scripts are `venv-create.sh`, `venv-delete.sh` and `bootstrap.sh`. `venv-reset.sh` pulls these three scripts together to make reinstalling your venv a single command. Here's the link if anyone is interested: https://github.com/adap/flower/tree/main/dev https://github.com/adap/flower/tree/main/dev
• thrownawayjests 3y ago
  Obligatory relevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/
  • punnerud 3y ago
    Need a new one with pyenv in it? (or to replace it)
• ktosobcy 3y ago
  Python (and Ruby) handling of versions and dependencies is truly awful... :-(
  • bvrmn 3y ago
    It's inevitable for non-dead interpreted languages.
  • lioeters 3y ago
    I've only used Python occasionally, and more than half of that time I ran into issues with the installed Python version or dependencies. Not only version 2 vs 3, but for example a certain project required exactly 3.11, not 3.10 or 3.12. Last time, I tried to install Caddy and broke all previously installed packages/commands - and after searching for a solution and trying different things, the best I could come up with was to switch versions (manually) before and after running a command to restore the environment. Now I understand I should have been using Pyenv the whole time. But it sure felt unfriendly for a newcomer to the language/ecosystem. Granted my learning approach is to jump in without much preparation, I wish the language designers or the ecosystem provided a better experience when things don't work right.
  • yxhuvud 3y ago
    How is it awful on the Ruby side? In my experience it is a solved problem since bundler was invented 15 years ago. It's been very free from pain during the last 10 years or so.
    • ktosobcy 3y ago
      AFAIR it was mostly because Ruby still wanted to use not-100%-fixed versions so there was still an issue with something updating slightly breaking the build. Just a cursory experience when trying to deal with Ruby. Though it _felt_ somewhat more stable than Python.
  • jscheel 3y ago
    In my experience, python has been far, far worse than ruby. Now, to be fair, I’m much more comfortable in the ruby ecosystem, but still, everything feels much more brittle in python package management.
• jpalomaki 3y ago
  After some trial and error I have now settled on the following "stack": 1)I manage python versions with pyenv 2)For each new project I create a new virtualenv with venv "PYENV_VERSION=3.10 python -m venv .venv" 3)Then I start jump into the venv and initiate the project with Poetry ("poetry init -n") and manage dependencies with Poetry. If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry. A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project. I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed. Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
  • p33p 3y ago
    Odd to me that you don’t let poetry create the venv? Why do this separately? Our flow is similar: pyenv (windows and linux), pipx, poetry. We’ve also defaulted poetry to utilize the current global version of Python and build the venv within the project folder.
    • lexicality 3y ago
      Personally I've had problems with poetry managing virtualenvs in the past so I just don't let it touch them any more. Maybe it's better now, but I don't see any reason to risk it, given how often Python seems to like to ruin my day. I also don't like that by default it wants you to use `poetry run` to run things. Sure you can configure it not to, but it still annoys me
      • Hackbraten 3y ago
        One issue I’ve been encountering is that Poetry isn’t aware of pyenv or `.python-version`. So what I do to initially build my venv is: pyenv exec pip install poetry && pyenv exec poetry install This creates the venv against the correct Python version, and I can now do without `pyenv exec` for this repository. And every time that `.python-version` changes (which I at most do a few times per year per project,) I throw away the `.venv`, do `pyenv install -s` and start over. Your `poetry run` point still stands, though.
• jmathai 3y ago
  More than Python versions, I use pyenv to keep from polluting the system’s site-packages. It also ensures that my requirements.txt is sufficient for the code to run.
• nlitened 3y ago
  Tools you can use to make sure the Python program you wrote keeps working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-virtualenv, virtualenvwrapper, pyenv-virtualenvwrapper, venv, pyvenv, conda, miniconda, poetry, docker, nix. Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
  • speedgoose 3y ago
    None. The software container image is the best bet but you need to keep the image and not only the building scripts.
    • nickjj 3y ago
      Yeah, containers don't fully solve this problem. We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result. Using pip to freeze your dependencies and writing a tiny shell script to generate a lock file at build time is better than nothing to solve this problem with nothing more than pip. It's what I do in https://github.com/nickjj/docker-flask-example https://github.com/nickjj/docker-flask-example and https://github.com/nickjj/docker-django-example https://github.com/nickjj/docker-django-example. It's not perfect but it solves 80% with minimal complexity.
    • Cu3PO42 3y ago
      This is not true. Nix actually solves the problem. If you package with Nix, you'll get the exact same version of Python with the exact same version of dependencies, including the exact same version of system libraries. Your build will work in a year just as it does today. You don't even need to keep any binary artifacts. Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
      • speedgoose 3y ago
        How does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively? Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
• kissgyorgy 3y ago
  I have been using pyenv for years and it's the best tool for managing Python versions. You can install any version, switch between them, use different versions for different projects, have multiple versions installed at the same type without problems. It really solved all my Python version managament problems, I never looked for any other tool like this since I started using it.
• chaoz_ 3y ago
  I've been using pyenv for several years now, but for some reason, the basic commands and overall integration don't feel as smooth as Node's nvm package. I wonder if that's because Python setup is technically harder than Node.
• rsyring 3y ago
  I've given up on trying to manage runtime dependencies with language specific tooling. Instead, I've moved to mise which handles the majority of them (Python, Node, Ruby, Terraform, etc.) the same way. It will also activate Python virtual envs: https://mise.jdx.dev/lang/python.html https://mise.jdx.dev/lang/python.html It has other nice helpers for development environments (tasks, env variables, etc.).
  • letmeinhere 3y ago
    And for those of you that are already on asdf-vm, it is compatible with most of those plugins. I have not yet made the jump but I'm tempted.
• rhdunn 3y ago
  I'm using pyenv to manage different projects on my machine. I use --system-site-packages to share libraries like torch, and some others -- especially to ensure those libraries are built with the same cuda version, avoid some issues when upgrading drivres, and to avoid having to keep multiple copies of several gigabyte libraries around. I'm not using things like anaconda for a similar reason I'm not using docker. -- I don't want a separate OS install for each service I'm running on my server. It works perfectly fine.
• synergy20 3y ago
  Just use the builtin "python -m venv" and life is good, why the others? I tried almost all of them, with various issues, now staying with the default venv, it's solid and get the job done.
  • jvansc 3y ago
    Yeah, my thoughts exactly. I’ve never found myself in a situation where venv did not do exactly what I needed it to do.
    • Karunamon 3y ago
      The problem there is that by default a virtual environment version of Python is just a symbolic link to the actual binaries installed elsewhere on the machine. If you do an OS upgrade or do anything that increments the system Python version, it could cause problems in the environment.
  • claytonjy 3y ago
    pyenv manages python versions, not virtual environments. venv won't help you grab python versions you don't have, and will happily install your requirements into a venv with the wrong python version.
  • kzrdude 3y ago
    The premise of controlling python version is that - you're sending the project to your friend / colleague and you want both of you to use the same python version so that you get the same behaviour. And then the same for your friend the buildserver. It's all about reproducibility.
    • synergy20 3y ago
      which venv does really well, what am I missing. in the "worst" case you can always do a docker
      • dragonwriter 3y ago
• rco8786 3y ago
  Not really a complaint - but I am surprised to see this on the front page of HN. This is an enormously popular project that's been around for over a decade that basically anyone who writes Python regularly is aware of. Only marginally less surprising than seeing a link directly to python.org.
• mococa 3y ago
  https://asdf-vm.com/ https://asdf-vm.com/ ASDF is better because it works with many more languages, other than only Python, like Rust, Go, Node, etc, and other tools, such as AWS/Google/Firebase/Azure CLIs.
  • deleted 3y ago
    [deleted]
  • hk1337 3y ago
    I used to be all-in for pyenv and rbenv both but have loved using asdf so much more.
  • antihero 3y ago
    Yeah, ASDF is fantastic, I'm glad stuff like pyenv and rvm existed and paved the way for it but ASDF is the way to go nowadays.
    • llimllib 3y ago
      asdf for python is a wrapper around pyenv, it didn't "pave the way", it's an important part of asdf. (Also I like mise better currently: https://github.com/jdx/mise https://github.com/jdx/mise)
      • secstate 3y ago
        Curious what you like about mise! I've been a strong advocate for asdf for a few years now. Aside from smooshing together tools that are often used together, but not always (direnv, make), I can't see what mise adds to the picture. And I like still having direnv for what direnv does and make for make does, rather than an all-in-one.
        • llimllib 3y ago
          the asdf CLI has really sharp edges and feels kind of aggressively unhelpful. Even if you just use mise as an asdf alternative, it has a nicer CLI and interacts with the same plugin ecosystem more smoothly. (I've found it to be an improvement on direnv, but I still use make) the comparison to asdf page has more: https://mise.jdx.dev/dev-tools/comparison-to-asdf.html https://mise.jdx.dev/dev-tools/comparison-to-asdf.html
• camgunz 3y ago
  I'll toss in my (macOS) workflow too: - Download whatever Python binaries I need from python.org - virtualenv --python=$PYTHON_VERSION ~/.virtualenvs/$PROJECT_NAME - pyproject.toml - pip-compile --generate-hashes There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough. Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier? python -m venv is fine, I've just gotten used to virtualenv. pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg. pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
  • Capricorn2481 3y ago
    Does it handle OS differences?
    • camgunz 3y ago
      Yep
      • Capricorn2481 3y ago
        How exactly? Isn't a requirements.txt file, generated from pip-compile, incapable of specifying OS stuff? It's also my understanding that pip-tools does not let you make layered requirements files with pyproject.toml files, only with .in files
        • camgunz 3y ago
          You can specify different dependencies in pyproject.toml and then use --extra with pip-compile to pull them in. I do this with dev and test sections for example. Constraints seem like they work like you'd expect here. pip-compile also has the -c flag if you want to use that. I guess I don't know what you mean by OS stuff, but maybe this works for that too?
• kerkeslager 3y ago
  This is the sort of tool that experienced devs all use, but nobody teaches or writes it into a tutorial on the larger language as a whole, because it feels like a side tool. For now I'm not seeing a lot of reasons to use PyEnv over the `venv` module that ships with Python 3.3+ [1]. I'm sure there are some thing PyEnv does that `venv` doesn't, but the fact that `venv` ships with Python greatly simplifies things. Basically my workflow is: 1. I'll create a Python project with, `mkdir dirname`, `cd dirname`, `git init`, `python3 -m venv .env`. This creates a hidden folder named `.env/` which contains the virtual environment. I'll then usually add `.env/` to my `.gitignore`, and also run `pip install --upgrade pip` and `pip install wheel`. 2. If I'm using an existing project, I'll do `git clone `projname`, `cd dirname`, `python3 -m venv .env`, `pip install -m requirements.txt` (requirements.txt is the idiomatic name for the dependencies list in Python projects). 3. I have the following lines in my `~/.bashrc` (hidden file that contains Bash settings): # Gets a directory named .env or .venv if it exists in the currend directory or any of its parents get_env() { if [ -d "$1/.env" ] ; then echo "$1/.env" else if [ -d "$1/.venv" ] ; then echo "$1/.venv" else if [ -d "$1/.." ] ; then get_env "$1/.." fi fi fi } get_absolute_path() { python3 -c "import os; print(os.path.realpath('$1'))" } on_prompt() { # Load a virtualenv environment if it exists in a file named .env env_folder=$(get_env $(pwd)) if [ -d "$env_folder" ] ; then if [[ $VIRTUAL_ENV != $(get_absolute_path $env_folder) ]] ; then echo "Activating env '$env_folder'" source "$env_folder/bin/activate" fi else if [ -d "$VIRTUAL_ENV" ] ; then deactivate fi fi } # Call on_prompt() every time the command prompt executes PROMPT_COMMAND=on_prompt What this does is when I `cd` or `pushd` into a directory or subdirectory of a directory that contains a `.env/` folder, it loads the virtual environment, and when I leave said directories, it exits the virtual environment. 4. When I install a dependency, I use `pip freeze` to get the dependency string, and I append that line to the file `requirements.txt`. For example, if I do `pip install django`, I get `Django==5.0.1` in my output from `pip freeze`, so I'll append that line to my `requirements.txt`. 5. To upgrade dependencies, I edit the version number in the `requirements.txt` file, and then run `pip install -m requirements.txt`. This makes sure that the requirements file stays up-to-date with my locally installed dependencies. 6. To get valid version numbers for a file, you can do `pip install <package>==`, which is basically asking pip to install an invalid version number. This causes it to list valid version numbers. For example, you can do `pip install django==` to view all available Django versions. [1] https://docs.python.org/3/library/venv.html https://docs.python.org/3/library/venv.html
• crabbone 3y ago
  Several years in ops in a large company where I had to hand-hold couple hundreds of programmers a lot of whom needed to install Python for development on their computers... One of the troubleshooting steps when "something mysterious" happens on developer's computer (eg. package installation fails for inexplicable reasons, Python "standard" library components missing or present when shouldn't be, incorrect component version etc.) was to remove pyenv. This step was often met with resentment and arguments... but, in most cases I was able to win :) ---- Now, here's a larger point: some programs solve the problem by actually solving the problem, while other programs solve the problem by adding more code around the problem, which, usually, creates new problems while only partially solving the original problem. An example of the former: fsck -- you run it, it looks at your filesystem, tries to fix it, if it's broken and then gets out of the way entirely. An example of the later: Kubernetes -- you start by having a problem of resource allocation / management and you end up with a problem of resource allocation / management compounded by problems with component version management, configuration management etc. pyenv falls into the second category of programs. The problem it's trying to solve is: install and use multiple versions of Python. There's really no need for an extra helper program to solve this problem. Multiple versions of CPython can be installed and used together without the use of any extra tools. I.e. the solution to this "problem" is simply to learn how to do that. Those who advocate for the use of pyenv and the likes usually make an argument for "simplicity". I.e. in their mind, not needing to know how to install multiple versions of Python is a bonus. Something that, potentially, saves them several hours of reading the documentation and perhaps saving them a tiny bit of typing when setting up Python initially. I contend that this calculation is off because it doesn't account for the problems down the lane. In other words: pyenv helps until it doesn't, and then it becomes a liability. Debugging is always more difficult if you have more wrappers between you and your problem. So, while individual users will not face a lot of problems with "wrapper solutions", those who service such users will face such problems a lot more frequently. That's why, as an ops person, I really dislike "wrapper solutions" -- for me, they complicate the task, never help.
  • maleldil 3y ago
    > Multiple versions of CPython can be installed and used together without the use of any extra tools. I.e. the solution to this "problem" is simply to learn how to do that. How do you do that? For example, if you're on Ubuntu LTS but want to use the latest Python version, how do you do that? The system package manager won't have it. Do you rely on a third-party PPA? What about other distros, where you don't have that?
• nxpnsv 3y ago
  I use this every day everywhere. Together with pyenv-virtualenv and poetry it dramatically improved my python experience…
  • claytonjy 3y ago
    if you're using poetry, how does pyenv-virtualenv fit in?
    • nxpnsv 3y ago
      I use poetry for dependency management and pyenv-virtual for venvs. I found this set up to work better for me than poetry's venv. Maybe things have changed, and I'm out of date on that, but now I'm just used to this setup....
• jolux 3y ago
  Plug for Mise (https://mise.jdx.dev/ https://mise.jdx.dev/) which is like asdf (which is like pyenv, but for any language you might want all in one tool) but written in Rust and without shims (by default) such that it’s much faster at running the activated runtime. It installs runtimes in parallel and will even download the correct plugin if you don’t already have it installed. It can also replace direnv, which I haven’t done because I’m using it as a drop-in replacement for asdf with the .tool-versions files my team already has in every repo (it supports .python-version as well if you’ve been using pyenv). It works great as a “better asdf”/“better pyenv” for my purposes.
• KingOfCoders 3y ago
  Migrated some small scripts from Python to Go because of the virtual env madness.
• savrajsingh 3y ago
  This is my favorite toolset -- pyenv and pyenv-virtualenv! So good. Such a great system imo. This is my preferred path through the forest of options here. :)
• chasenjohnson 3y ago
  This is the way. I moved everyone I work with at over to the pyenv, virtualenv, poetry stack. Once you get set up it’s pretty smooth sailing. It is a shame that so many tools are needed to work with python, but I was happy to finally find something that works.
  • Hackbraten 3y ago
    Curiously, post-checkout, how do you go about creating your venv and installing your project dependencies? The shortest ritual I could figure out so far is: pyenv install -s && pyenv exec pip install poetry && pyenv exec poetry install and I wonder what might be an easier incantation that still works.
• joshuamcginnis 3y ago
  I really love miniconda for managing python versions and deps myself.
  • tryptophan 3y ago
    Conda/Miniconda randomly breaks every few months or so when updating and I can't fix it. Idk why I stick with it, its been like this for years.
    • joshuamcginnis 3y ago
      Odd. I always do `conda activate base && conda update conda` on Mac and never have any issues.
• Decabytes 3y ago
  I recently used Pyenv to install an older version of Python on my system. The reason for this, is I wanted to create a Docker image using 3.8 slim-buster and I wanted to be explicit about the Python libraries I needed. By using Pyenv to install the same version of python as is used by slim-buster I was able to pip freeze my virtual env, and use the same version of libraries for the Docker image no problem. Oh the things I have to do in Python to avoid dependency hell
• RamiAwar 3y ago
  Pls dont. Just install several pythons with different binary names. ex. python3.8, python3.9, python3.11 No need to complicate things. If using Poetry, you can just do `poetry env use python3.8`.
  • scriptme 3y ago
    Here's all the project boilerplate you need with pyenv when working on a new project. > $ pyenv install 3.8 > $ pyenv local 3.8 > $ python3 -m venv .venv > $ source .venv/bin/activate and you're done. Not really sure what you might consider difficult about pyenv, but it's just a tool to instigate your python venvs which is a built tool for most modern versions of python.
    • 0x008 3y ago
      > Not really sure what you might consider difficult about pyenv - compiling from source - need to install build-essentials on docker images which takes up lot of space and takes a long time
  • Hackbraten 3y ago
    What do you do when (not if) your system package manager eventually drops that Python version from their core repositories? The good thing about pyenv is that it makes you rely less on package maintainers. You can come back to your Python project 10 years later and have a real chance that it still works, even if you have changed workstations in between.
• liveoneggs 3y ago
  Seeing pyenv inside of docker containers is always a treat
• MrYellowP 3y ago
  In windows I just use subst to switch python versions. Python always sits on drive P:. Never understood what all the fuzz about this was about. Sure, I have the same package on my disk multiple times, but if I cared about that there'd be easy ways to get around that problem just using OS functionality. From my perspective, it doesn't come easier than running a batch file to switch between python versions.
• cyrialize 3y ago
  I've been using Rye[0] lately, which has been pretty good. It's really just a wrapper around a bunch of underlying tools - it's nice to not have to worry about those and let Rye do it's thing. All that being said, the creator of Rye is 100% cognizant of that XKCD comic, this [1] is a nice read. I'm not super well versed in Python tooling at all. I've had to work a lot in Python in the past 6+ months, and I become super confused when I tried making a Python project in my spare time. I settled on Rye because it just seemed to be the easiest to use. [0]: https://rye-up.com/ https://rye-up.com/ [1]: https://github.com/astral-sh/rye/discussions/6 https://github.com/astral-sh/rye/discussions/6
• cgopalan 3y ago
  I used pyenv in my previous company. At first it was fine, but after a while with multiple versions of Python installed, things stopped working. The right virtual env was not activated, etc. (It could have also been because our software updates like OS and security upgrades were pushed by desktop support). I removed all of it, and just resorted to installing multiple versions of Python the old way (downloading from python.org) and then use `python<version> -m venv /my/virtual/env` to manage my virtual environments. Things are more stable, and I don't feel like its magic. I am not going back to pyenv.
  • andreif 3y ago
    Works fine for me on macOS. I am using venv the same way you do except some global ones for certain tools.
    • andreif 3y ago
      Also, when working on packages supporting multiple versions it was very helpful to use tox with pyenv.
    • cgopalan 3y ago
      I thought one of the advantages of pyenv was that you switched to also creating virtual envs using pyenv and managed virtual environments with it too. Maybe not.
      • andreif 3y ago
        Pyenv is for Python versions. There is a plugin for virtualenvs that can be useful for some specific cases but I rarely use it nowadays.
• Joel_Mckay 3y ago
  I wonder if it ever occurs to people the whole concept of versioned environments for compatibility is fundamentally a terrible design kludge. Someone needs to standardize a single Python environment at least annually. As these tools are just turning Python into a perpetual Alpha build. Best of luck, =)
• Karellen 3y ago
  I long for the day when python is mature enough for an implementation to arise where one binary can run as all previous versions of the language and stdlib, so you can do allpy --std=py3.4 and have it "just work", in the same sort of way that you can currently do gcc -std=c90 (And also therefore `#!/usr/bin/allpy --std=py3.4`) I mean, imagine if C compilers required that you have a separate compiler binary and copy of the standard headers (or even worse, all of stdlib!) for each version of the C standard?!? Instead of just putting new language features on version flags, and make stdlib functions available/marked as deprecated based on version checks?
  • martinky24 3y ago
    I wonder if "maturity" is the problem, or just different design goals. You speak like this is an inherent feature that comes with some magical level of "maturity", but if the feature isn't a legitimate design goal of the product, it isn't "immature" not to have it.
• zaptheimpaler 3y ago
  pip+venv mostly work fine for me, but there are some libraries like say numpy/scipy that are heavy so I want to share them across projects instead of installing multiple copies. I use pyenv to create a python version separate from the system version and install global dependencies into that. It's been useful for that and is just a nice way to manage multiple versions of python. So pyenv is a great addition to the traditional pip+venv setup. unrelated rant - Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts. I've resorted to making an "actual_requirements.txt" file manually listing only direct dependencies and whatever version constraints make sense. I wish pip would fix this.
  • akurtzhs 3y ago
    > I've resorted to making an "actual_requirements.txt" file manually listing only direct dependencies and whatever version constraints make sense. Pip-tools has an elegant solution for this. You write an requirements.in where you list packages and optionally versions, then pip-compile turns that into a requirements.txt with every package including dependencies specified by versions. You can use it to update specific packages as well.
  • globular-toast 3y ago
    > Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts. My friend, here is what you seek: https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools requirements.txt is flat because it's really the output of `pip freeze`. It's supposed to completely and exactly rebuild the environment. Unfortunately it's far too flexible and people abuse it by putting in only direct dependencies etc. If you're writing packages, you don't need a requirements.txt at all, by the way. Package dependencies (only direct dependencies) live in pyproject.toml with the rest of the package config. requirements.txt (and pip tools) are only for when you want to freeze the whole environment, like for a server deployment.
• airstrike 3y ago
  I've been using python on and off since Django 0.96, which was released in 2007. I don't recall ever needing anything other than virtualenvwrapper and pip—and even some of the annoyances these tools had early on have been solved by now... https://virtualenvwrapper.readthedocs.io/en/latest/ https://virtualenvwrapper.readthedocs.io/en/latest/ If you really need different versions of python, you can just `mkvirtualenv -p python3 venvname` I feel like every other tool out there has to explain what problem they solve that virtualenvwrapper doesn't Don't install anything globally, creates lots of envs, and feel free to have different versions of python installed side-by-side with some "main" version preferably symlinked as `python` and `python3` The end
  • packetlost 3y ago
    You don't even need virtualenv much less virtualenv wrapper. On Debian/Ubuntu distros (and most others, with small modifications) you just need python3-venv. Then create your venv with `python3 -m venv myenv` or `python3.11 -m venv my311venv` or whatever version of Python you need. Less is more!
    • airstrike 3y ago
      Agreed, but `workon` is too convenient not to have
      • kstrauser 3y ago
        I reimplemented the parts of `workon` I actually used with a small shell function.
  • cgopalan 3y ago
    You should look at the command `python -m venv`. Its built in, and is a breeze to create virtual environments. I guess it does not provide shortcuts for activation etc, but I am ok with that.
  • stairlane 3y ago
    Glad I’m not the only one. As soon as I started using Pyenv for local development I bloated my machine with more versions of python than I could manage. Obviously user error, but it never solved a problem I couldn’t get around with virtualenv.
• tortal 3y ago
  [dead]
• j7ake 3y ago
  Python package management is so broken. Think of the gymnastics that every python package has to go through in order for users to install it properly. Absolutely insane that this is still happening in such a widely used programming language.
• tapirl 3y ago
  A similar tool for Go: https://go101.org/apps-and-libs/gotv.html https://go101.org/apps-and-libs/gotv.html
• ijijijjij 3y ago
  to switch, I use on the command like "python", or "python3"... seems a lot easier to me and you dont even need an extra package.
• someotherperson 3y ago
  Was a pyenv user for the longest time, but I switched to Docker containers instead. I put together bash functions that take in a positional argument for the image and port (or randomizes it if not specified), and it automatically mounts the current working directory to the volume. Now I can just start a container without having to think about any of this stuff. And a config file per project if I don't want to pass in any variables at all.
• reddit_clone 3y ago
  Casual Python user. I found all these things very confusing. Then I found Pipenv. Sticking with it. (Emacs has support for this).
• neon5077 3y ago
  I really find it quite astounding just how much absolute nonsense python programmers are willing to put up with. Needing more than one, MAYBE two versions of your language installed is insanity. If I build a Python widget and send it to my non-programmer coworker, there is exactly zero chance it will work until I walk over and manually set up the correct language. Instead I use a language that natively compiles to an exe. Python is pretty neat, but the concept of a program that just works anywhere is so utterly alien that I genuinely cannot find any practical use for it. I can't just install a pip package to use globally, I have to set up some goddamn virtual environment because everything is so brittle that nothing works without the exact correct language version. It's like NPM all over again. Dependency hell is so bad that your package manager is now a dependency so we need a package manager manager to manage installing the correct package manager version to then install the correct packages. Every single time I've come up to some Python thing or other, I spend about fifteen minutes fucking with it before giving up and using a tool built in a sane language. The fact that I can't just send any random person a program and expect it to run at all is just lunacy. And no, compiled Python does not count because that's even more brittle tooling on top of all the other bullshit I have to deal with. If it doesn't work out of the box, it's a bad tool and I have far better things to do with my very valuable time.
  • afiodorov 3y ago
    I've been programming in Python for years and I agree: Python isn't really for consumers. But hey it is used for LLMs that can generate React apps, so there's that. I am only half-kidding. But it appears the industry as a whole is concentrating effort on fewer languages and Python is one of them. If I want to distribute my app - I write it in React, but I still often use python in the process: prototyping, asking GPT to rewrite X from python to typescript, making mock servers for tests / preparing data for unit tests.
  • globular-toast 3y ago
    You're just using the wrong tool for the job.
  • andreif 3y ago
    It’s totally up to your knowledge and experience to produce code that supports multiple versions. I had no issue with doing it. That’s what you pay for dynamic language that evolves rapidly.
• jrjarrett 3y ago
  Why not just use a tool like asdf (https://asdf-vm.com/ https://asdf-vm.com/) or mise (https://mise.jdx.dev/ https://mise.jdx.dev/)? These tools have the advantage of not being multi-taskers and can manage version for all your tools. You wouldn’t need pyenv and npm and rvm and… We’ve even started committing the .mise.toml files for projects to our repos. That way, since we work on multiple projects that may need multiple versions of the same tool, it’s handled and documented.