7 ms·
Testing multiple Python versions with nox and pyenv
- thenipper 4y agoMy one complaint about nox is the last time I used it you couldn’t run tests in parallel which while not a blocker is somewhat annoying.
- jw_cook 4y agoLooks like there's a discussion about running sessions in parallel here: https://github.com/wntrblm/nox/issues/544 https://github.com/wntrblm/nox/issues/544 I also think that would be a good addition, although most of the time I would probably still prefer to run sessions in serial, and parallelize tests within each session using pytest-xdist. That gives you fairly fine-grained control over how tests are distributed among workers, grouped either by test function, class, module, file, or arbitrary groups using a decorator: https://pytest-xdist.readthedocs.io/en/latest/distribution.html https://pytest-xdist.readthedocs.io/en/latest/distribution.h...
- claytonjy 4y agoIs there a good explanation as to why python-level tools like nox and poetry seem to avoid integrating with pyenv? Pyenv seems to be the defacto python version manager, and Pipenv seemed to integrate well with it. Why has the python tooling community moved away from it?
- nerdponx 4y agoI don't think it's for any particular reason, other than that nobody has done it yet. I would imagine that the response of most projects would be "PRs accepted", unless they consider it explicitly out of scope. I could see some projects not being interested in automatically installing versions using Pyenv. But I think most projects wouldn't mind the option to defer to Pyenv before searching PATH, or to otherwise configure a custom Python search path that could be derived from Pyenv and/or other sources. Note that Pyenv is itself basically a pile of shell scripts, and it doesn't work on Windows (although there is a Windows port of it). There might also be a general expectation that this functionality is too complicated to support in a way that makes everyone happy, and that users need to figure it out on their own, e.g. using the technique described in the article here. You can also go the other way and use the Pyenv-Virtualenv plugin.
- tempest_ 4y agoMost people using poetry do not care about python versions outside the one they are currently using. It is a concern for library producers but they are definitely the minority.
- drewcoo 4y agoIt seems strange to me that a language that proudly proclaims there is one way to do it seems to embrace TIMTOWTDI at the edges. My uninformed guess is that Guido didn't consider how to install anything to be part of the language so it didn't get much scrutiny. https://idioms.thefreedictionary.com/TIMTOWTDI https://idioms.thefreedictionary.com/TIMTOWTDI
- crabbone 4y agoPython never followed that or any other absolutist claim. Python is a huge ball of yarn comprised of donations of, perhaps hundreds individual developers w/o strong ties to each other. If you look inside the project, you find plenty of distinct styles and approaches coexisting and overlapping. The moderation is very weak / non-existent. Installation utilities, however, were in Python since very early on. "distutil" dates back to, probably, one of the first public releases. It's since then was always part of the "standard" library, but it's going to be removed in the name of chaos, anarchy and entropy in one of the upcoming releases. There was never any great design in Python in general, or in distutil in particular, but through a series of bad decisions, which were quickly carved in stone, we ended up with the system we have today. Setuptools was a first community project which grew out of recognition of inadequacy of distutil, but, instead of trying to replace it, it was designed to be a drop-in replacement in terms of interface and was made to depend on distutil in order to do anything useful. This result in setuptools being a lot more convoluted and rigid than desired. Few people wanted to participate in this project, and so it was proclaimed dead several times, setuptools2 was supposed to replace it etc... but the decisions carved in stone wouldn't allow anyone to really dislodge it. Gradually setuptools grew more hair, another pair of horns and wings by merging with distribution, easy_install, resouces_pkg and a bunch of other smaller evils, making itself ever more indispensable. Somewhat in parallel to that process, pip was starting to pick up speed. Pip promised a lot, but never delivered on those promises (eg. setuptools was never able to uninstall packages, while pip has "uninstall" command, it's broken by design). And then anyone who made any sense abandoned Python. Python community was left with Python Foundation people or PyPA in this instance, who are not competent to deal with packaging. The entire activity today around packaging in Python may be described as chaos. It's adding random features based on the assumption that existing features work when they don't. Trying to get rid of the only useful tools while replacing them with tools that rely on the tools being replaced etc. It reads like some sort of an absurdist novel if you have to dive deep into it. And yet, it's a technology used by millions, shuffling trillions of goods and services around the world...
- deleted 4y ago[deleted]
- jdxcode 4y agopyenv is really slow. It'll increase the time it takes to run `python` by a ton. For me it makes `python -V` take ~120ms but without pyenv it only takes ~13ms. I wrote an alternative that has 0 overhead called rtx. https://github.com/jdxcode/rtx https://github.com/jdxcode/rtx
- infamia 4y agoThis is amazing! Thank you for this wonderful project. It looks like you've taken care many issues I've experienced with asdf (startup performance, shims, bad/confusing commands, etc.). It looks like I might not need to add anything to .*rc files if rtx is in my path? Either way, I can't wait to try it!
- jdxcode 4y ago> I might not need to add anything to .*rc files if rtx is in my path Generally speaking you'll probably want to use `rtx activate` in your shell rc. However you could use the shims instead if you just wanted to add a directory to PATH instead. Of course you'd be getting rid of one of the major selling points of rtx by using shims, but it's so much faster than asdf the shims aren't as bad as they are in asdf.
- crabbone 4y agoThere's no reason to use pyenv, pipenv or nox at all, ever. Whether they want to cooperate with each other, or stand together and cry in a corner -- the community shouldn't care. The community should recognize how little value these tools offer (more like negative value, really), and demand from the people responsible for this part of tooling (i.e. PyPA) to get off their rear and make themselves useful, or step down and clear the way for people who know how to do this stuff. The proliferation of useless tools that advertise themselves as solutions to unnecessary problems created by the core of Python development simply shouldn't be happening. If Python community wants to overcome the ridiculous state it's in right now, it needs to concentrate on fixing the problems at their core, not on creating another layer of duct tape around the rotten core.
- BiteCode_dev 4y agoPyenv compiles python. This excludes it for pretty much every solution by default, as compiling can fail in so many weird ways. Nobody wants to support something that can turn even the simplest ticket into a game of cluedo.
- nerdponx 4y agoThis is a great Pyenv usage tip in general. Note that Hatch supports parameterized environments much like Nox, although it's not as freely configurable. See https://hatch.pypa.io/latest/environment/#matrix https://hatch.pypa.io/latest/environment/#matrix This is particularly nice because your build tool is also your test/script runner, meaning you have one less dependency to manage. The Pyenv technique described here should work as well with Hatch as with Nox.
- srvmshr 4y agoAsking sincerely: With the various tools like pyenv, poetry, virtualenv, venv and the likes, what seems to be different? What seems to be the best practices today for maintaining projects? I would be grateful for insights. (Full disclosure: I just use conda in ML & for any lightweight work, I defer to venv which seems to be baked in with Python 3. I have never tried hard to wrangle with this conundrum & many folks are happy with choosing something and sticking to it without asking "why")
- user5678 4y ago[dead]
- masklinn 4y agoMost of the tools you mention don't even overlap. - pyenv is used to install multiple versions of python (and even distributions e.g. you can install conda via pyenv), independent of the distribution, it can have a plugin to more easily manage virtualenvs for those but that's optional - virtualenv and venv are largely the same thing, the latter is a subset of the former merged directly into python, it's used to create isolated sets of installed packages (site-packages), as installing everything globally is messy and inconvenient and can be problematic when two project have different requirementws - poetry is a packaging and dependency management tool, that is definitely the messiest and most confusing part of the ecosystem as there is a lot of stuff and movement, frankly I don't really get it, and the entire thing annoys me to no end
- notpushkin 4y agoPersonally, I find Poetry really nice. It abstracts away pip and venv, but in a way that you can still understand what's going on and jump in if you need. Once you understand that it doesn't do much else, it is easy.
- srvmshr 4y agoCan you please explain the pyenv usecase a bit more? Let us say, I needed Py3.6 for something - I can just use virtualenv with appropriate flags to install 3.6 tooling. How does PyEnv help further. I have read often pyenv helps maintain different version of python on same system. But if that is also possible by virtualenv/venv what am I missing?
- axpy906 4y agoI use tox extensively. Nox looks far easier to configure.
- nerpderp82 4y agoI use tox with a barebones ini file, haven't run into any issues that would necessitate a rewrite or a fork. Python feels like it has become JavaScript with a plethora of tools that all do the same things poorly. https://tox.wiki/en/latest/config.html https://tox.wiki/en/latest/config.html
- claytonjy 4y agoYou're not wrong about the proliferation of tools, but I'm hopeful it's a necessary waypoint to consolidate on fewer tools. Take dependency managers; for a long time there was just virtualenv+setuptools, then came Conda, then Pipenv. That was it for awhile, until Pipenv got displaced by Poetry. Now we have at least three others vying for mindshare: hatch, PDM, and Pyflow. They all have a ton of overlap with Poetry, but they're all a bit different. I think venv/Conda/Pipenv was a tolerable set: barebones, heavyweight for ML, and a tool that feels like what other langs have. I don't think we can handle 4 Pipenv replacements for long, so I hope we see some of these tools replace eachother in the coming years. I'm not sure if the standardization of pyproject.toml encourages or discourages this proliferation, but I think it's a huge win for the community vs setup.py.
- nerpderp82 4y agoI feel like a boomer just compiling Python from source and using venv/pip/setuptools/wheel until all the stupidity blows over. I wasn't even aware of hatch, PDM and Pyflow. Every new tool is just trolling the Python community. I even submitted a couple pip patches, never again.
- deleted 4y ago[deleted]
- qwertox 4y ago`pyenv` was recommended to me as a way to decouple my Python from the OS's one, in order to be able to run a more current one. But it has been irritating me more that benefiting me, because apparently I need to recompile Python on every new patch release, also for ARM on a RasPi, which takes forever, and then manage the virtual environments related to it. I can't see it as a solution to my problem. My problem was that I was using `virtualenv` and when I upgraded Ubuntu 20.04 to 22.04 it blew up all my projects until I recreated all the virtual environments. And I have a lot of them, at least three dozens dispersed on around 15 Linux installations. So I need to find a way to either migrate everything automatically via a script, or use something like `pyenv` to lock my projects to a specific version, so currently I'm holding back a couple of 20.04 installations. I don't know what to do, this kind of exploded on me, I should have seen it coming.
- ypeter 4y agoAre there any advantages of using asdf instead of Pyenv for Python version management (or disadvantages)?
- frankensteins 4y agoMy current system is using Python3.11 For virtualenv, you can create python3.10 using virtualenv SOME_PY_ENV_NAME -p python3.10 This is the result: which python3 /Users/user_name/tmp/SOME_PY_ENV_NAME/bin/python3 python3 --version Python 3.10.10
- frankensteins 4y agoI was wrong here.