6 ms·

Asking sincerely: With the various tools like pyenv, poetry, virtualenv, venv and the likes, what seems to be different? What seems to be the best practices tod

• user5678 4y ago
  [dead]
• masklinn 4y ago
  Most of the tools you mention don't even overlap. - pyenv is used to install multiple versions of python (and even distributions e.g. you can install conda via pyenv), independent of the distribution, it can have a plugin to more easily manage virtualenvs for those but that's optional - virtualenv and venv are largely the same thing, the latter is a subset of the former merged directly into python, it's used to create isolated sets of installed packages (site-packages), as installing everything globally is messy and inconvenient and can be problematic when two project have different requirementws - poetry is a packaging and dependency management tool, that is definitely the messiest and most confusing part of the ecosystem as there is a lot of stuff and movement, frankly I don't really get it, and the entire thing annoys me to no end
  • notpushkin 4y ago
    Personally, I find Poetry really nice. It abstracts away pip and venv, but in a way that you can still understand what's going on and jump in if you need. Once you understand that it doesn't do much else, it is easy.
  • srvmshr 4y ago
    Can you please explain the pyenv usecase a bit more? Let us say, I needed Py3.6 for something - I can just use virtualenv with appropriate flags to install 3.6 tooling. How does PyEnv help further. I have read often pyenv helps maintain different version of python on same system. But if that is also possible by virtualenv/venv what am I missing?
    • noitpmeder 4y ago
      Venv helps maintain separate virtual environments for a given python installation, but it won't actually install python. E.g. if you have Python3.6 installed venv can use it, but you can use PyEnv to install 3.7, 3.8, 3.9, ... Which venv can then use.
      • coldtea 4y ago
        Basically Python team sat down to consider how they can make their deployment setup worse, and they came up with the idea to split packaging, dependency management, virtual environments/isolation, and Python version management to each need a different too, and then had different random projects make several tries at of each tool, all at more or less equivalent popularity...
        • InfamousRece 4y ago
          I was told the opposite - that the current mess is because the “Python team” did not sit down to consider deployment setups at all.
    • woodruffw 4y ago
      You're missing that venv and virtualenv can't do that: they can only create environments for your current Python interpreter, not arbitrary Python versions. There's no flag that I'm aware of that makes venv install a different Python version than the one it was invoked with.
      • aiiane 4y ago
        The --python flag for virtualenv will let you specify a different Python executable to use, but requires that executable to already be installed somewhere on the system.
        • georgyo 4y ago
          Getting another python executable is normal the challenge though.
  • MrJohz 4y ago
    Poetry is basically NPM, Cargo, or Composer, but for Python, and my impression is generally that people who have used those tools see it as incredibly valuable and useful, and people who haven't used those tools don't really understand what the fuss is about. (Disclaimer, I am in the former camp.) It's a way of wrapping all the different facets of maintaining a package and its dependencies into a single tool, which as much of the details transparently taken care of. So for example, if I clone a new project, and run `poetry install`, then: 1. A new venv is just automatically created, I don't need to think about accidentally clobbering my global Python installation, but I also don't need to think about creating my own venv every time. It just happens automatically. 2. All of the packages I need will be resolved and installed as I expect. Additionally, because there should already be a lock file in the repository, I will be installing the same versions as everyone else working on the same project, meaning we're not going to run into weird dependency issues. 3. I can automatically activate and run the environment with ease to run the test or linting tools that I need. 4. I can create wheels and even distribute them to PyPI or other repositories as necessary. 5. I can update the dependencies in the project based on semver requirements, but also individually, on a case-by-case basis, and go back to previous "known good" easily via git. But most of all: 6. If I switch to a different Poetry-based project, I can do all the same things in the same way, using exactly the same standardised interface. Which means that I can just jump into new projects, support other open source tools and libraries, join other teams, etc, with a significantly reduced onboarding time. No more funky Makefiles, no more figuring out which tool I'm missing: just use Poetry, and everything is consistent, and it Just Works™. FWIW, it's not perfect. It's not as powerful as package managers for other languages, so there's a handful of use cases like workspaces for multiple projects that are not well supported. It's also not officially blessed by the Python maintainers in the way that Cargo or NPM are, which means it's always slightly more of a risk, and means people need to install it before they use your project. Also even if it were perfect, brilliantly maintained, and used by nearly everyone, there's still a handful of use cases where it would break down - even with Cargo, some projects still have manual calls to rustc to handle their specific use cases. To me, right now it's an 80% solution (in the sense that 80% of projects will find 100% of their needs satisfied by it). I think it could become a 95% solution if it came popular enough, and if there were more core Python support. (Or alternatively: an alternative that was officially maintained by Python maintainers like venv or pip could be a 95% solution.)
• recuter 4y ago
  venv is a bundled virtualenv subset. It came after it. Now you don't need to install virtualenv since venv comes bundled with python. Don't concern yourself. pyenv, poetry are wrappers and do different things. You would use pyenv to create a virtual environment. Or venv directly. Or the older virtualenv tool. Stick to pyenv as it is easiest. You can setup your shell to activate/deactivate automagically for you when you cd into your project directory using something like: https://github.com/direnv/direnv https://github.com/direnv/direnv I would avoid poetry. Edit: bah, somebody answered already tl;dr - Setup direnv and pyenv once and foggetaboutit.
  • srvmshr 4y ago
    I have read on occasion that pyenv seems to be woefully unmaintained. Is that true? Is it a good move to learn using poetry instead. Another observation I have is that although pip installs in environment are sanitized, they share a global cache (or tarball dump under .cache/ e.g). Is there any way, pip packages can be further compartmentalized? Is there any merit to pipx that I hear mentioned at times. Thank you for answering
    • recuter 4y ago
      pipx is for installing python scripts as cli tools, as an end user. Say there is a thing called ytb-dlp that lets you download YouTube videos. Instead of creating a virtualenv for it yourself, activating it, pip installing this thing inside it, and then having to cumbersomely use it like that.. pipx install ytb-dlp. It will do all of that for you and put it on your path. > I have read on occasion that pyenv seems to be woefully unmaintained. If that is true, is it a good move to learn using poetry instead. No.
      • srvmshr 4y ago
        That explains much better: pipx is basically python cli related ops. Thanks for the ELi5 explanation
    • masklinn 4y ago
      [flagged]
• flashgordon 4y ago
  I had the same question. I also just stick with venv with diff py versions. What does pyenv give that venv doesn't? Similarly when you are used to a venv + pip why poetry? Oh in all this I now see nix nudging it's way in and I am feeling a bit anxious I a missing out on something!
  • masklinn 4y ago
    > What does pyenv give that venv doesn't? A way to install those different py version. That's literally its entire purpose.
• theptip 4y ago
  I recommend pyenv, pyenv-virtualenv, and poetry for Mac development these days. It’s a nice modern experience. If you don’t care about different Python versions then the built in “python3 -m venv” is fine and you can drop pyenv. You can use Poetry to make and manage virtual envs too. If you don’t need lock files (say you are doing ML or solo scripting) then pip with requirements.txt is fine. Lock files start to become useful when you are collaborating with a team so you know you have exactly the same deps (so theoretically the same behavior).
  • crabbone 4y ago
    Venv isn't built into Python. You are using Python distributions intended for developers, that's why you will usually have it. Python can be configured to create a distribution without it (which is what typically happen on Linux, where Python is part of the distribution). Pip with requirements.txt is never fine. It's always the wrong answer to any problem you may think you are solving with it. The problem is it's a very commonly given wrong answer, so, people are used to it. The reason why it's the wrong answer is this: You intend for your project to be deployed somehow somewhere. So, you do need a process to find the pieces of the project you are working on to put them where you are deploying them, be it a Docker image, a Python package, an RPM etc. But, pip + requirements.txt doesn't produce any of that. At best, you may hope to have it as an intermediary step in your process. Now, a very important piece of your deployment is the code you wrote. That's something that's not in the requirements.txt. You sort-of have "pip install -e ." to do this, but it's awfully bad for this purpose because instead of actually installing anything, it's a wrapper around "setup.py develop", which, in turn doesn't install anything, but modifies Python's path where it searches for installed packages. This has a lot of bad / unintended consequences. For example, if you accidentally introduced a dependency on an artifact on your filesystem, it may or may not translate properly into your deployment environment. This is especially bad when you need to use Python code linked with some native libraries, but on your system and on deployment system they are found in different places. It may result in you confusing the libraries (and versions) you think you are using with those actually being used. In order to avoid all of that, a better way is to create a distributable Python package (i.e. Wheel) and install that, but Wheel has its own way of specifying dependencies, which has nothing to do with requirements.txt. So, you either just don't need that, or try to combine the two somehow... ---- Bottom line: I package couple dozens Python packages for internal use. Those I have full control over don't have requirements.txt and don't use pip at all. They are easy to install and to test, portable between different environments etc. There are teams which produce their own Python packages which I have to package, and in many cases I wasn't able to convince them to abandon the bad (but popular) practices s.a. using pip or / and requirement.txt. This is a constant source of problems when it comes to dependency management. But, authors are convinced they are doing it right...
• crabbone 4y ago
  If you use conda, you are in a different world, so, none of what you listed above is of any use to you. But, say, you wanted to go from Anaconda Python to what I call PyPI Python, then there are two distinct ways. Way 1: You prefer consensus over quality Python infrastructure tooling is god awful. It's so awful it's on the level of its own. You just don't get things so bad from other popular tech. Python also has a community that's driven by personalities and dogma, because most of Python community is very bad at programming, so, instead of deciding for themselves the community is dedicated to following "best practices" which they don't understand, but which guarantee that a randomly selected person from the same community will be able to use your project. If this is your case, then strike out poetry and virtualenv from your list, and that's the typical setup. For people who are afraid of compiling Python pyenv provides a workable solution, while venv is the most popular tool for creation of virtual environments mostly because it's bundled with Python binary distributions (or those built by pyenv). Pyenv has its own virtual environments, but there's no reason to use those. Poetry is a project that is mostly dedicated to installing Wheels, which is Python packages. It also has its own virtual environments. The problem with Poetry is that it can only deal with a subset of all packages that may be installed using other tools, i.e. pip and setuptools. It's somewhat better organized, but because it's a non-solution for a large fraction of problems in its domain, it will only work for you if you can make sure you will never need anything from that fraction. Virtualenv is just a Python 2.X thing. You are unlikely to need it today. Basically, it does what venv does, however venv doesn't support Python 2.X. Way 2: You want it to work well I have bad news for you... that's not going to happen. Yet, you can improve your quality of life by ditching pyenv. There's no reason to use it, if you aren't afraid of building Python from source. That's how I always do it, and that's what allows me to make debug builds, if I need those, or apply patches if I need them, or simply build slimmer Python distributions, if I don't need some features (s.a. tkinter, or image processing libraries etc.) ---- NB. venv isn't baked into Python 3.X. You may choose to build a distribution w/o it (which is often the case for Python installed with Linux distros), but it's common in distributions intended for developers.
• jillesvangurp 4y ago
  I've been using miniconda as well recently to beat some sense into python. Conda is nice for local development. And then I package it up using docker. Dockerized development environments are of course nice as well and I plan to do more with those. Conda seems a bit heavy weight in comparison. It seems you end up with all these layers of half-assed virtualization with python these days. So, I used conda install python and pipenv. And then you pipenv shell and install some dependencies. Seems convoluted to me but it works. Of course none of this is relevant when you use docker. Kind of the whole point of docker. Isolate everything at the file system level and it stops being a problem that python wants to barf files all over it and make an unholy mess of things. The proper fix would be to not do that and be more like go, rust, and others and apply some hygiene with respect to where things go (i.e. not in some global directory). Would that be such a massive change for python to not do that?
• BiteCode_dev 4y ago
  After 10 years using all of them, in about 50 companies (as a freelancer) with various constraints, contexts, systems and skillsets involved, here are my definitive conclusions: - stick to regular python installers from python.org if you can. Don't use homebrew, pyenv, conda, the windows store, asdf, etc. They have subtle and least subtle variations that will bite you. - always use "-m" when you run a python tool, even in venv. This will let you be explicit about which python you use. On windows, use the "py" command to create the venv. No need for pyenv. - if you are in linux, research how to install python seriously. If the command is simple, you probably got it wrong. E.g: Ubuntu has 3 packages to install, not one. People giving you a one true way are lying to you. Pyenv is a trap. - learn to use basic venv and pip. Never do something outside of a venv, even for small scripts. You can have one big venv for all your small scripts. Don't even install black outside of a venv. You can get very far with just that. Forget about pipx. - if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not flint. Not pdm. Not anything else. They will fail you. Pip-tools is just good enough and just reliable enough. Embrace manual venv handling. It's not a big deal. Automated tools will betray you. - learn to be comfortable with the PATH and the PYTHONPATH. No matter what you do, something may need tinkering. - tox is indeed the best tool to test several envs, but people that need it are rare. Don't bother for now. - if you want to build a package, use hatch. But you probably don't. - if you have to use conda, don't use pip or venv at all. Don't use piptool, poetry or anything. Stay on pure conda land. It sucks, but the alternative of mixing them is worse.
  • antman 4y ago
    How do you handle multiple python versions? I have found pyenv the simplest cross system approach
    • BiteCode_dev 4y ago
      On mac and linux, python executables are suffixed, on windows, use the py command (hence using the official installer). Once you have created the venv, it doesn't matter.
  • ActorNightly 4y ago
• noloblo 4y ago
  python3 -m venv. . bin/activate pip install dependencies