5 ms·

After 10 years using all of them, in about 50 companies (as a freelancer) with various constraints, contexts, systems and skillsets involved, here are my defini

After 10 years using all of them, in about 50 companies (as a freelancer) with various constraints, contexts, systems and skillsets involved, here are my definitive conclusions: - stick to regular python installers from python.org if you can. Don't use homebrew, pyenv, conda, the windows store, asdf, etc. They have subtle and least subtle variations that will bite you. - always use "-m" when you run a python tool, even in venv. This will let you be explicit about which python you use. On windows, use the "py" command to create the venv. No need for pyenv. - if you are in linux, research how to install python seriously. If the command is simple, you probably got it wrong. E.g: Ubuntu has 3 packages to install, not one. People giving you a one true way are lying to you. Pyenv is a trap. - learn to use basic venv and pip. Never do something outside of a venv, even for small scripts. You can have one big venv for all your small scripts. Don't even install black outside of a venv. You can get very far with just that. Forget about pipx. - if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not flint. Not pdm. Not anything else. They will fail you. Pip-tools is just good enough and just reliable enough. Embrace manual venv handling. It's not a big deal. Automated tools will betray you. - learn to be comfortable with the PATH and the PYTHONPATH. No matter what you do, something may need tinkering. - tox is indeed the best tool to test several envs, but people that need it are rare. Don't bother for now. - if you want to build a package, use hatch. But you probably don't. - if you have to use conda, don't use pip or venv at all. Don't use piptool, poetry or anything. Stay on pure conda land. It sucks, but the alternative of mixing them is worse.

antman 4y ago
How do you handle multiple python versions? I have found pyenv the simplest cross system approach
- BiteCode_dev 4y ago
  On mac and linux, python executables are suffixed, on windows, use the py command (hence using the official installer). Once you have created the venv, it doesn't matter.
ActorNightly 4y ago
Couple of points/expansions >if you are in linux, research how to install python seriously. I.e, build from source, after installing necessary dependencies (https://devguide.python.org/getting-started/setup-building/#build-dependencies https://devguide.python.org/getting-started/setup-building/#...). Use sudo make altinstall to install the specific version. Usually distro provided ones are pretty old anyways, so you want the faster 3.11 and on. >Never do something outside of a venv. Not even install black. You can get very far with just that. If you want to just write scripts to test things, working outside venv is fine as long as you manually install pip using get-pip.py, then invoke pip using python3.xx -m pip (make a bash alias for this command, and then delete the binaries in ~/.local/bin). This will install all the binaries to ~/.local/lib/python3.xx/site-packages, which keeps everything isolated. >if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not anything else. Also, stick to setup.py for module installation definition. It keeps everything in one place and allows you to run custom workflows. For example, if you wanna run flake8 on install to check your code, with pyproject.toml, it requires pyproject.toml and setup.cfg. With setup.py, you can do this all in one file. Not only that, it gives you extra options to run certain commands. >if you want to build a package, use hatch. setuptools is fine for building a wheel - all you are doing is moving files around and zipping them. As far as OS goes, if you are on Windows, use WSL2 and run python there. Don't even bother with Windows python. If you are on mac, use `asdf` for python installation.
- BiteCode_dev 4y ago
  I would advice against all of those recommandations. It would be very long to explain point by point, but in short, my experience is that if someone is asking OP's question, they don't have the experience to deal with the modes of failure of what you are proposing There is not a lot of things I'm very adament about. Yet when it comes to python env setup, I have a strong opinion on the matter, built from seing hundreds of people failing at it in a thousand ways. Looking for the best solution will not work. You have to go to the minimal one that does the job, which have the least possible ways to fail. And still work in 5 years. Most people like you, giving advices on HN, know too much about it to realize you solve lots of problems without even thinking when you perform your workflow. But the average python user don't have this knowledge. They just want to get things done and have something that works. I am very confident in these claims, although a HN comment is not the best format to demonstrate it. I should write an article on it honestly at this point because so many devs are suffering for this. A new team has been put in place to work on packaging officially in python. But the fruits will take years to mature.
c7THEC2DDFVV2V 4y ago
Similar experience, but disagree with almost everything you have said. - stick to regular python installers. Don't use homebrew, pyenv, conda, the windows store, etc. For me pyenv has been a lifesaver, easier to manage multiple versions. Are you manually managing multiple python versions? - always use "-m" when you run a python tool, even in venv. This will let you be explicit about which python you use. On windows, use the "py" command to create the venv. Unsure how I feel about this, I do normally point direct to the binary being used rather than using "-m", Although "-m" still requires you to be in the correct venv. How does it allow you to be specific about the python you use though? - if you are in linux, research how to install python seriously. If the command is simple, you probably got it wrong. E.g: Ubuntu has 3 packages to install, not one. People giving you a one true way are lying to you. Pyenv is a trap. poetry has been another lifesaver that removes the boring writing of a setup.{py,cfg}. Pretty much everything can be managed through it, and it feels like a legit build tool for python. No more broken requirements because of a differing versions in transitive dependencies. - learn to use basic venv and pip. Never do something outside of a venv. Not even install black. You can get very far with just that. Can't disagree with this, although poetry will manage your venvs for you. Understanding venv and pip is useful, but not fundamental. Greatest success for devs has been getting them away from manually managing things, by and large perm staff don't care about learning lots of tools well, it's just a job. - if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not flint. Not pdm. Not anything else. They will fail you. Pip-tools is just good enough and just reliable enough. Embrace manual venv handling. It's not a big deal. Automated tools will betray you. lumping poetry in with pip-tools is weird. it's not a replacement, it's a tool with greater scope than pip-tools. pip-tools provides a subset of poetry functionality i.e. dependency management. Manual handling is useful to learn, but is laborious after a while, hence tools such as poetry. - learn to be comfortable with the PATH and the PYTHONPATH. No matter what you do, something may need tinkering. This is a really interesting point, normally modifying PYTHONPATH is a smell in my experience, but yes is good to understand. Understanding the PATH in general is more useful as a dev than specific to python imo. - tox is indeed the best tool to test several envs, but people that need it are rare. Don't bother for now. Agreed, its a nice thing to do, but rarely do I see it unless you are building libs, and need to support multiple versions. - if you want to build a package, use hatch. But you probably don't. Why would you use hatch over a tool such as poetry, are they equivalent? I'm interested in exploring hatch, but the comparisons I have found seem to be fundamentally misinformed about poetry so its hard to take comparison seriously. - if you have to use conda, don't use pip or venv at all. Don't use piptool, poetry or anything. Stay on pure conda land. It sucks, but the alternative of mixing them is worse. This is a really good point, but is always a frustrating thing to try and solve time and time again, but still encounter problems.

antman 4y ago

How do you handle multiple python versions? I have found pyenv the simplest cross system approach

BiteCode_dev 4y ago
On mac and linux, python executables are suffixed, on windows, use the py command (hence using the official installer). Once you have created the venv, it doesn't matter.

ActorNightly 4y ago

Couple of points/expansions >if you are in linux, research how to install python seriously. I.e, build from source, after installing necessary dependencies (https://devguide.python.org/getting-started/setup-building/#build-dependencies https://devguide.python.org/getting-started/setup-building/#...). Use sudo make altinstall to install the specific version. Usually distro provided ones are pretty old anyways, so you want the faster 3.11 and on. >Never do something outside of a venv. Not even install black. You can get very far with just that. If you want to just write scripts to test things, working outside venv is fine as long as you manually install pip using get-pip.py, then invoke pip using python3.xx -m pip (make a bash alias for this command, and then delete the binaries in ~/.local/bin). This will install all the binaries to ~/.local/lib/python3.xx/site-packages, which keeps everything isolated. >if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not anything else. Also, stick to setup.py for module installation definition. It keeps everything in one place and allows you to run custom workflows. For example, if you wanna run flake8 on install to check your code, with pyproject.toml, it requires pyproject.toml and setup.cfg. With setup.py, you can do this all in one file. Not only that, it gives you extra options to run certain commands. >if you want to build a package, use hatch. setuptools is fine for building a wheel - all you are doing is moving files around and zipping them. As far as OS goes, if you are on Windows, use WSL2 and run python there. Don't even bother with Windows python. If you are on mac, use `asdf` for python installation.

BiteCode_dev 4y ago
I would advice against all of those recommandations. It would be very long to explain point by point, but in short, my experience is that if someone is asking OP's question, they don't have the experience to deal with the modes of failure of what you are proposing There is not a lot of things I'm very adament about. Yet when it comes to python env setup, I have a strong opinion on the matter, built from seing hundreds of people failing at it in a thousand ways. Looking for the best solution will not work. You have to go to the minimal one that does the job, which have the least possible ways to fail. And still work in 5 years. Most people like you, giving advices on HN, know too much about it to realize you solve lots of problems without even thinking when you perform your workflow. But the average python user don't have this knowledge. They just want to get things done and have something that works. I am very confident in these claims, although a HN comment is not the best format to demonstrate it. I should write an article on it honestly at this point because so many devs are suffering for this. A new team has been put in place to work on packaging officially in python. But the fruits will take years to mature.

c7THEC2DDFVV2V 4y ago

Similar experience, but disagree with almost everything you have said. - stick to regular python installers. Don't use homebrew, pyenv, conda, the windows store, etc. For me pyenv has been a lifesaver, easier to manage multiple versions. Are you manually managing multiple python versions? - always use "-m" when you run a python tool, even in venv. This will let you be explicit about which python you use. On windows, use the "py" command to create the venv. Unsure how I feel about this, I do normally point direct to the binary being used rather than using "-m", Although "-m" still requires you to be in the correct venv. How does it allow you to be specific about the python you use though? - if you are in linux, research how to install python seriously. If the command is simple, you probably got it wrong. E.g: Ubuntu has 3 packages to install, not one. People giving you a one true way are lying to you. Pyenv is a trap. poetry has been another lifesaver that removes the boring writing of a setup.{py,cfg}. Pretty much everything can be managed through it, and it feels like a legit build tool for python. No more broken requirements because of a differing versions in transitive dependencies. - learn to use basic venv and pip. Never do something outside of a venv. Not even install black. You can get very far with just that. Can't disagree with this, although poetry will manage your venvs for you. Understanding venv and pip is useful, but not fundamental. Greatest success for devs has been getting them away from manually managing things, by and large perm staff don't care about learning lots of tools well, it's just a job. - if you want to have better tooling, use pip-tools. Not pipenv. Not poetry. Not flint. Not pdm. Not anything else. They will fail you. Pip-tools is just good enough and just reliable enough. Embrace manual venv handling. It's not a big deal. Automated tools will betray you. lumping poetry in with pip-tools is weird. it's not a replacement, it's a tool with greater scope than pip-tools. pip-tools provides a subset of poetry functionality i.e. dependency management. Manual handling is useful to learn, but is laborious after a while, hence tools such as poetry. - learn to be comfortable with the PATH and the PYTHONPATH. No matter what you do, something may need tinkering. This is a really interesting point, normally modifying PYTHONPATH is a smell in my experience, but yes is good to understand. Understanding the PATH in general is more useful as a dev than specific to python imo. - tox is indeed the best tool to test several envs, but people that need it are rare. Don't bother for now. Agreed, its a nice thing to do, but rarely do I see it unless you are building libs, and need to support multiple versions. - if you want to build a package, use hatch. But you probably don't. Why would you use hatch over a tool such as poetry, are they equivalent? I'm interested in exploring hatch, but the comparisons I have found seem to be fundamentally misinformed about poetry so its hard to take comparison seriously. - if you have to use conda, don't use pip or venv at all. Don't use piptool, poetry or anything. Stay on pure conda land. It sucks, but the alternative of mixing them is worse. This is a really good point, but is always a frustrating thing to try and solve time and time again, but still encounter problems.