12 ms·
Tools you can use to make sure the Python program you wrote keeps working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-virtualenv, virtualenvwrapper
by nlitened 3y ago
Tools you can use to make sure the Python program you wrote keeps working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-virtualenv, virtualenvwrapper, pyenv-virtualenvwrapper, venv, pyvenv, conda, miniconda, poetry, docker, nix.
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
- speedgoose 3y agoNone. The software container image is the best bet but you need to keep the image and not only the building scripts.
- nickjj 3y agoYeah, containers don't fully solve this problem. We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result. Using pip to freeze your dependencies and writing a tiny shell script to generate a lock file at build time is better than nothing to solve this problem with nothing more than pip. It's what I do in https://github.com/nickjj/docker-flask-example https://github.com/nickjj/docker-flask-example and https://github.com/nickjj/docker-django-example https://github.com/nickjj/docker-django-example. It's not perfect but it solves 80% with minimal complexity.
- Cu3PO42 3y agoThis is not true. Nix actually solves the problem. If you package with Nix, you'll get the exact same version of Python with the exact same version of dependencies, including the exact same version of system libraries. Your build will work in a year just as it does today. You don't even need to keep any binary artifacts. Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
- speedgoose 3y agoHow does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively? Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
- Cu3PO42 3y agoIt depends. If the dependency is in nixpkgs (Nix' own package repository), someone will have done the work of figuring out the dependencies properly and you can just use that. If it's not, you can either do that yourself and pin particular versions, or you can integrate with a tool like Poetry: Poetry could generate a lockfile for you that you then reference from Nix to get the versions of Python packages. You'd still need to specify any native dependencies manually, though.
- ris 3y agoUltimately the true bellwether of compatibility is the package's own test suite, which we try to get working (and integrated into the build process) for as many packages as possible. For packages which have poor/non-existent test suites, often a downstream package's test suite will expose compatibility problems (we've found bugs in zlib security patches using curl's test suite for instance). nixpkgs maintainers are frequently the first to notify a project author of incompatibility with new versions of another package.
- pxc 3y ago> How does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively? In Nix lingo, a package successfully building or running against an implicit or unmanaged dependency is called 'impurity'. To help keep builds 'pure', Nix builds everything in a sandbox and does various other tricks to ensure that a package being built can't/doesn't find any dependencies at build time that you don't explicitly tell Nix to include in the build environment. (This is also increasingly how Linux distros build their packages, to solve the same problem.) For the most part, if building (and running the test suite during the build) a Python package with Nix succeeds, you can be confident that you've got all the dependencies sorted out— even the ones upstream forgot to tell you about. > Do you have to find them out and fix them manually or can you generate a lock file and hope for the best? At install time, you can be confident that Nix will bring along all of the system-level dependencies your Python package needs. You don't have to find and fill any gaps at install time 10 years from now or whatever. When you're writing your Nix package for the first time, you'll be doing a mix of generating the Nix code that defines your package from some upstream, Python-specific lockfile and making manual corrections when the build fails. Nix doesn't have any magic for figuring out dependencies that are left out of poetry.lock or whatever, or for disambiguating guaranteed-compatible exact versions from requirements.txt.
- riwsky 3y agoGo
- ptsneves 3y agoPyinstaller, provided you build it in a reasonably old glibc to avoid glibc incompatibilities. I know that is a caveat but it is a one time build time operation.
- hermitdev 3y agoDoesn't Pyinstaller also require CPython to have been built with '--enable-shared'? I _think_ this is the default for most system-installed Pythons that I've seen, but it's not the default if you build Python from source or install via pyenv.
- bvrmn 3y agopython -m venv myenv . myenv/bin/activate <install reqs> <your program> Ensures you have the same python environment. The other part is OS state.
- pnt12 3y agoThis approach is full of footguns. Forget to install virtualenv? Have fun getting rid of packages in your global system. Sub-dependencies? You'll have to think of a strategy for it. Most people don't and the reproducibility of their project suffers. Developer vs production dependencies ? You'll have to think of a strategy for it. And I think I'm forgetting one, but that's already enough.
- _dain_ 3y ago>The other part is OS state And that's the whole problem. "The same Python environment" doesn't mean much because it underspecifies the full runtime dependency chain.
- bvrmn 3y agoIt's inherit problem for all languages based on some runtime. Java, PHP, Ruby, whatever have the same issue. And TBH it's not very major issue to start with. No other solution besides fixed OS (containers, nix) could solve this problem.
- deleted 3y ago[deleted]
- kzrdude 3y agoYou missed hatch, uv, rye, pdm. I think rye hits most of the points pretty well, it ensures both python and package versions.
- _dain_ 3y agoOnly Nix does. The builds are deterministic, transitively all the way back to gcc and libc.
- diggan 3y ago> Which ones did I miss? Using python and python libraries only from your package manager (like APT) for a specific OS version. Micromamba as well, lightweight version of conda/miniconda.
- anze3db 3y agoYou've missed: pdm, uv, pip-tools, pipx, rye, and probably some others. Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now. It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
- rbanffy 3y agoShameless plug: don't forget the wonderful pip-chill for simplifying gigantic requirements files (and for stripping out version numbers to make canaries easier to do).
- otabdeveloper4 3y ago> Which of them actually ensure your program always works the same as when you first wrote it, without asterisks? Nix does, if you want to actually invest the effort into the "without asterisks" part.
- infecto 3y agoI am always curious how many people, outside of those building code for third party clients, actually hit this problem? In the 10+ years of using Python I have never had a problem using the core tools. The ecosystem is far from perfect but it has never cause me a problem. Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
- renegat0x0 3y agoI am not sure about what 'core' means in this context. In my edperience I had many many problems with OS packages vs pip installed ones. There were really strange dependency issues. In some cases I encountered dependency hell. Even if somebody said to me that the core means also using virtualized en I would disagree as places over Internet not always explicitly guide you in that route.
- wodenokoto 3y agoJust getting me and one coworker in sync on our dev machines often causes problems. Even more when we try to deploy to a cloud service which relies on pip.
- thebytefairy 3y agoIf I were writing applications only for myself it'd be fine, but I've definitely run into issues when writing code with others - teammates, open source repos, etc.
- crabbone 3y agoWorking in CI in a company that has couple dozens of Python packages: I'd say about once a week. There are some weeks with no incidents, and there are weeks when everything is broken for many days straight. NB. The latest incident was Friday when I've discovered that some CI pipeline ran `setup.py install` that down the lane invoked easy_install, which doesn't have a policy of ignoring bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground when it was trying to install scikit-learn which wanted NumPy >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't realize that this version should be OK (also, it shouldn't have installed non-release versions anyways).
- tootie 3y agoWhile gross ugly Maven lets you specify the bytecode version and every JDK can produce backwards-compatible jars and has done for many years
- fbuilesv 3y agoWhat package managers are people using in other languages to make sure that software "always works the same as when you first wrote it, without asterisks"? I'd like to understand how they solve the "package no longer exists in a central registry" problem.
- crabbone 3y agoThis is not as much about a package manager as it is about conventions and necessary dependencies. Standards ensure that going forward language semantics and syntax don't change. Having minimal dependencies ensures program longevity. Package manager cannot solve these problems, no matter how good it is at its job. Python doesn't have a standard, it's heavily reliant on dependencies which are very plentiful and similarly unregulated. A program written in C that uses only functionality described in some POSIX standard will endure decades unmodified. Even Python helloworld program went stale sometime ago, even though it's just one line.
- Shrezzing 3y agoDotnet uses Nuget[1]. Packages in the system are immutable, & never changing. They can be unlisted, but never deleted (except in limited & extreme cases, like malware), which means even if a package maintainer stops publishing new versions to the repository, existing packages will continue to be publicly available for as long as Microsoft continues to exist. [1] https://www.nuget.org/ https://www.nuget.org/
- mixmastamyk 3y agoOften a package says it works with OS version X, and not X+1. It may be true or false. But what you described does not solve either version of that problem. Or the "says it will work with > X," but doesn't.
- zaptheimpaler 3y agoI think you're referring to vendoring dependencies? In python/pip for example, you can download the source for a package and point to the folder directly as a dependency instead of the version or a git URL. Most package managers/languages support some version of that. I suppose if you wanted to vendor all dependencies by default, keep them updated etc it would take a little more scripting or extra tools.
- codethief 3y agoasdf can be used as an alternative to pyenv. (In fact, it is not only meant for the Python ecosystem, so it can also replace nvm and others.) For me, the combination of asdf and Poetry has worked quite well recently: I use asdf to pin the Python & Poetry version and then use Poetry to pin everything else.
- leetrout 3y agoPoint of clarification: asdf uses python-build which is from pyenv. https://github.com/asdf-community/asdf-python?tab=readme-ov-file https://github.com/asdf-community/asdf-python?tab=readme-ov-...
- codethief 3y agoTrue. I mentioned asdf mainly because it can be used with many languages & tools and is not restricted to the Python ecosystem.
- aequitas 3y agoYou're missing Tox[0] & Nox[1]. Tox is actually quite nice when dealing with testing code across different versions of Python. [0] https://tox.wiki https://tox.wiki [1] https://nox.thea.codes/en/stable/ https://nox.thea.codes/en/stable/
- spacephysics 3y agoYou didn’t miss Poetry, but I have to say up until I started using Poetry python in large projects was a pain in the tooling department to setup and maintain over longer periods of time. It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
- john-radio 3y agoWhat does poetry deliver beyond what pyenv gets you? I haven't used either tool super extensively.
- claytonjy 3y agopyenv only manages python versions, while poetry manages dependencies and virtual environments. They are complimentary, but do not overlap.
- hermitdev 3y agoTo add to this, with poetry, you basically do `poetry env use python3.12` to create a python virtualenv on python3.12 (you can use whatever python version pyenv supports, doesn't even have to be CPython). The generated virtual env name is a little wonky. I'm not sure exactly what the scheme is, but it's basically `$(package)-$(some sort of hash?)-$(pyversion)`. I can't speak for all tools, but at least VS Code detects the poetry env and suggests it (and indicates it as a Poetry env) when you go to configure the project interpreter.
- bootsmann 3y agoYou can use the poetry config to enable poetry putting the virtualenv into the project folder itself, which allows most IDEs to discover it.
- crabbone 3y ago> Which of them actually ensure your program always works the same as when you first wrote it, without asterisks? There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
- 2wrist 3y agoI have been looking at Rye, I like it so far. The other one is Docker, which some people hate. I don't mind it, in some cases I prefer it.
- airstrike 3y agoYou only need pip and virtualenvwrapper requirements.txt is a text file, not a separate tool... if you want your program to work exactly as intended in the future, there are tools like py2exe and py2app for that.
- mixmastamyk 3y agoDon't need any wrappers either. venv has been built into python for ages.
- airstrike 3y agoyes but `workon` is too good not to have
- mixmastamyk 3y agoLooks like what could be a shell alias to cd and activate? I always keep a few tabs open to my repos folder, so not sure it would help me much. Doesn't seem to support fish, though activate does.
- physicles 3y agoI modified my bash prompt to detect and auto-activate Python environments, and show if one is currently active. Haven’t thought about activate in years. It’s great.
- mixmastamyk 3y agoYes, I did that on my work laptop with fish, since I only ever used it with the work project. On my personal machine I don't bother with venvs, so also haven't thought about it in years. But was trying to figure out what the "workon" command did for GP.
- airstrike 3y ago
- deleted 3y ago[deleted]
- masto 3y agoI remember when I was a childish Perl fan trading insults with the childish Python fans of the time, one of the things "they" always chided "us" about was that we had too many different ways to accomplish the same thing...
- whalesalad 3y agopyenv is all you need. pip is part of python. requirements.txt is just a convention used to store the list of stuff to hand to pip.
- reddit_clone 3y agoYou missed Pipenv :-) Edit: Oh I see you didn't. I can't read.
- andreif 3y agoDependencies are for the weak. You only need Python. Just support a range of versions if you intend to distribute.