3 ms·
How do you get a reproducible build in python for the same os/arch? As in, what concrete steps do you take? This is very easy in nearly every other language th
by jshen 3y ago
How do you get a reproducible build in python for the same os/arch? As in, what concrete steps do you take?
This is very easy in nearly every other language that is popular. No one ever answers this clearly in threads like this short of saying “use poetry” which makes my point. I’ve asked many times.
- woodruffw 3y agoI explicitly said that you can't. Python's packaging ecosystem wasn't designed with reproducibility in mind, and has never claimed to prioritize reproducibility. The best you can do is get close, and hash-pinning gets you pretty close. I'm not aware of any other major language or language packaging ecosystem that makes reproducibility straightforward. Certainly not Ruby or NPM, and not even brand new ones like Rust's Crates. Java appears to be the closest[1], but is operating with significant advantages (distributing reproducible bytecode to all users, minimizing system dependencies, etc.). Edit: In addition to hash-pinning, you can instruct `pip` to only install built distributions, i.e. wheels. If you do both hash-pinning and built distributions only, your package installation step _should_ reproduce exactly on machines of the same OS, architecture, and Python version. But again, this is guaranteed nowhere. [1]: https://reproducible-builds.org/docs/jvm/ https://reproducible-builds.org/docs/jvm/
- jshen 3y agoYou are splitting pedantic hairs in order to avoid talking about the obvious. Python's dependency management is much worse than ruby's and nearly every other popular language. In ruby you add dependencies to a Gemfile then ... $ bundle install $ git add Gemfile Gemfile.lock and other members of your team can have the same build as you. requirements.txt doesn't solve this basic need. Edit: formatting.
- woodruffw 3y agoI think you're confusing lockfiles with reproducibility. Lockfiles are good, but they don't guarantee reproducibility: a locked (or pinned, hashed, etc.) dependency might always be the exact same source artifact, but it can install in different ways (e.g. due to local toolchain differences, different versions of dependencies, sensitivity to timestamps, sensitivity to user-controlled environment variables, etc.). Reproducibility is a much harder problem than dependency locking, and (again) I'm not aware of any language level packaging ecosystem that really supports it out of the box. Python doesn't have reproducible builds, but it does have lockfiles (via hashed and pinned requirements). They're not particularly good (for all the reasons mentioned upthread), but they do indeed exist. If you use them as I've said, then your environment will be approximately as repeatable as with any other language packaging ecosystem (and arguably more so in some cases, since wheel installs are reproducible where gem installs aren't).
- jshen 3y agoWhat is the most common python flow for dependency locking? Is it in any of the PEPs?
- woodruffw 3y agoThat would be PEP 508. 508 notably excludes hash-pinning, which is a significant limitation -- hash-pinning is defined by how `pip` implements it.
- jshen 3y agoThanks, is there any kind of tutorial on how you should use this in a python project? Edit: I think this is one of the biggest problems someone coming to python has. Python advocates say some version of, "you can roughly do that" but there isn't a clear explanation of how to do it. Edit 2: I see that the official docs have a Pipenv flow outlined. Is Pipenv the way people do this in python these days?
- woodruffw 3y ago> Thanks, is there any kind of tutorial on how you should use this in a python project? The PyPUG docs contain examples of using all of the PEPs mentioned above. TL;DR: all you need for 99% of use cases is a pyproject.toml.
- jshen 3y agoThose docs say to use Pipenv, or am I looking at the wrong docs? Really not sure why python people can’t articulate a clear flow to follow. It’s all riddles.
- plonk 3y agoBinary wheels are just archives, I don't see differences in the way they install between different systems. Source wheels that contain C/C++ code are so annoying to install on Windows that we don't use them. But most packages provide binary wheels anyway.