4 ms·

Is there anything in there about managing dependencies within a python project? What is the canonical way to do that in python today?

• woodruffw 3y ago
  It depends (unfortunately) on what you mean by a Python project: * If you mean a thing that's ultimately meant to be `pip` installable, then you should use `pyproject.toml` with PEP 518 standard metadata. That includes the dependencies for your project; the PyPUG linked above should have an example of that. * If you mean a thing that's meant to be deployed with a bunch of Python dependencies, then `requirements.txt` is probably still your best bet.
  • jshen 3y ago
    I meant the second. requirements.txt is a really bad solution for that, and that is the frustration many of us have that have used languages with much better solutions.
    • eyegor 3y ago
      Requirements feels like a dirty hack but it does work fine. It has ==, ~=, and >= for version numbers, as well as allowing you to flag dependencies for different target os, etc. And then you can add setup.py if you need custom steps. But yes, it feels dirty to maintain requirements.txt, requirements-dev.txt, etc. Poetry is the most common solution that I've seen in the wild. You spec everything using pyproject.toml and then "poetry install" and it will manage a venv of its own. But you still need to tell people to "pip install poetry" as step 0 which is annoying. If you don't care about deploying python files, and rather just the final product, I'd recommend either nuitka or pyinstaller. These are for bundling your project into an executable without a python runtime needed (--onefile type of options for single file output). Neither supports cross compilation though.
      • jshen 3y ago
        What flow do you use with requirements.txt that gives you reproducible builds across a team and environments? Using ==, ~=, and >= will not give you reproducible builds.
        • woodruffw 3y ago
          Hash-pinning with requirements.txt will get you the closest to this, but it's not possible in the general case to have a cross-environment reproducible build with Python. The closest you can hope for is a build that reproduces in the same environment. This problem is shared by the majority of language packaging ecosystems; the only one I'm aware of that might avoid it is Java.
          • jshen 3y ago
            How do you get a reproducible build in python for the same os/arch? As in, what concrete steps do you take? This is very easy in nearly every other language that is popular. No one ever answers this clearly in threads like this short of saying “use poetry” which makes my point. I’ve asked many times.
            • woodruffw 3y ago
              I explicitly said that you can't. Python's packaging ecosystem wasn't designed with reproducibility in mind, and has never claimed to prioritize reproducibility. The best you can do is get close, and hash-pinning gets you pretty close. I'm not aware of any other major language or language packaging ecosystem that makes reproducibility straightforward. Certainly not Ruby or NPM, and not even brand new ones like Rust's Crates. Java appears to be the closest[1], but is operating with significant advantages (distributing reproducible bytecode to all users, minimizing system dependencies, etc.). Edit: In addition to hash-pinning, you can instruct `pip` to only install built distributions, i.e. wheels. If you do both hash-pinning and built distributions only, your package installation step _should_ reproduce exactly on machines of the same OS, architecture, and Python version. But again, this is guaranteed nowhere. [1]: https://reproducible-builds.org/docs/jvm/ https://reproducible-builds.org/docs/jvm/
  • starlevel003 3y ago
    > * If you mean a thing that's meant to be deployed with a bunch of Python dependencies, then `requirements.txt` is probably still your best bet. This is exactly how we got in this mess. Using ``setup.cfg`` or ``pyproject.toml`` for all projects makes this easy as now your deployable project can be installed via pip like every other one. 1. ``python -m virtualenv .`` 2. ``source ./bin/activate.fish`` 3. ``pip install -U https://my.program.com/path/to/tarball.tar.xz https://my.program.com/path/to/tarball.tar.xz``
  • butterNaN 3y ago
    Wait, why are ghey two different things? Can't pip be used for deployment?
    • woodruffw 3y ago
      It can -- the second flow also uses `pip`. The terminology here is confusing: the first is the flow that produces a "distribution" (i.e., an sdist or bdist), while the second is the flow that produces an "environment" (i.e., a specific set of packages installed in some prefix).