5 ms·

I've been mostly out of the python game for quite a while, but I never had that much issue with: pip install -r requirements.txt . Seems like a lot of people h

I've been mostly out of the python game for quite a while, but I never had that much issue with: pip install -r requirements.txt . Seems like a lot of people have tried their hand at various tooling, so there must be more to it than I am aware of.

xnyan 2y ago
Just two reasons (there are more): 1) uv is vastly faster than pip. Just using uv pip install -r requirements.txt and nothing else is a win. 2) uv can handle things like downloading the correct python person, creating a venv (or activating an existing venv if one exists) and essentially all the other cognitive load in a way that's completely transparent to the user. It means you can give someone a Python project and a single command to run it, and you can have confidence it will work on regardless of the platform or a dozen other little variables that trip people up.
- locusofself 2y ago
  I'll give it a whirl soon
svachalek 2y ago
This works if you only have one python project on your system, but most python developers need virtual environments to deal with various projects.
- locusofself 2y ago
  sure, even so I think I had like one or two bash aliases to create/switch virtualenvs
- thisgoodlife 2y ago
  I create a .venv directory for each project(even for those test projects named pytest, djangotest). And each project has its own requirements file. Personally, Python packaging has never been a problem.
  - andrewflnr 2y ago
    Yeah, this is where I've been for a while. Maybe it helps that I don't do any ML work with lots of C or Fortran libraries that depend on exact versions of Python or whatever. But for just writing an application in Python, venv and pip are fine. I'll probably still try uv eventually if everyone really decides they're adopting it, but I won't rush.
  - NewJazz 2y ago
    uv basically does that + python version handling + conveniences like auto-activating venv and installing dependencies
  - kyawzazaw 2y ago
    it was a massive problem at our company's hackathon. just so many hours wasted
  - lmm 2y ago
    What do you do when you accidentally run pip install -r requirements.txt with the wrong .venv activated? If your answer is "delete the venv and recreate it", what do you do when your code now has a bunch of errors it didn't have before? If your answer is "ignore it", what do you do when you try to run the project on a new system and find half the imports are missing? None of these problems are insurmountable of course. But they're niggling irritations. And of course they become a lot harder when you try to work with someone else's project, or come back to a project from a couple of years ago and find it doesn't work.
    - zahlman 2y ago
      >What do you do when you accidentally run pip install -r requirements.txt with the wrong .venv activated? As someone with a similar approach (not using requirements.txt, but using all the basic tools and not using any kind of workflow tool or sophisticated package manager), I don't understand the question. I just have a workflow where this isn't feasible. Why would the wrong venv be activated? I activate a venv according to the project I'm currently working on. If the venv for my current code isn't active, it's because nothing is active. And I use my one global Pip through a wrapper, which (politely and tersely) bonks me if I don't have a virtual environment active. (Other users could rely on the distro bonking them, assuming Python>=3.11. But my global Pip is actually the Pipx-vendored one, so I protect myself from installing into its environment.) You might as well be asking Poetry or uv users: "what do you do when you 'accidentally' manually copy another project's pyproject.toml over the current one and then try to update?" I'm pretty sure they won't be able to protect you from that. >If your answer is "delete the venv and recreate it", what do you do when your code now has a bunch of errors it didn't have before? If it did somehow happen, that would be the approach - but the code simply wouldn't have those errors. Because that venv has its own up-to-date listing of requirements; so when I recreated the venv, it would naturally just contain what it needs to. If the listing were somehow out of date, I would have to fix that anyway, and this would be a prompt to do so. Do tools like Poetry and uv scan my source code and somehow figure out what dependencies (and versions) I need? If not, I'm not any further behind here. >And of course they become a lot harder when you try to work with someone else's project, or come back to a project from a couple of years ago and find it doesn't work. I spent this morning exploring ways to install Pip 0.2 in a Python 2.7 virtual environment, "cleanly" (i.e. without directly editing/moving/copying stuff) starting from scratch with system Python 3.12. (It can't be done directly, for a variety of reasons; the simplest approach is to let a specific version of `virtualenv` make the environment with an "up-to-date" 20.3.4 Pip bootstrap, and then have that Pip downgrade itself.) I can deal with someone else's (or past me's) requirements.txt being a little wonky.
raziel2p 2y ago
if you come back to a project you haven't worked on for a year or two, you'll end up with new versions of dependencies that don't work with your code or environment any more. you can solve this with constraints, pip-tools etc., but the argument is uv does this better
aaravchen 2y ago
A big thing that trips people up until they try to use a public project (from source) or an older project, is the concept of a dependencies file and a lock file. The dependency file (what requirements.txt is supposed to be), just documents the things you depend on directly, and possibly known version constraints. A lock file captures the exact version of your direct and indirect dependencies at the moment in time it's generated. When you go to use the project, it will read the lock file, if it exists, and match those versions for anything listed directly or indirectly in the dependency file. It's like keeping a snapshot of the exact last-working dependency configuration. You can always tell it to update the lock file and it will try to recaclulate everything from latest that meets your dependency constraints in the dependency file, but if something doesn't work with that you'll presumably have your old lock file to fall back on _that will still work_. It's a standard issue/pattern in all dependency managers, but it's only been getting attention for a handful of years with the focus on reproducibility for supply chain verification/security. It has the side effect of helping old projects keep working much longer though. Python has had multiple competing options for solutions, and only in the lat couple years did they pick a format winner.
- nickelpro 2y ago
  Then why even have the dependency file? If the dependency file is wrong, and describes versions that are incompatible with the project, it should be fixed. Duplicating that information elsewhere is wrong. Lockfiles have a very obvious use case: Replicable builds across machines in CI. You want to ensure that all the builds in the farm are testing the same thing across multiple runs, and that new behaviors aren't introduced because numpy got revved in the middle of the process. When that collective testing process is over, the lockfile is discarded. You should not use lockfiles as a "backup" to pyproject.toml. The version constraints in pyproject.toml should be correct. If you need to restrict to a single specific version, do so, "== 2.2.9" works fine.
  - zahlman 2y ago
    >Then why even have the dependency file? Horses for courses. Dependency files - whether the project's requirements (or optional requirements, or in the future, other arbitrary dependency groups) in `pyproject.toml`, or a list in a `requirements.txt` file (the filename here is actually arbitrary) don't describe versions at all, in general. Their purpose is to describe what's needed to support the current code: its direct dependencies, with only as much restriction on versions as is required. The base assumption is that if a new version of a dependency comes out, it's still expected to work (unless a cap is set explicitly), and has a good chance of improving things in general (better UI, more performant, whatever). This is suitable for library development: when others will cite your code as a dependency, you avoid placing unnecessary restrictions on their environment. Lockfiles are meant to describe the exact version of everything that should be in the environment to have exact reproducible behaviour (not just "working"), including transitive dependencies. The base assumption is that any change to anything in the environment introduces an unacceptable risk; this is the tested configuration. This is suitable for application development: your project is necessarily the end of the line, so you expect others to be maximally conservative in meeting your specific needs. You could also take this as an application of Postel's Law. >Lockfiles have a very obvious use case: Replicable builds across machines in CI. There are others who'd like to replicate their builds: application developers who don't want to risk getting bug reports for problems that turn out to be caused by upstream updates. > You should not use lockfiles as a "backup" to pyproject.toml. The version constraints in pyproject.toml should be correct. If you need to restrict to a single specific version, do so, "== 2.2.9" works fine. In principle, if you need a lockfile, you aren't distributing a library package anyway. But the Python ecosystem is still geared around the idea that "applications" would be distributed the same way as libraries - as wheels on PyPI, which get set up in an environment, using the entry points specified in `pyproject.toml` to create executable wrappers. Pipx implements this (and rejects installation when no entry points are defined); but the installation will still ignore any `requirements.txt` file (again, the filename is arbitrary; but also, Pipx is delegating to Pip's ordinary library installation process, not passing `-r`). You can pin every version in `pyproject.toml`. Your transitive dependencies still won't be pinned that way. You can explicitly pin those, if you've done the resolution. You still won't have hashes or any other supply-chain info in `pyproject.toml`, because there's nowhere to put it. (Previous suggestions of including actual lockfile data in `pyproject.toml` have been strongly rejected - IIRC, Hatch developer Ofek Lev was especially opposed to this.) Perhaps in the post-PEP 751 future, this could change. PEP 751 specifies both a standard lockfile format (with all the sorts of metadata that various tools might want) and a standard filename (or at least filename pattern). A future version of Pipx could treat `pylock.toml` as the "compiled" version of the "source" dependencies in `pyproject.toml`, much like Pip (and other installers) treat `PKG-INFO` (in an sdist, or `METADATA` in a wheel) as the "compiled" version (dependency resolution notwithstanding!) of other metadata.
tiltowait 2y ago
I write internal tools using Python at work. These tools are often used by non-Python devs. I am so, so tired of adding a blurb on creating a venv. (Of course, the alternative—"install this software you've never heard of"—isn't fantastic either. But once they do have it, it'd be pretty neat to be able to tell them to just "uvx <whatever>".)
- zahlman 2y ago
  Is it not the same blurb every time that you could copy and paste? Or you can make sure you have an entry point - probably a better UX for your coworkers anyway - and run them through a `pipx` install. Or you could supply your own Bash script or whatever. Or since you could use a simple packager like pex (https://docs.pex-tool.org/ https://docs.pex-tool.org/). (That one even allows you to embed a Python executable, if you need to and if you don't have to worry about different platforms.) Maybe even the standard library `zipapp` works for your needs.

xnyan 2y ago

Just two reasons (there are more): 1) uv is vastly faster than pip. Just using uv pip install -r requirements.txt and nothing else is a win. 2) uv can handle things like downloading the correct python person, creating a venv (or activating an existing venv if one exists) and essentially all the other cognitive load in a way that's completely transparent to the user. It means you can give someone a Python project and a single command to run it, and you can have confidence it will work on regardless of the platform or a dozen other little variables that trip people up.

locusofself 2y ago
I'll give it a whirl soon

svachalek 2y ago

This works if you only have one python project on your system, but most python developers need virtual environments to deal with various projects.

locusofself 2y ago
sure, even so I think I had like one or two bash aliases to create/switch virtualenvs
thisgoodlife 2y ago
I create a .venv directory for each project(even for those test projects named pytest, djangotest). And each project has its own requirements file. Personally, Python packaging has never been a problem.
- andrewflnr 2y ago
  Yeah, this is where I've been for a while. Maybe it helps that I don't do any ML work with lots of C or Fortran libraries that depend on exact versions of Python or whatever. But for just writing an application in Python, venv and pip are fine. I'll probably still try uv eventually if everyone really decides they're adopting it, but I won't rush.
- NewJazz 2y ago
  uv basically does that + python version handling + conveniences like auto-activating venv and installing dependencies
- kyawzazaw 2y ago
  it was a massive problem at our company's hackathon. just so many hours wasted
- lmm 2y ago
  What do you do when you accidentally run pip install -r requirements.txt with the wrong .venv activated? If your answer is "delete the venv and recreate it", what do you do when your code now has a bunch of errors it didn't have before? If your answer is "ignore it", what do you do when you try to run the project on a new system and find half the imports are missing? None of these problems are insurmountable of course. But they're niggling irritations. And of course they become a lot harder when you try to work with someone else's project, or come back to a project from a couple of years ago and find it doesn't work.
  - zahlman 2y ago
    >What do you do when you accidentally run pip install -r requirements.txt with the wrong .venv activated? As someone with a similar approach (not using requirements.txt, but using all the basic tools and not using any kind of workflow tool or sophisticated package manager), I don't understand the question. I just have a workflow where this isn't feasible. Why would the wrong venv be activated? I activate a venv according to the project I'm currently working on. If the venv for my current code isn't active, it's because nothing is active. And I use my one global Pip through a wrapper, which (politely and tersely) bonks me if I don't have a virtual environment active. (Other users could rely on the distro bonking them, assuming Python>=3.11. But my global Pip is actually the Pipx-vendored one, so I protect myself from installing into its environment.) You might as well be asking Poetry or uv users: "what do you do when you 'accidentally' manually copy another project's pyproject.toml over the current one and then try to update?" I'm pretty sure they won't be able to protect you from that. >If your answer is "delete the venv and recreate it", what do you do when your code now has a bunch of errors it didn't have before? If it did somehow happen, that would be the approach - but the code simply wouldn't have those errors. Because that venv has its own up-to-date listing of requirements; so when I recreated the venv, it would naturally just contain what it needs to. If the listing were somehow out of date, I would have to fix that anyway, and this would be a prompt to do so. Do tools like Poetry and uv scan my source code and somehow figure out what dependencies (and versions) I need? If not, I'm not any further behind here. >And of course they become a lot harder when you try to work with someone else's project, or come back to a project from a couple of years ago and find it doesn't work. I spent this morning exploring ways to install Pip 0.2 in a Python 2.7 virtual environment, "cleanly" (i.e. without directly editing/moving/copying stuff) starting from scratch with system Python 3.12. (It can't be done directly, for a variety of reasons; the simplest approach is to let a specific version of `virtualenv` make the environment with an "up-to-date" 20.3.4 Pip bootstrap, and then have that Pip downgrade itself.) I can deal with someone else's (or past me's) requirements.txt being a little wonky.

raziel2p 2y ago

if you come back to a project you haven't worked on for a year or two, you'll end up with new versions of dependencies that don't work with your code or environment any more. you can solve this with constraints, pip-tools etc., but the argument is uv does this better

aaravchen 2y ago

A big thing that trips people up until they try to use a public project (from source) or an older project, is the concept of a dependencies file and a lock file. The dependency file (what requirements.txt is supposed to be), just documents the things you depend on directly, and possibly known version constraints. A lock file captures the exact version of your direct and indirect dependencies at the moment in time it's generated. When you go to use the project, it will read the lock file, if it exists, and match those versions for anything listed directly or indirectly in the dependency file. It's like keeping a snapshot of the exact last-working dependency configuration. You can always tell it to update the lock file and it will try to recaclulate everything from latest that meets your dependency constraints in the dependency file, but if something doesn't work with that you'll presumably have your old lock file to fall back on _that will still work_. It's a standard issue/pattern in all dependency managers, but it's only been getting attention for a handful of years with the focus on reproducibility for supply chain verification/security. It has the side effect of helping old projects keep working much longer though. Python has had multiple competing options for solutions, and only in the lat couple years did they pick a format winner.

nickelpro 2y ago
Then why even have the dependency file? If the dependency file is wrong, and describes versions that are incompatible with the project, it should be fixed. Duplicating that information elsewhere is wrong. Lockfiles have a very obvious use case: Replicable builds across machines in CI. You want to ensure that all the builds in the farm are testing the same thing across multiple runs, and that new behaviors aren't introduced because numpy got revved in the middle of the process. When that collective testing process is over, the lockfile is discarded. You should not use lockfiles as a "backup" to pyproject.toml. The version constraints in pyproject.toml should be correct. If you need to restrict to a single specific version, do so, "== 2.2.9" works fine.
- zahlman 2y ago
  >Then why even have the dependency file? Horses for courses. Dependency files - whether the project's requirements (or optional requirements, or in the future, other arbitrary dependency groups) in `pyproject.toml`, or a list in a `requirements.txt` file (the filename here is actually arbitrary) don't describe versions at all, in general. Their purpose is to describe what's needed to support the current code: its direct dependencies, with only as much restriction on versions as is required. The base assumption is that if a new version of a dependency comes out, it's still expected to work (unless a cap is set explicitly), and has a good chance of improving things in general (better UI, more performant, whatever). This is suitable for library development: when others will cite your code as a dependency, you avoid placing unnecessary restrictions on their environment. Lockfiles are meant to describe the exact version of everything that should be in the environment to have exact reproducible behaviour (not just "working"), including transitive dependencies. The base assumption is that any change to anything in the environment introduces an unacceptable risk; this is the tested configuration. This is suitable for application development: your project is necessarily the end of the line, so you expect others to be maximally conservative in meeting your specific needs. You could also take this as an application of Postel's Law. >Lockfiles have a very obvious use case: Replicable builds across machines in CI. There are others who'd like to replicate their builds: application developers who don't want to risk getting bug reports for problems that turn out to be caused by upstream updates. > You should not use lockfiles as a "backup" to pyproject.toml. The version constraints in pyproject.toml should be correct. If you need to restrict to a single specific version, do so, "== 2.2.9" works fine. In principle, if you need a lockfile, you aren't distributing a library package anyway. But the Python ecosystem is still geared around the idea that "applications" would be distributed the same way as libraries - as wheels on PyPI, which get set up in an environment, using the entry points specified in `pyproject.toml` to create executable wrappers. Pipx implements this (and rejects installation when no entry points are defined); but the installation will still ignore any `requirements.txt` file (again, the filename is arbitrary; but also, Pipx is delegating to Pip's ordinary library installation process, not passing `-r`). You can pin every version in `pyproject.toml`. Your transitive dependencies still won't be pinned that way. You can explicitly pin those, if you've done the resolution. You still won't have hashes or any other supply-chain info in `pyproject.toml`, because there's nowhere to put it. (Previous suggestions of including actual lockfile data in `pyproject.toml` have been strongly rejected - IIRC, Hatch developer Ofek Lev was especially opposed to this.) Perhaps in the post-PEP 751 future, this could change. PEP 751 specifies both a standard lockfile format (with all the sorts of metadata that various tools might want) and a standard filename (or at least filename pattern). A future version of Pipx could treat `pylock.toml` as the "compiled" version of the "source" dependencies in `pyproject.toml`, much like Pip (and other installers) treat `PKG-INFO` (in an sdist, or `METADATA` in a wheel) as the "compiled" version (dependency resolution notwithstanding!) of other metadata.

tiltowait 2y ago

I write internal tools using Python at work. These tools are often used by non-Python devs. I am so, so tired of adding a blurb on creating a venv. (Of course, the alternative—"install this software you've never heard of"—isn't fantastic either. But once they do have it, it'd be pretty neat to be able to tell them to just "uvx <whatever>".)

zahlman 2y ago
Is it not the same blurb every time that you could copy and paste? Or you can make sure you have an entry point - probably a better UX for your coworkers anyway - and run them through a `pipx` install. Or you could supply your own Bash script or whatever. Or since you could use a simple packager like pex (https://docs.pex-tool.org/ https://docs.pex-tool.org/). (That one even allows you to embed a Python executable, if you need to and if you don't have to worry about different platforms.) Maybe even the standard library `zipapp` works for your needs.