7 ms·

Well, big fan of uv. But... the 86GB python dependency download cache on my primary SSD, most of which can be attributed to the 50 different versions of torch,

by sieve 2y ago

Well, big fan of uv. But... the 86GB python dependency download cache on my primary SSD, most of which can be attributed to the 50 different versions of torch, is testament to the fact that even uv cannot salvage the mess that is pip. Never felt this much rage at the state of a language/build system in the 25 years that I have been programming. And I had to deal with Scala's SBT ("Simple Build Tool") in another life.

simonw 2y ago
I don't think pip is to blame for that. PyTorch is sadly an enormous space hog. I just started a fresh virtual environment with "python -m venv venv" - running "du -h" showed it to be 21MB. After running "venv/bin/pip install torch" it's now 431MB. The largest file in there is this one: 178M ./lib/python3.10/site-packages/torch/lib/libtorch_cpu.dylib There's a whole section of the uv manual dedicated just to PyTorch: https://docs.astral.sh/uv/guides/integration/pytorch/ https://docs.astral.sh/uv/guides/integration/pytorch/ (I just used find to locate as many libtorch_cpu.dylib files as possible on my laptop and deleted 5.5GB of them)
- sieve 2y ago
  I use uv pip to install dependencies for any LLM software I run. I am not sure if uv re-implements the pip logic or hands over resolution to pip. But it does not change the fact that I have multiple versions of torch + multiple installations of the same version of torch in the cache. Compare this to the way something like maven/gradle handles this and you have to wonder WTF is going on here.
  - simonw 2y ago
    uv implements its own resolution logic independently of pip. Maybe your various LLM libraries are pinning different versions of Torch? Different Python versions each need their own separate Torch binaries as well. At least with uv you don't end up with separate duplicate copies of PyTorch in each of the virtual environments for each of your different projects!
    - sieve 2y ago
      > Different Python versions each need their own separate Torch binaries as well Found this the hard way. Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird.
      - woodruffw 2y ago
        > Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird. Yes, it's essentially that: CPython doesn't guarantee exact ABI stability between versions unless the extension (and its enclosing package) intentionally build against the stable ABI[1]. The courteous thing to do in the Python packaging ecosystem is to build "abi3" wheels that are stable and therefore don't need to be duplicated as many times (either on the index or on the installing client). Torch doesn't build these wheels for whatever reason, so you end up with multiple slightly different but functionally identical builds for each version of Python you're using. TL;DR: This happens because of an interaction between two patterns that Python makes very easy: using multiple Python versions, and building/installing binary extensions. In a sense, it's a symptom of Python's success: other ecosystems don't have these problems because they have far fewer people running multiple configurations simultaneously. [1]: https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html
        sieve 2y ago
        My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. I am planning to shift some of my stuff to pypy (so a "fast" python exists, kind of). But some dependencies can be problematic, I have heard.
        Numerlor 2y ago
        python just didn't have much momentum until relatively recently, despite it's age. There are efforts to speed it up going on now backed by Microsoft. For pypy it's in a weird spot as the things it does fast are the ones you'd usually just offload to a module implemented in C
- conradev 2y ago
  uv should hard link files if they’re identical like Nix does If a package manager stores more than it needs to, it is a package manager problem.
  - d0mine 2y ago
    What makes you think it doesn't?
    - conradev 2y ago
      This from above: > I just used find to locate as many libtorch_cpu.dylib files as possible on my laptop and deleted 5.5GB of them but maybe it wasn’t actually 5.5 GB!
      - rfoo 2y ago
        It was like a dozen different versions of libtorch_cpu.dylib. So hardlink does not help.
      - rat87 2y ago
        It probably is 5.5GB. UV caches versions of packages and symlinks to them but pytorch is infamous for many different versions especially for different features. And since its compiled dependency even if UV was to attempt the the more complicated method of symlinking to use a single version of identical files it wouldn't help much. You'd probably need to store binary diffs or chunks of files that are binary identical, at that point your code would probably start to resemble a file system in user space and time to switch to a particular version of the files (ie create thek as qctual files in filesystem) would be much higher. Also I believe uvs cache is separate from the pip cache so you could have different copies in both. I think there's a uv cache prune command. Arguably it should offer to install a from job to do it periodically
  - Spivak 2y ago
    You're about to be pleasantly surprised then. https://docs.astral.sh/uv/reference/settings/#link-mode https://docs.astral.sh/uv/reference/settings/#link-mode It's even the default. Here's where it's implemented if you're curious https://github.com/astral-sh/uv/blob/f394f7245377b6368b9412d88575446602f22525/crates/uv-install-wheel/src/linker.rs#L280 https://github.com/astral-sh/uv/blob/f394f7245377b6368b9412d...
- nighthawk454 2y ago
  Testing this out locally, repro'd pretty similar numbers on macOS ARM and docker. Unfortunately the CPU-only build isn't really any smaller either, thought it was
RockRobotRock 2y ago
A lot of that is cuda blobs, right?
- sieve 2y ago
  ROCm. About 40%. But there is duplication there as well. Two 16GB folders containing the exact same version.
  - IgorPartola 2y ago
    Sounds like a great use case for ZFS’s deduplication at block level.
    - 22c 2y ago
      or, you know.. symlinks
      - flakes 2y ago
        Main issue with symlink is needing to choose the source of truth— one needs to be the real file, and the other point to it. You also need to make sure they have the same lifetimes to prevent dangling links. Hardlink is somewhat better because both point to the same inode, but will also not work if the file needs different permissions or needs to be independently mutable from different locations. Reflink hits the sweetspot where it can have different permissions, updates trigger CoW preventing confusing mutations, and all while still reducing total disk usage.
        22c 2y ago
        I don't disagree but I think some of these problems could potentially be solved by having somewhat of a birds nest of a filesystem for large blobs, eg. /blobs/<sha256_sum>/filename.zip and then symlinking/reflinking filename.zip to wherever it needs to be in the source tree... It's more portable than hardlinks, solves your "source of truth" problem and has pretty wide platform support. Platforms that don't support symlinks/reflinks could copy the files to where they need to be then delete the blob store at the end and be no worse off than they are now. Anyway, I'm just a netizen making a drive-by comment.
paulddraper 2y ago
> torch Ah found the issue.
rsyring 2y ago
If you are using uv: $ uv cache prune $ uv cache clean Take your pick and schedule to run weekly/monthly.
- code_biologist 2y ago
  Light user of uv here. `prune` just saved me 1.1GiB. Thanks!
teej 2y ago
You can specify platform specific wheels in your pyproject.toml [[tool.uv.index]] name = "pytorch-cu124" url = "https://download.pytorch.org/whl/cu124" explicit = true [[tool.uv.index]] name = "pytorch-cpu" url = "https://download.pytorch.org/whl/cpu" explicit = true [tool.uv.sources] torch = [ { index = "pytorch-cu124", marker = "platform_system != 'Darwin'" }, { index = "pytorch-cpu", marker = "platform_system == 'Darwin'" }, ]
globular-toast 2y ago
Try building uv itself. Cargo used something like 40GiB of disk space somehow.
- sieve 2y ago
  I am only criticizing python because I am actively using it. The node ecosystem and the rust one seem to have their own issues. I have zero interest in either of them so I haven't looked into them in detail. However, I have to deal with node on occasion because a lot of JS/CSS tooling is written using it. It has a HUGE transitive dependency problem.
aragilar 2y ago
That problem is very much not pip (pip is only the installer), the issue is: * We have a conflict between being easy to use (people don't need to work out which version of cuda/which gpu settings/libraries/etc. to use) vs install size (it's basically the x86 vs arm issue, except at least 10 fold larger). Rather than making it the end-users problem, packages bundle all possible options into a single artifact (MacOS does this same, but see the 10 fold larger issue). * The are almost fundamental assumptions (and the newer Python packaging tools, including uv, very much rely on these) that Python packaging makes that are inherently about how a system should act (basically, frozen, no detection available apart from the "OS"), which do not align with having hardware/software that much be detected. One could do this via sdists, but Windows plus the issues around dynamic metadata make this a non-starter (and hence tools like conda, spack and others from the more scientific side of the ecosystem have been created—notably on the more webby side, this problems are solved either via vendoring non-python libraries, or making it someone/something else's problem, hence docker or the cloud for databases or other companion services). * Frankly, more and more developers have no idea how systems are built (and this isn't just a Python issue). Docker lets people hide their sins, with magical invocations that just work (and static linking in many cases sadly does the same). There are tools out of the hyperscalars which are designed to solve these problems, but they solve it by creating tools that experts can wrangle many systems and hence imply you have a team which can do the wrangling. Can this be solved? Maybe, but not by a new tool (on its own). It would require a lot of devs who may not see much improvement to their workflow change their workflow for others (to newer ones which remove the assumptions which are built in to the current workflows), plus a bunch of work by key stakeholders (and maybe even the open sourcing of some crown jewels), and I don't see that happening.
- sieve 2y ago
  > people don't need to work out which version of cuda/which gpu settings/libraries/etc. to use This is not true in my case. The regular pytorch does not work on my system. I had to download a version specific to my system from the pytorch website using --index-url. > packages bundle all possible options into a single artifact Cross-platform Java apps do it too. For e.g., see https://github.com/xerial/sqlite-jdbc https://github.com/xerial/sqlite-jdbc. But it does not become a clusterfuck like it does with python. After downloading gigabytes and gigabytes of dependencies repeatedly, the python tool you are trying to run will refuse to do so for random reasons. You cannot serve end-users a shit-sandwich of this kind. The python ecosystem is a big mess and, outside of a few projects like uv, I don't see anyone trying to build a sensible solution that tries to improve both speed/performance and packaging/distribution.
EdwardDiego 2y ago
> And I had to deal with Scala's SBT ("Simple Build Tool") in another life. I feel you.
- amelius 2y ago
  For someone who was just about to give Scala a try, what's wrong with it and are there alternative build tools?
  - ATMLOTTOBEER 2y ago
    It defines a DSL for your build that looks roughly like Scala code. But… it’s not! And there is a confusing “resolution” system for build tasks/settings. It’s also slow as shit. See https://www.lihaoyi.com/post/SowhatswrongwithSBT.html https://www.lihaoyi.com/post/SowhatswrongwithSBT.html for a comprehensive takedown. If you’re interested in just playing around with scala I would use https://scala-cli.virtuslab.org https://scala-cli.virtuslab.org Or for larger projects, the thing the author of the linked article is plugging (mill).
  - paulddraper 2y ago
    I've use sbt and don't have a problem. But Mill is very good. [1] [1] https://mill-build.org/mill/index.html https://mill-build.org/mill/index.html

by sieve 2y ago

simonw 2y ago

I don't think pip is to blame for that. PyTorch is sadly an enormous space hog. I just started a fresh virtual environment with "python -m venv venv" - running "du -h" showed it to be 21MB. After running "venv/bin/pip install torch" it's now 431MB. The largest file in there is this one: 178M ./lib/python3.10/site-packages/torch/lib/libtorch_cpu.dylib There's a whole section of the uv manual dedicated just to PyTorch: https://docs.astral.sh/uv/guides/integration/pytorch/ https://docs.astral.sh/uv/guides/integration/pytorch/ (I just used find to locate as many libtorch_cpu.dylib files as possible on my laptop and deleted 5.5GB of them)

sieve 2y ago
I use uv pip to install dependencies for any LLM software I run. I am not sure if uv re-implements the pip logic or hands over resolution to pip. But it does not change the fact that I have multiple versions of torch + multiple installations of the same version of torch in the cache. Compare this to the way something like maven/gradle handles this and you have to wonder WTF is going on here.
- simonw 2y ago
  uv implements its own resolution logic independently of pip. Maybe your various LLM libraries are pinning different versions of Torch? Different Python versions each need their own separate Torch binaries as well. At least with uv you don't end up with separate duplicate copies of PyTorch in each of the virtual environments for each of your different projects!
  - sieve 2y ago
    > Different Python versions each need their own separate Torch binaries as well Found this the hard way. Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird.
    - woodruffw 2y ago
      > Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird. Yes, it's essentially that: CPython doesn't guarantee exact ABI stability between versions unless the extension (and its enclosing package) intentionally build against the stable ABI[1]. The courteous thing to do in the Python packaging ecosystem is to build "abi3" wheels that are stable and therefore don't need to be duplicated as many times (either on the index or on the installing client). Torch doesn't build these wheels for whatever reason, so you end up with multiple slightly different but functionally identical builds for each version of Python you're using. TL;DR: This happens because of an interaction between two patterns that Python makes very easy: using multiple Python versions, and building/installing binary extensions. In a sense, it's a symptom of Python's success: other ecosystems don't have these problems because they have far fewer people running multiple configurations simultaneously. [1]: https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html
      - sieve 2y ago
        My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. I am planning to shift some of my stuff to pypy (so a "fast" python exists, kind of). But some dependencies can be problematic, I have heard.
        Numerlor 2y ago
        python just didn't have much momentum until relatively recently, despite it's age. There are efforts to speed it up going on now backed by Microsoft. For pypy it's in a weird spot as the things it does fast are the ones you'd usually just offload to a module implemented in C
conradev 2y ago
uv should hard link files if they’re identical like Nix does If a package manager stores more than it needs to, it is a package manager problem.
- d0mine 2y ago
  What makes you think it doesn't?
  - conradev 2y ago
    This from above: > I just used find to locate as many libtorch_cpu.dylib files as possible on my laptop and deleted 5.5GB of them but maybe it wasn’t actually 5.5 GB!
    - rfoo 2y ago
      It was like a dozen different versions of libtorch_cpu.dylib. So hardlink does not help.
    - rat87 2y ago
      It probably is 5.5GB. UV caches versions of packages and symlinks to them but pytorch is infamous for many different versions especially for different features. And since its compiled dependency even if UV was to attempt the the more complicated method of symlinking to use a single version of identical files it wouldn't help much. You'd probably need to store binary diffs or chunks of files that are binary identical, at that point your code would probably start to resemble a file system in user space and time to switch to a particular version of the files (ie create thek as qctual files in filesystem) would be much higher. Also I believe uvs cache is separate from the pip cache so you could have different copies in both. I think there's a uv cache prune command. Arguably it should offer to install a from job to do it periodically
- Spivak 2y ago
  You're about to be pleasantly surprised then. https://docs.astral.sh/uv/reference/settings/#link-mode https://docs.astral.sh/uv/reference/settings/#link-mode It's even the default. Here's where it's implemented if you're curious https://github.com/astral-sh/uv/blob/f394f7245377b6368b9412d88575446602f22525/crates/uv-install-wheel/src/linker.rs#L280 https://github.com/astral-sh/uv/blob/f394f7245377b6368b9412d...
nighthawk454 2y ago
Testing this out locally, repro'd pretty similar numbers on macOS ARM and docker. Unfortunately the CPU-only build isn't really any smaller either, thought it was

RockRobotRock 2y ago

A lot of that is cuda blobs, right?

sieve 2y ago
ROCm. About 40%. But there is duplication there as well. Two 16GB folders containing the exact same version.
- IgorPartola 2y ago
  Sounds like a great use case for ZFS’s deduplication at block level.
  - 22c 2y ago
    or, you know.. symlinks
    - flakes 2y ago
      Main issue with symlink is needing to choose the source of truth— one needs to be the real file, and the other point to it. You also need to make sure they have the same lifetimes to prevent dangling links. Hardlink is somewhat better because both point to the same inode, but will also not work if the file needs different permissions or needs to be independently mutable from different locations. Reflink hits the sweetspot where it can have different permissions, updates trigger CoW preventing confusing mutations, and all while still reducing total disk usage.
      - 22c 2y ago
        I don't disagree but I think some of these problems could potentially be solved by having somewhat of a birds nest of a filesystem for large blobs, eg. /blobs/<sha256_sum>/filename.zip and then symlinking/reflinking filename.zip to wherever it needs to be in the source tree... It's more portable than hardlinks, solves your "source of truth" problem and has pretty wide platform support. Platforms that don't support symlinks/reflinks could copy the files to where they need to be then delete the blob store at the end and be no worse off than they are now. Anyway, I'm just a netizen making a drive-by comment.

paulddraper 2y ago

> torch Ah found the issue.

rsyring 2y ago

If you are using uv: $ uv cache prune $ uv cache clean Take your pick and schedule to run weekly/monthly.

code_biologist 2y ago
Light user of uv here. `prune` just saved me 1.1GiB. Thanks!

teej 2y ago

You can specify platform specific wheels in your pyproject.toml [[tool.uv.index]] name = "pytorch-cu124" url = "https://download.pytorch.org/whl/cu124" explicit = true [[tool.uv.index]] name = "pytorch-cpu" url = "https://download.pytorch.org/whl/cpu" explicit = true [tool.uv.sources] torch = [ { index = "pytorch-cu124", marker = "platform_system != 'Darwin'" }, { index = "pytorch-cpu", marker = "platform_system == 'Darwin'" }, ]

globular-toast 2y ago

Try building uv itself. Cargo used something like 40GiB of disk space somehow.

sieve 2y ago
I am only criticizing python because I am actively using it. The node ecosystem and the rust one seem to have their own issues. I have zero interest in either of them so I haven't looked into them in detail. However, I have to deal with node on occasion because a lot of JS/CSS tooling is written using it. It has a HUGE transitive dependency problem.

aragilar 2y ago

That problem is very much not pip (pip is only the installer), the issue is: * We have a conflict between being easy to use (people don't need to work out which version of cuda/which gpu settings/libraries/etc. to use) vs install size (it's basically the x86 vs arm issue, except at least 10 fold larger). Rather than making it the end-users problem, packages bundle all possible options into a single artifact (MacOS does this same, but see the 10 fold larger issue). * The are almost fundamental assumptions (and the newer Python packaging tools, including uv, very much rely on these) that Python packaging makes that are inherently about how a system should act (basically, frozen, no detection available apart from the "OS"), which do not align with having hardware/software that much be detected. One could do this via sdists, but Windows plus the issues around dynamic metadata make this a non-starter (and hence tools like conda, spack and others from the more scientific side of the ecosystem have been created—notably on the more webby side, this problems are solved either via vendoring non-python libraries, or making it someone/something else's problem, hence docker or the cloud for databases or other companion services). * Frankly, more and more developers have no idea how systems are built (and this isn't just a Python issue). Docker lets people hide their sins, with magical invocations that just work (and static linking in many cases sadly does the same). There are tools out of the hyperscalars which are designed to solve these problems, but they solve it by creating tools that experts can wrangle many systems and hence imply you have a team which can do the wrangling. Can this be solved? Maybe, but not by a new tool (on its own). It would require a lot of devs who may not see much improvement to their workflow change their workflow for others (to newer ones which remove the assumptions which are built in to the current workflows), plus a bunch of work by key stakeholders (and maybe even the open sourcing of some crown jewels), and I don't see that happening.

sieve 2y ago
> people don't need to work out which version of cuda/which gpu settings/libraries/etc. to use This is not true in my case. The regular pytorch does not work on my system. I had to download a version specific to my system from the pytorch website using --index-url. > packages bundle all possible options into a single artifact Cross-platform Java apps do it too. For e.g., see https://github.com/xerial/sqlite-jdbc https://github.com/xerial/sqlite-jdbc. But it does not become a clusterfuck like it does with python. After downloading gigabytes and gigabytes of dependencies repeatedly, the python tool you are trying to run will refuse to do so for random reasons. You cannot serve end-users a shit-sandwich of this kind. The python ecosystem is a big mess and, outside of a few projects like uv, I don't see anyone trying to build a sensible solution that tries to improve both speed/performance and packaging/distribution.

EdwardDiego 2y ago

> And I had to deal with Scala's SBT ("Simple Build Tool") in another life. I feel you.

amelius 2y ago
For someone who was just about to give Scala a try, what's wrong with it and are there alternative build tools?
- ATMLOTTOBEER 2y ago
  It defines a DSL for your build that looks roughly like Scala code. But… it’s not! And there is a confusing “resolution” system for build tasks/settings. It’s also slow as shit. See https://www.lihaoyi.com/post/SowhatswrongwithSBT.html https://www.lihaoyi.com/post/SowhatswrongwithSBT.html for a comprehensive takedown. If you’re interested in just playing around with scala I would use https://scala-cli.virtuslab.org https://scala-cli.virtuslab.org Or for larger projects, the thing the author of the linked article is plugging (mill).
- paulddraper 2y ago
  I've use sbt and don't have a problem. But Mill is very good. [1] [1] https://mill-build.org/mill/index.html https://mill-build.org/mill/index.html