4 ms·

I use uv pip to install dependencies for any LLM software I run. I am not sure if uv re-implements the pip logic or hands over resolution to pip. But it does no

by sieve 2y ago

I use uv pip to install dependencies for any LLM software I run. I am not sure if uv re-implements the pip logic or hands over resolution to pip. But it does not change the fact that I have multiple versions of torch + multiple installations of the same version of torch in the cache. Compare this to the way something like maven/gradle handles this and you have to wonder WTF is going on here.

simonw 2y ago
uv implements its own resolution logic independently of pip. Maybe your various LLM libraries are pinning different versions of Torch? Different Python versions each need their own separate Torch binaries as well. At least with uv you don't end up with separate duplicate copies of PyTorch in each of the virtual environments for each of your different projects!
- sieve 2y ago
  > Different Python versions each need their own separate Torch binaries as well Found this the hard way. Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird.
  - woodruffw 2y ago
    > Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird. Yes, it's essentially that: CPython doesn't guarantee exact ABI stability between versions unless the extension (and its enclosing package) intentionally build against the stable ABI[1]. The courteous thing to do in the Python packaging ecosystem is to build "abi3" wheels that are stable and therefore don't need to be duplicated as many times (either on the index or on the installing client). Torch doesn't build these wheels for whatever reason, so you end up with multiple slightly different but functionally identical builds for each version of Python you're using. TL;DR: This happens because of an interaction between two patterns that Python makes very easy: using multiple Python versions, and building/installing binary extensions. In a sense, it's a symptom of Python's success: other ecosystems don't have these problems because they have far fewer people running multiple configurations simultaneously. [1]: https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html
    - sieve 2y ago
      My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. I am planning to shift some of my stuff to pypy (so a "fast" python exists, kind of). But some dependencies can be problematic, I have heard.
      - Numerlor 2y ago
        python just didn't have much momentum until relatively recently, despite it's age. There are efforts to speed it up going on now backed by Microsoft. For pypy it's in a weird spot as the things it does fast are the ones you'd usually just offload to a module implemented in C
        sieve 2y ago
        > There are efforts to speed it up Well, the extensions are going to complicate this a lot. A fast JIT interpreter cannot reach into a library and do its magic there the way HotSpot/V8 can with native Java/JS code.
        simonw 2y ago
        As a long time Pythonista I was going to push back against your suggestion that Python didn't have much momentum until recently, but then I looked at the historic graph on https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ and yeah, Python's current huge rise in popularity didn't really get started until around 2018. (TIOBE's methodology is a bit questionable though, as far as I can tell it's almost entirely based on how many search engine hits they get for "X programming". https://www.tiobe.com/tiobe-index/programminglanguages_definition/ https://www.tiobe.com/tiobe-index/programminglanguages_defin...)
        tialaramex 2y ago
        Yes, TIOBE is garbage. The biggest problem is that because they're coy about methodology we don't even know what we're talking about. Rust's "Most Loved" Stack Overflow numbers were at least a specific thing where you can say OK that doesn't mean there's more Rust software or that Rust programmers get paid more, apparently the people programming in Rust really like Rust, more so than say, Python programmers loved Python - so that's good to know, but it's that and not anything else.
  - zahlman 2y ago
    >Something to do with breakage in ABI perhaps. There is a "stable ABI" which is a subset of the full ABI, but no requirement to stick to it. The ABI effectively changes with every minor Python version - because they're constantly trying to improve the Python VM, which often involves re-working the internal representations of built-in types, etc. (Consider for example the improvements made to dictionaries in Python 3.6 - https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-compactdict https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-compa... .) Of course they try to make proper abstracted interfaces for those C structs, but this is a 34 year old project and design decisions get re-thought all the time and there are a huge variety of tiny details which could change and countless people with legacy code using deprecated interfaces. The bytecode also changes with every minor Python version (and several times during the development of each). The bytecode file format is versioned for this reason, and .pyc caches need to be regenerated. (And every now and then you'll hit a speed bump, like old code using `async` as an identifier which subsequently becomes a keyword. That hit TensorFlow once: https://stackoverflow.com/questions/51337939 https://stackoverflow.com/questions/51337939 .)
    - sieve 2y ago
      Very different way of doing things compared to the JVM which is what I have most experience with. Was some kind of FFI using dlopen and sharing memory across the vm boundary ever considered in the past, instead of having to compile extensions alongside a particular version of python? I remember seeing some ffi library, probably on pypi. But I don't think it is part of standard python.
      - zahlman 2y ago
        You can in fact use `dlopen`, via the support provided in the `ctypes` standard library. `freetype-py` (https://github.com/rougier/freetype-py https://github.com/rougier/freetype-py) is an example of a project that works this way. To my understanding, though, it's less performant. And you still need a stable ABI layer to call into. FFI can't save you if the C code decides in version N+1 that it expects the "memory shared across the vm boundary" to have a different layout.

by sieve 2y ago

simonw 2y ago

uv implements its own resolution logic independently of pip. Maybe your various LLM libraries are pinning different versions of Torch? Different Python versions each need their own separate Torch binaries as well. At least with uv you don't end up with separate duplicate copies of PyTorch in each of the virtual environments for each of your different projects!

sieve 2y ago
> Different Python versions each need their own separate Torch binaries as well Found this the hard way. Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird.
- woodruffw 2y ago
  > Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird. Yes, it's essentially that: CPython doesn't guarantee exact ABI stability between versions unless the extension (and its enclosing package) intentionally build against the stable ABI[1]. The courteous thing to do in the Python packaging ecosystem is to build "abi3" wheels that are stable and therefore don't need to be duplicated as many times (either on the index or on the installing client). Torch doesn't build these wheels for whatever reason, so you end up with multiple slightly different but functionally identical builds for each version of Python you're using. TL;DR: This happens because of an interaction between two patterns that Python makes very easy: using multiple Python versions, and building/installing binary extensions. In a sense, it's a symptom of Python's success: other ecosystems don't have these problems because they have far fewer people running multiple configurations simultaneously. [1]: https://docs.python.org/3/c-api/stable.html https://docs.python.org/3/c-api/stable.html
  - sieve 2y ago
    My use of python is somewhat recent. But the two languages that I have used a lot of - Java and JS - have interpreters that were heavily optimized over time. I wonder why that never happened with python and, instead, everyone continues to write their critical code in C/Rust. I am planning to shift some of my stuff to pypy (so a "fast" python exists, kind of). But some dependencies can be problematic, I have heard.
    - Numerlor 2y ago
      python just didn't have much momentum until relatively recently, despite it's age. There are efforts to speed it up going on now backed by Microsoft. For pypy it's in a weird spot as the things it does fast are the ones you'd usually just offload to a module implemented in C
      - sieve 2y ago
        > There are efforts to speed it up Well, the extensions are going to complicate this a lot. A fast JIT interpreter cannot reach into a library and do its magic there the way HotSpot/V8 can with native Java/JS code.
      - simonw 2y ago
        As a long time Pythonista I was going to push back against your suggestion that Python didn't have much momentum until recently, but then I looked at the historic graph on https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ and yeah, Python's current huge rise in popularity didn't really get started until around 2018. (TIOBE's methodology is a bit questionable though, as far as I can tell it's almost entirely based on how many search engine hits they get for "X programming". https://www.tiobe.com/tiobe-index/programminglanguages_definition/ https://www.tiobe.com/tiobe-index/programminglanguages_defin...)
        tialaramex 2y ago
        Yes, TIOBE is garbage. The biggest problem is that because they're coy about methodology we don't even know what we're talking about. Rust's "Most Loved" Stack Overflow numbers were at least a specific thing where you can say OK that doesn't mean there's more Rust software or that Rust programmers get paid more, apparently the people programming in Rust really like Rust, more so than say, Python programmers loved Python - so that's good to know, but it's that and not anything else.
- zahlman 2y ago
  >Something to do with breakage in ABI perhaps. There is a "stable ABI" which is a subset of the full ABI, but no requirement to stick to it. The ABI effectively changes with every minor Python version - because they're constantly trying to improve the Python VM, which often involves re-working the internal representations of built-in types, etc. (Consider for example the improvements made to dictionaries in Python 3.6 - https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-compactdict https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-compa... .) Of course they try to make proper abstracted interfaces for those C structs, but this is a 34 year old project and design decisions get re-thought all the time and there are a huge variety of tiny details which could change and countless people with legacy code using deprecated interfaces. The bytecode also changes with every minor Python version (and several times during the development of each). The bytecode file format is versioned for this reason, and .pyc caches need to be regenerated. (And every now and then you'll hit a speed bump, like old code using `async` as an identifier which subsequently becomes a keyword. That hit TensorFlow once: https://stackoverflow.com/questions/51337939 https://stackoverflow.com/questions/51337939 .)
  - sieve 2y ago
    Very different way of doing things compared to the JVM which is what I have most experience with. Was some kind of FFI using dlopen and sharing memory across the vm boundary ever considered in the past, instead of having to compile extensions alongside a particular version of python? I remember seeing some ffi library, probably on pypi. But I don't think it is part of standard python.
    - zahlman 2y ago
      You can in fact use `dlopen`, via the support provided in the `ctypes` standard library. `freetype-py` (https://github.com/rougier/freetype-py https://github.com/rougier/freetype-py) is an example of a project that works this way. To my understanding, though, it's less performant. And you still need a stable ABI layer to call into. FFI can't save you if the C code decides in version N+1 that it expects the "memory shared across the vm boundary" to have a different layout.