5 ms·

There’s also no package caching across venvs(?). Every time I have to install something AI it’s the same 2GB torch + few hundred megabytes libs + cuda update of

• kzrdude 2y ago
  uv by astral is cleaning up and improving a lot in this area right now
  • nubinetwork 2y ago
    Pip, conda, uv... how many package managers do we need?
    • IanCal 2y ago
      Just one, but one that's decent. We'll keep inventing new ones until then. Maybe beyond.
      • cdchn 2y ago
        I don't understand the obsession with reimplementing Python utilities or ecosystem elements in Rust.
    • kzrdude 2y ago
      Uv's adoption rate says that there was a huge demand for it
    • maleldil 2y ago
      uv is essentially an improved version of pip, virtualenv and pip-tools, but from the outside it's pretty much the same thing. The CLI imitates most of pip's options too. It's not like conda, which is a completely different ecosystem, with dodgy integration with pip.
      • nerdponx 2y ago
        UV isn't "improved" yet, at least not until it reaches some semblance of feature parity. However it's a great experiment and I expect that it will end up being a net benefit to the Python ecosystem.
        • kzrdude 2y ago
          Uv is never going to be a drop in replacement. But they are already improving upon pip when it comes to cache and fast install. Uv will replace pip I think even if it doesn't have exactly the same features.
      • mr_mitm 2y ago
        It aims at replacing them all, including venv, pipx, and more. > Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more. > Our goal is to evolve uv into such a tool. https://github.com/astral-sh/uv/blob/main/README.md https://github.com/astral-sh/uv/blob/main/README.md
        • maleldil 2y ago
          I meant to say what uv is _right now_, which AFAIK is replacement for virtualenv, some of pip, and pip-tools. I understand they have a vision for the future, but I'll believe it when I see it. As it stands, they're already a very useful tool.
    • nerdponx 2y ago
      Conda is a lot more than a Python package manager specifically, it's more like an entire userspace (akin to Homebrew or Pkgsrc), that happens to be focused on Python because that's where it originated.
  • lelandbatey 2y ago
    I thought "poetry" was the package manager that was on the ascent in Python, is that no longer true?
    • iamaredpanda 2y ago
      Poetry isn't really a package manager as much as packaging management (as in packaging and publishing your specific package) and dependency management.
    • kzrdude 2y ago
      Poetry style solutions still increasing I think but there is more than poetry competing for it. Uv too will soon join the group competing in this area. Poetry is challenged by not being as Python PEP compatible (it predated some of the "standards") as well as performance (resolve, lock and install)
• robertlagrant 2y ago
  It would be nice if all packages (and each version) were cached somewhere global and then just referred to by a local environment, to deduplicate. I think Maven does this.
  • LelouBil 2y ago
    Yes, pnpm does it too
    • adithyassekhar 2y ago
      God I love pnpm. Literally the only thing I install with npm.
  • Galanwe 2y ago
    The downloaded packages themselves are shared in pip's cache. Once installed though they cannot be shared, for a number of reasons: - Compiled python code is (can be) generated and stored alongside the installed source files, sharing that installed package between envs with different python versions would lead to weird bugs - Installed packages are free to modify their own files (especially when they embed data files) - Virtualenv are essentially non relocatable and installed package manifests use absolute paths Really that shouldn't be a problem though. Downloaded packages are cached by pip, as well as locally built wheels of source packages, meaning once you've installed a bunch of packages already, installing new ones in new environments is a mere file extraction. The real issue IMHO causing install slowness is that package maintainers have not taken the habit to be mindful of the size of their packages. If you install multiple data science packages for instances, you can quickly get to multi GB virtualenvs, for the sole reason that package maintainers didn't bother to remove (or separate as "extra") their test datasets, language translation files, static assets, etc.
    • lye 2y ago
      They can be, pip developers just have to care about this. Nothing you described precludes file-based deduplication, which is what pnpm does for JS projects: it stores all library files in a global content-addressable directory, and creates {sym,hard,ref}links depending on what your filesystem supports. Being able to mutate files requires reflinks, but they're supported by e.g. XFS, you don't have to go to COW filesystems that have their own disadvantages. You can do something like that manually for virtualenvs too, by running rmlint on all of them in one go. It will pick an "original" for each duplicate file, and will replace duplicates with reflinks to it (or other types of links if needed). The downside is obvious: it has to be repeated as files change, but I've saved a lot of space this way. Or just use a filesystem that supports deduplication natively like btrfs/zfs. https://github.com/sahib/rmlint https://github.com/sahib/rmlint
• woodruffw 2y ago
  Unless you've configured `pip` in a special way, I don't think this is true? All invocations of `pip` across different environments should share the global HTTP cache, which you can confirm with `pip cache dir`. (What `pip` can't cache is the actual extraction of distributions between venvs, because distribution installation sometimes includes arbitrary code. So you might be hitting that, but you should never need to re-download the same wheels or sdists while the cache is hot.)
  • wruza 2y ago
    I’ll try to check, thanks! But I usually let things as they are, and never touched pips settingss. If my intervention is required, it is different from ”works out of box”. In my AI cases I’ve seen one of the pips picking from where it stopped occasionally (rare cases where you can re-setup), but that doesn’t go across projects. The fact that I’m on windows and have four different python installations may add to this issue, but I’m pretty sure and aware which `which python` I’m using when. AI projects tend to download their own minicondas, but it’s unclear why they don’t share a cache (if that’s the issue even). Anyway, windows or not, out of box user experience is barely acceptable.
    • wruza 2y ago
      Update: All my installations including all "condas" known to me return c:\users\user\appdata\local\pip\cache I just tried to unpack and install the same `text-generation-webui-main` that I installed yesterday and it started to download 2.5gb torch at 12mb/s eta 0:03:07. I made screenshots for those who will claim it's not true. There was also a full ready-to-use in advance 2.5gb file in pip\cache\http\...\...\...\{hashfilename}, which is very fun to navigate to btw. I interrupted the download and activated this new environment: [prompt] cd text-generation-webui-main [prompt] installer_files\conda\condabin\conda activate installer_files\env [prompt] pip cache dir c:\users\user\appdata\local\pip\cache I have no idea how they do that.
      • woodruffw 2y ago
        Honestly, I don't know what's happening there. My only other guess is that the wheel you're downloading is so big that it somehow busts `pip`'s cache, meaning that you end up always re-downloading it. But I'm not sure there's even a cache limit like that. It sounds like this might be a bug, either in `pip` or in the caching middleware it uses. You should consider filing a bug upstream with your observed behavior. Edit: 2.5GB is coincidentally just over the maximum i32 value, so it's possible this is even something silly like a `seek(3)` or other file offset overflowing and resulting in a bad cache. Edit 2: `pip` is using `CacheControl`, which I maintain. I suspect what's happening is that `CacheControl`'s underlying use of `msgpack` is failing above stores of `2^32 - 1` bytes in individual binary objects, since that's the `msgpack` limit. https://github.com/msgpack/msgpack/blob/master/spec.md https://github.com/msgpack/msgpack/blob/master/spec.md I'll try and fix that, if that's what it actually is.
• OptionOfT 2y ago
  And this gets complicated with having multiple versions of Python. I can have 5 venvs for 3 versions of Python, and nothing is shared.
• EnergyAmy 2y ago
  Those issues are handled quite nicely with nix: https://github.com/nix-community/poetry2nix/ https://github.com/nix-community/poetry2nix/
• nerdponx 2y ago
  Pip in its default configuration uses a global wheel/sdist cache. This might be a problem on your end.
• formerly_proven 2y ago
  pip has a global cache, you're probably using unpinned dependencies and getting a different patch version every other day.
• amelius 2y ago
  Another problem: You can only use 1 core with pip install. I.e., you cannot use the equivalent of --jobs=$(nproc) This aggravates the problem.
  • woodruffw 2y ago
    This is (in part) a knock-on consequence of Python packaging's dynamic metadata: installing a package sometimes means running arbitrary Python code to determine the package's dependencies, meaning that the resolver not only has to backtrack but also has to run arbitrary code in a fixed state (i.e., with references to other dependencies that should not change in parallel streams of execution).
• amelius 2y ago
  You're lucky that you're not installing on an nVidia Jetson system that is pinned to Python 3.8 which forces you to use old versions for almost everything else, unless you want to give up on CUDA.
  • ethbr1 2y ago
    NVidia embedded: the worst embedded tooling, except for all the other ones. Every time I peek at "business as usual" in embedded land, I decide I'd rather professionally orient myself in another direction.
  • wruza 2y ago
    Not that lucky, since I’m using 3.12 (iirc) for my opencv-related projects, 3.10 for… can’t remember what, probably kohya_ss, then 3.9 for few tools like rembg and a few left from nodejs setup via choco, not sure if it still needs them. Not counting minicondas here cause it’s hard to even guess.
• martinky24 2y ago
  This isn't true...?
  • wruza 2y ago
    I shouldn’t have this issue then. But a couple of days ago I tried new versions of kohya, sdwebui and tgwebui and none of their convoluted setup batches shared downloads. Btw sdwebui broke itself irrepairably in half an hour by failing installing an extension that works in a sibling folder. Kohya spoiled right away without my intervention. Only tgwebui survived. I’m not doung anything wrong, since there’s not much to do apart from waiting for hours. I’m not updating in-folder because of deep scars from the last time I did that. Maybe these issues are not inherent to python, but for some reason AI projects tend to go out of their way to make easy installers. That must mean something at least.
• woodson 2y ago
  The wheels are cached (check ~/.cache/pip/wheels on Linux/Macos)