5 ms·

I agree. If any of the stuff was worthwhile to pursue, it would be merged into pip.

woodruffw 1y ago
This doesn't generalize: you could have said the same thing about pip versus easy_install, but pip clearly has worthwhile improvements over easy_install that were never merged back into the latter.
zahlman 1y ago
Pyx represents the server side, not the client side. The analogue in the pre-existing Python world is PyPI. Many ideas are being added to recent versions of pip that are at least inspired by what uv has done — and many things are possible in uv specifically because of community-wide standards development that also benefits pip. However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). That has a lot to do with why I'm making PAPER. For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process. It remembers entire wheels, but finding them in that cache requires knowing the URL from which they were downloaded. Because PyPI organizes the packages in its own database with its own custom URL scheme, pip would have to reach out to PyPI across the Internet in order to figure out where it put its own downloads!
- notatallshaw 1y ago
  > However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. I'm not saying here that pip doesn't have some gnarly internal details, just that the bigger thing holding it back is the lack of maintainer resources. > For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process. I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy. That said, I do think there are other good examples to your point, like uv's ability to prefetch package metadata, I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver.
  - zahlman 1y ago
    Good to see you again. > FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. Fair enough. I'm sure if someone were paying me a competitive salary to develop my projects, they'd be getting done much faster, too. > I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy. I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files. > uv's ability to prefetch package metadata, You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays. > I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver. Ugh, yeah. I know the resolution logic has been extracted as a specific package, but it's been challenging trying to figure out how to actually use that in a project that isn't pip.
    - notatallshaw 1y ago
      > I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files. I'm talking about the fact there is no `uv pip download` or `uv pip wheel`. I'm sure we'll discuss when you add this feature to uv, but I personally find uv's cache already grows too big too fast, so adding more to it makes me concerned. > You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays. Yeah, one of the way uv speeds up resolving is it pre-downloads .metadata files and checks if their requirements are identical to versions it had already checked, so it can quickly rule them out. It's a clever use of a collector and an advanced resolver.
  - 1y ago
- BrenBarn 1y ago
  > For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. Conda has been able to do this for years.
  - zahlman 1y ago
    Good to know. "which in turn are not all original" may have been an understatement. I don't know a lot of these details, because my interest in these things is more recent, and not so much driven by the complex multi-language projects.
lijok 1y ago
It’s not that complex - just try it
nilamo 1y ago
Pip is broken and has been for years, they're uninterested in fixing the search. Or even removing the search or replacing it with a message/link to the package index. imo, if pip's preference is to ship broken functionality, then what is/is not shipped with pip is not meaningful.
- woodruffw 1y ago
  This is not a charitable interpretation. The more charitable read is that fixing search is non-trivial and has interlocking considerations that go beyond what pip's volunteer maintainers reasonably want to or can pick up. (And for the record: it isn't their fault at all. `pip search` doesn't work because PyPI removed the search API. PyPI removed that API for very good reasons[1].) [1]: https://github.com/pypa/pip/issues/5216 https://github.com/pypa/pip/issues/5216
  - nilamo 1y ago
    That was 7 years ago. If it's not coming back, the CLI should make that clear, instead of giving a temporary "cannot connect" message that implies it could work, if you wait a minute and try again.
    - woodruffw 1y ago
      It was three years ago; 2018 is when they considered removing the command, not when the search API was actually removed from PyPI. And this is part of the interlocking considerations I mentioned: there are private indices that supply the XML-RPC API, and breaking them doesn't seem justifiable[1]. Edit: fixed the link. [1]: https://github.com/pypa/pip/issues/5216#issuecomment-1235329876 https://github.com/pypa/pip/issues/5216#issuecomment-1235329...
      - nilamo 1y ago
        Does that seem like a real solution to you? That it's ok to represent a never-functional operation as one that might maybe work? ...because it could work of you jump through a bunch of undocumented hoops? It's so wild to me that so many people are apparently against making a user-friendly update. The whole thing seems very against pep8 (its surprising, complicated, non-specific, etc)
benrutter 1y ago
I'm assuming by pip you mean pypi (the package registry) - I think you're making the mistake of thinking it has a lot more resources than it does, because of how much it punches above its weight. Pypi is powered by warehouse which has around 3 developers maintaining it[0]. They're doing an awesome job, but the funding and resource available to them are probably substantially less than Astral could have with a paid offering. [0] https://github.com/pypi/warehouse/graphs/contributors https://github.com/pypi/warehouse/graphs/contributors

This doesn't generalize: you could have said the same thing about pip versus easy_install, but pip clearly has worthwhile improvements over easy_install that were never merged back into the latter.

zahlman 1y ago

Pyx represents the server side, not the client side. The analogue in the pre-existing Python world is PyPI. Many ideas are being added to recent versions of pip that are at least inspired by what uv has done — and many things are possible in uv specifically because of community-wide standards development that also benefits pip. However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). That has a lot to do with why I'm making PAPER. For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process. It remembers entire wheels, but finding them in that cache requires knowing the URL from which they were downloaded. Because PyPI organizes the packages in its own database with its own custom URL scheme, pip would have to reach out to PyPI across the Internet in order to figure out where it put its own downloads!

notatallshaw 1y ago
> However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. I'm not saying here that pip doesn't have some gnarly internal details, just that the bigger thing holding it back is the lack of maintainer resources. > For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process. I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy. That said, I do think there are other good examples to your point, like uv's ability to prefetch package metadata, I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver.
- zahlman 1y ago
  Good to see you again. > FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. Fair enough. I'm sure if someone were paying me a competitive salary to develop my projects, they'd be getting done much faster, too. > I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy. I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files. > uv's ability to prefetch package metadata, You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays. > I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver. Ugh, yeah. I know the resolution logic has been extracted as a specific package, but it's been challenging trying to figure out how to actually use that in a project that isn't pip.
  - notatallshaw 1y ago
    > I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files. I'm talking about the fact there is no `uv pip download` or `uv pip wheel`. I'm sure we'll discuss when you add this feature to uv, but I personally find uv's cache already grows too big too fast, so adding more to it makes me concerned. > You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays. Yeah, one of the way uv speeds up resolving is it pre-downloads .metadata files and checks if their requirements are identical to versions it had already checked, so it can quickly rule them out. It's a clever use of a collector and an advanced resolver.
- 1y ago
BrenBarn 1y ago
> For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. Conda has been able to do this for years.
- zahlman 1y ago
  Good to know. "which in turn are not all original" may have been an understatement. I don't know a lot of these details, because my interest in these things is more recent, and not so much driven by the complex multi-language projects.

lijok 1y ago

It’s not that complex - just try it

nilamo 1y ago

Pip is broken and has been for years, they're uninterested in fixing the search. Or even removing the search or replacing it with a message/link to the package index. imo, if pip's preference is to ship broken functionality, then what is/is not shipped with pip is not meaningful.

woodruffw 1y ago
This is not a charitable interpretation. The more charitable read is that fixing search is non-trivial and has interlocking considerations that go beyond what pip's volunteer maintainers reasonably want to or can pick up. (And for the record: it isn't their fault at all. `pip search` doesn't work because PyPI removed the search API. PyPI removed that API for very good reasons[1].) [1]: https://github.com/pypa/pip/issues/5216 https://github.com/pypa/pip/issues/5216
- nilamo 1y ago
  That was 7 years ago. If it's not coming back, the CLI should make that clear, instead of giving a temporary "cannot connect" message that implies it could work, if you wait a minute and try again.
  - woodruffw 1y ago
    It was three years ago; 2018 is when they considered removing the command, not when the search API was actually removed from PyPI. And this is part of the interlocking considerations I mentioned: there are private indices that supply the XML-RPC API, and breaking them doesn't seem justifiable[1]. Edit: fixed the link. [1]: https://github.com/pypa/pip/issues/5216#issuecomment-1235329876 https://github.com/pypa/pip/issues/5216#issuecomment-1235329...
    - nilamo 1y ago
      Does that seem like a real solution to you? That it's ok to represent a never-functional operation as one that might maybe work? ...because it could work of you jump through a bunch of undocumented hoops? It's so wild to me that so many people are apparently against making a user-friendly update. The whole thing seems very against pep8 (its surprising, complicated, non-specific, etc)

benrutter 1y ago

I'm assuming by pip you mean pypi (the package registry) - I think you're making the mistake of thinking it has a lot more resources than it does, because of how much it punches above its weight. Pypi is powered by warehouse which has around 3 developers maintaining it[0]. They're doing an awesome job, but the funding and resource available to them are probably substantially less than Astral could have with a paid offering. [0] https://github.com/pypi/warehouse/graphs/contributors https://github.com/pypi/warehouse/graphs/contributors