7 ms·

Why it took 4 years to get a lock files specification

123 points by birdculture 1y ago

flowerthoughts 1y ago
> A lock file is meant to record all the dependencies your code needs to work along with how to install those dependencies. It's about dependency locking in Python packaging.
klustregrif 1y ago
Overall it feels like UV is the best thing to happen to python packaging in two decades, by circumventing the endless non productive discussions on peps and instead just building something that works and is fast. In Rust naturally.
- webology 1y ago
  UV is great but also builds on existing PEPs. While they have the ability experiment (which is great), they also benefit from those "endless non productive discussions on peps" as you called it. I think UV proves that dedicated funding can make a huge impact on a project and benefit for the community. They are a doing a damn, good job.
  - tormeh 1y ago
    They mostly took inspiration from other languages for UV. Cargo (Rust) was a huge inspiration, but they got stuff from Ruby as well, I believe. There was an episode on "the changelog" about it. Don't remember them saying anything about PEPs, although that might just be me not having listened to the entire thing. However, the Charlie Marsh was extremely insistent on the advantages of being a polyglot and hiring people with diverse programming experiences. So I think it's quite safe to assume that played a bigger role than just PEPs.
    - zahlman 1y ago
      > Don't remember them saying anything about PEPs, although that might just be me not having listened to the entire thing. However, the Charlie Marsh was extremely insistent on the advantages of being a polyglot and hiring people with diverse programming experiences. So I think it's quite safe to assume that played a bigger role than just PEPs. Not that they took inspiration from PEPs, but they sought to implement those standards (for interoperability) and have been active in the discussion of new packaging-related PEPs.
  - the__alchemist 1y ago
    I can't speak for the UV team. My 2C on how I would treat the PEPs: If there is an accepted one, and implementing it doesn't go too strongly against your competing design goals, do it for compatibility. This does not imply that the PEP is driving your design, or required to make your software. It is a way to improve compatibility.
  - zahlman 1y ago
    > dedicated funding can make a huge impact on a project Where does Astral's funding come from, anyway?
  - nxpnsv 1y ago
    Rye was already pretty good before it was donated to astral and renamed to uv though...
    - the__alchemist 1y ago
      I wrote an earlier one (rust, inspired by Cargo, managed deps, scripts, and py versions) called PyFlow that I abandoned, because nobody used it. "Why should I use this when we have pip, pipenv, and poetry?"
      - aiiizzz 1y ago
        Programming is 80% marketing, eh?
      - zahlman 1y ago
        It probably doesn't help that there seem to be a few other projects out there with the same name or a similar name, which do completely different things.
- smallmancontrov 1y ago
  Yet more proof that "the best way to do anything in python is to not do it in python."
  - paulddraper 1y ago
    Find something better than Django. Or matplotlib. Or PyTorch.
    - maleldil 1y ago
      PyTorch is a C++ project with a Python wrapper.
    - aeve890 1y ago
      >Find something better than Django Rails. QED.
      - paulddraper 1y ago
        Rails is a good project no doubt. Django comes batteries included for basic apps, including an admin.
        aeve890 1y ago
        Counterpoint, you still need a 3rd party library to implement a sane rest API in Django. Same with async jobs.
        paulddraper 1y ago
        I would definitely not use Django to do REST API. It can work...but that's not what it was designed for.
    - deleted 1y ago
      [deleted]
  - deleted 1y ago
    [deleted]
- scrollaway 1y ago
  Can someone explain why UV is so praised when Poetry achieved a lot of that good several years earlier? Maybe I missed the train but I’ve been using poetry since its first version and all the benefits people praise UV for have long been in my builds.
  - mixmastamyk 1y ago
    Ruff was the gateway drug for me, much better than the black/isort/etc combo. Led me to try uv, which fixed a couple of egregious bugs in pip. Add speed and it’s a no brainer. I don’t think poetry has these advantages, and heard about bugs early on. Is that completely fair? Probably not. But it’s obvious astral tools have funding and a competent team.
  - Game_Ender 1y ago
    Speed and simplicity. Now I can fetch one binary on a system and in seconds fetch everything needed to run a Python tool or work on a code base. I can do all that without having to even worry about virtual ends, or Python versions too.
  - zahlman 1y ago
    Poetry was in some senses before its time; people were frustrated with pip, but didn't really understand the problems they were encountering, meanwhile the ecosystem had started trying to move away from doing everything with Setuptools for everyone. There's a ton that I'd love to explain here about the history of pyproject.toml etc. but the point is that Poetry had its own idea about what it would mean to be an all-in-one tool... and so did everyone else. Meanwhile, the main packaging people had been designing standards with the expectation of making a UNIX-style tool ecosystem work. Everyone seems to like uv's answer better, but I'm still a believer in composable toolchains, since I've already been using those forever. I actually was an early Poetry adopter for a few reasons. In particular, I would have been fine sticking with Setuptools for building projects if it had supported PEPS 517/518/621 promptly. 621 came later but Poetry's workaround was nicer than Setuptools' to me. And it was easier to use with the new pyproject.toml setup, and I really wanted to get away from the expectation of using setup.py even for pure-Python projects. But that was really it. The main selling point point of Poetry was (and is) that they offered a lockfile and dependency resolution, but these weren't really things I personally needed. So there was nothing really to outweigh the downsides: * The way Poetry does the actual installation is, as far as I can tell, not much different from what pip does. And there are a ton of problems with that model. * The early days of Poetry were very inconsistent in terms of installation and upgrade procedures. There was at least once that it seemed that the only thing that would work was a complete manual uninstall and reinstall, and I had to do research to figure out what I had to remove for the uninstallation as there was nothing provided to automate that. * In the end, Poetry didn't have PEP 621 support for about four years (https://github.com/python-poetry/roadmap/issues/3 https://github.com/python-poetry/roadmap/issues/3 ; the OP was already almost a year after PEP acceptance in https://discuss.python.org/t/_/5472/109 https://discuss.python.org/t/_/5472/109); there was this whole thing about how you were supposed to use pyproject.toml to describe the basic metadata of your project for packaging purposes, but if you used Poetry then you used Masonry to build, and that meant using a whole separate metadata configuration. Setuptools was slow in getting PEP 621 support off the ground (and really, PEP 621 itself was slow! It's hard to justify expecting anyone to edit pyproject.toml manually without PEP 621!), but Poetry was far slower still. I had already long given up on it at that point. So for me, Poetry was basically there to provide Masonry, and Masonry was still sub-par. I was still creating venvs manually, using `twine` to upload to PyPI etc. because that's just how I think. Writing something like `poetry shell` (or `uv run`) makes about as much sense to me as `git run-unit-tests` would.
- frizlab 1y ago
  Yeah yeah, in rust… You know rust is not the only language in which one can write good code, right?
  - zx8080 1y ago
    Good point! What are the other peoper languages in addition to golang and maybe c++ for such tooling? However people... don't like (hate? fear?) c++ nowadays. And I doubt that golang fits well here. So what else? (I'm not pushing towards rust! It just seems rust got into some vacuum, however not fitting it fully like a square thing in a round hole.)
    - frizlab 1y ago
      That I’m sure of: Swift is extremely underrated and actually very close functionally to rust. I don’t know other languages, but I’ve heard a lot of good things about zig.
      - zx8080 1y ago
        > swift Not everyone would want to get into Apple vendor-locked language (formally not, but it's like saying Chromium is opensource and not related to Alphabet/Google) and its development environment. Especially after JetBrains closed its swift oriented ide [0]. I hate xcode, sorry, after developing for too many years in it. 0: https://www.jetbrains.com/objc/ https://www.jetbrains.com/objc/
        frizlab 1y ago
        Swift has an open-source and fully supported first party LSP integration and VSCode plugin, as well as a very active and welcoming community (forums). Using Swift w/o Xcode is now 100% possible and very much recommended by some people when no Apple development-platform specifics are required (native iOS/macOS apps). The language was initially very much Apple-platforms oriented (had to be), but now that pretty much all the Apple stuff works well they moved beyond that. Finally where the language comes from does not impact whether you can write good code with it.
charcircuit 1y ago
The post didn't answer why it took over 4 years. Why couldn't everyone be flied to the same place and have it all figured out in a week instead of having the process drag on for years?
- quotemstr 1y ago
  An endless multiplication of veto points in a consensus culture is a failure mode. Funny thing is people embedded in such a culture will see its slowness as a feature and sneer at the world as it leaves them behind. Python was better with a BDFL. The best thing that could happen to Python right now would be for someone to fork it. Maybe just have Astral run away with the whole language. This lock file format should have taken a weekend, not four damn years.
  - vinnymac 1y ago
    A governance model where “time-based decision thresholds” directly damaged veto power as time passes could be used to harness this sort of BDFL-only power in a consensus culture.
  - zahlman 1y ago
    > Python was better with a BDFL. The problem is that the BDFL also didn't want to think about packaging, certainly not the issues that inspired Conda. I wouldn't mind seeing a PyPy-like fork done in Rust. Maybe take the opportunity to redesign the standard library, too.
pjmlp 1y ago
No worries, it is taking longer to be able to rely on C++ modules for portable code, having Valhala available on JVM, Android supporting anything beyond Java 17, or a JIT in CPython, some things take their time to finally become widespread, for various kinds of reasons.
deleted 1y ago
[deleted]
eqvinox 1y ago
I was hoping part of this delay was due to people arguing lock files are poor engineering to begin with, but alas, no mention of that. I guess we've just given up on any kind of package version flexibility.
- lexicality 1y ago
  > I guess we've just given up on any kind of package version flexibility. Presumably because decades of experience has demonstrated that humans are extremely bad at maintaining compatibility between releases and dealing with fallout from badly specified package versions is probably second only to NULL in terms of engineering time wasted? Or possibly it's just because a lot of the Python ecosystem doesn't even try and follow semver and you have no guarantee that any two versions are compatible with each other without checking the changelog and sacrificing a small chicken...
  - BugsJustFindMe 1y ago
    > Or possibly it's just because a lot of the Python ecosystem doesn't even try and follow semver Even if they try, semver can only ever be a suggestion of the possibility of compatibility at best because people are not oracles and they misjudge the effects of changes all the time.
    - 9dev 1y ago
      The PHP ecosystem is almost universally semver and has been going strong for years now, without any major accidental breaking change related outages. A little discipline and commitment to backwards compatibility and it isn’t too hard, really?
      - withinboredom 1y ago
        PHP compatibility and its commitment (across the ecosystem) to backwards compatibility is actually pretty cool. If there is one thing PHP does right, it’s this.
    - 1y ago
huflungdung 1y ago
[dead]
JimDabell 1y ago
Unfortunately: > Some of uv's functionality cannot be expressed in the pylock.toml format; as such, uv will continue to use the uv.lock format within the project interface. > However, uv supports pylock.toml as an export target and in the uv pip CLI. — https://docs.astral.sh/uv/concepts/projects/layout/#the-lockfile https://docs.astral.sh/uv/concepts/projects/layout/#the-lock...
- testdelacc1 1y ago
  That is pretty unfortunate. Would have been cool if all of them could have used the same file. Of course there must have been really good reasons for this decision. I hope no one will hold it against the maintainers of any of the projects. Especially because it looks like it is easy to move between the two lock files.
- sirodoht 1y ago
  Which functionality of uv's cannot be expressed in the pylock.toml format?
  - maleldil 1y ago
    More information here: https://github.com/astral-sh/uv/issues/12584 https://github.com/astral-sh/uv/issues/12584 > The biggest limitation is that there's no support for arbitrary entrypoints to the graph, because pylock.toml includes a fixed marker for each package entry rather than recording a graph of dependencies.
miiiiiike 1y ago
I've exited countless Python/Django threads discussing future plans. Year -1: The community has a problem. Year 0: Proposal to fix the problem. Year 1: A small but vocal subset of the Python/Django community pops up in every thread: "It's not actually a problem." or "It's not an issue that my project would ever encounter so limited resources shouldn't be expended on it." Year 2: People are choosing other solutions because Python/Django isn't addressing the problem. Year 3: We'll form a committee. The committee doesn't think it's a problem. Year 4: The community has a problem. Fine. Why doesn't the community write a Python Enhancement Proposal/Django Enhancement Proposal (PEP/DEP)? Years 5-10: PEP/DEP ignored. Year 11: The community has a problem. PEP/DEP implemented and released. Year 12-22: Major packages refuse to support the change. Year 23: Last version not supporting the change deprecated. Year 23+1 day: Fork of last deprecated version Python not supporting change created and released. I have 15 years of code in Python still running but spend a little more than 50% of my time in other stacks. I don't notice as many people arguing against basic features, like a REST API package in Django, in non-Python/Django communities. The precursor to a Django REST API package, content negotiation, has been a draft DEP since early 2014 (https://github.com/django/deps/blob/main/draft/content-negotiation.rst https://github.com/django/deps/blob/main/draft/content-negot...). That's going on 12 years of stalled progress for a feature that needed to be released in 2010. With Python/Django you learn not to wait because nothing is going to change. And yes, Python/Django are open source. And yes agin, I donate over $1,000/year to support F/OSS projects that I depend on.
- noitpmeder 1y ago
  [flagged]
  - miiiiiike 1y ago
    It's not a rant. It's a man with several burned fingers and a drifter beard muttering warnings from a corner. How was the Python 2 to Python 3 migration for you? How many Python libraries are you responsible for packaging? Has that been fun for the past 15-20 years? How heavily do you rely on async? Almost a decade for Django+major libraries to support Python 3. More than a decade for Django async support. I'm not ungrateful. And, I wish that I believed that throwing money at the problem would solve it. Python has a large and diverse user base. That brings unique challenges that I don't see in other stacks.
truelson 1y ago
So… what is a good example of a consensus driven culture on something popular with a lot of opinions, some legacy use cases, that can get these things done quickly? This is a systems problem. Successful examples wanted.
- miiiiiike 1y ago
  Angular turned things around quickly. Corporate sure, but if you know anything about how Angular is used within Google that was a massive job.
- gurjeet 1y ago
  Postgres community is one example to look at, that I can think of. Linux may be other, but I'm not intimately aware of its inner workings. https://www.postgresql.org/community/ https://www.postgresql.org/community/ is a good start to get a feel of all things related to Postgres community.
- mkoubaa 1y ago
  You need a consensus driven culture with a final decision maker when that culture fails to reach a decision.
  - testdelacc1 1y ago
    In the general case what you say is true. But look at the specific example of PEP 751 (the thread we’re in). Normally you’d designate some person who has to gain consensus and add the feature to python or to the standard library. Even if everyone isn’t on board, they’ll get on board when they upgrade python. PEP 751 isn’t a python feature, it’s a feature that will be implemented by 3 projects - PDM, pip and uv. Consensus isn’t optional or nice to have, it’s necessary. If any of the 3 maintainers felt their needs weren’t met they wouldn’t have implemented it. Some projects wait too long for consensus because they prioritise not rushing into a suboptimal solution or hurting people’s feelings. Sometimes it’s ok to just go ahead and implement something. Pep 751 is not one of those projects.
    - 1y ago
1aH27JHq 1y ago
[flagged]
eduction 1y ago
Why did we have to call them “lock files?” There is an existing thing known as a lock file for actual file locking. Call them literally anything else. Freeze file, version spec, dependency pin… There really are only two hard problems in computer science, as the saying goes. Cache invalidation and epithet manufacturing (cough).
- PaulHoule 1y ago
  Locks don't actually work in POSIX in real life anyway.
  - procaryote 1y ago
    doesn't opening a file with O_CREAT|O_EXCL work in posix?
    - wpollock 1y ago
      POSIX only guarantees advisory locks; mandatory locks are an optional feature and are not supported in Linux. See for example <https://stackoverflow.com/questions/77931997/linux-mandatory-locking-for-file-locking https://stackoverflow.com/questions/77931997/linux-mandatory...>.
      - procaryote 1y ago
        But that is file locking, not lockfiles... Creating a file with the exclusive creation mode is a pretty common way to make a lock file and as far as I know that works well in posix.
- arccy 1y ago
  we should have called the other thing mutex files
- zahlman 1y ago
  Python is following the precedent of many other "language ecosystems" here.
blindriver 1y ago
This is why having a benevolent dictator sometimes results in better progress than committees. It’s a double edged sword obviously if the dictator has limited skill but having someone like a Steve Jobs or Linus clears the way for progress when things like “consensus” causes decisions to take years or die from inertia. I’ve seen this first hand at FAANGs where bureaucracy kills great ideas because bureaucrats in key areas don’t want to lift a finger to make changes. The big counter example is c++ which I feel is too productive and should slow down their decisions by a factor of 3.
riedel 1y ago
Interesting to see the seemingly canonical meaning of lockfile (semaphore vs package version lock) change over the years. I at least was curious, how one could specify a format for typical empty files.

123 points by birdculture 1y ago

> A lock file is meant to record all the dependencies your code needs to work along with how to install those dependencies. It's about dependency locking in Python packaging.

klustregrif 1y ago

Overall it feels like UV is the best thing to happen to python packaging in two decades, by circumventing the endless non productive discussions on peps and instead just building something that works and is fast. In Rust naturally.

webology 1y ago
UV is great but also builds on existing PEPs. While they have the ability experiment (which is great), they also benefit from those "endless non productive discussions on peps" as you called it. I think UV proves that dedicated funding can make a huge impact on a project and benefit for the community. They are a doing a damn, good job.
- tormeh 1y ago
  They mostly took inspiration from other languages for UV. Cargo (Rust) was a huge inspiration, but they got stuff from Ruby as well, I believe. There was an episode on "the changelog" about it. Don't remember them saying anything about PEPs, although that might just be me not having listened to the entire thing. However, the Charlie Marsh was extremely insistent on the advantages of being a polyglot and hiring people with diverse programming experiences. So I think it's quite safe to assume that played a bigger role than just PEPs.
  - zahlman 1y ago
    > Don't remember them saying anything about PEPs, although that might just be me not having listened to the entire thing. However, the Charlie Marsh was extremely insistent on the advantages of being a polyglot and hiring people with diverse programming experiences. So I think it's quite safe to assume that played a bigger role than just PEPs. Not that they took inspiration from PEPs, but they sought to implement those standards (for interoperability) and have been active in the discussion of new packaging-related PEPs.
- the__alchemist 1y ago
  I can't speak for the UV team. My 2C on how I would treat the PEPs: If there is an accepted one, and implementing it doesn't go too strongly against your competing design goals, do it for compatibility. This does not imply that the PEP is driving your design, or required to make your software. It is a way to improve compatibility.
- zahlman 1y ago
  > dedicated funding can make a huge impact on a project Where does Astral's funding come from, anyway?
- nxpnsv 1y ago
  Rye was already pretty good before it was donated to astral and renamed to uv though...
  - the__alchemist 1y ago
    I wrote an earlier one (rust, inspired by Cargo, managed deps, scripts, and py versions) called PyFlow that I abandoned, because nobody used it. "Why should I use this when we have pip, pipenv, and poetry?"
    - aiiizzz 1y ago
      Programming is 80% marketing, eh?
    - zahlman 1y ago
      It probably doesn't help that there seem to be a few other projects out there with the same name or a similar name, which do completely different things.
smallmancontrov 1y ago
Yet more proof that "the best way to do anything in python is to not do it in python."
- paulddraper 1y ago
  Find something better than Django. Or matplotlib. Or PyTorch.
  - maleldil 1y ago
    PyTorch is a C++ project with a Python wrapper.
  - aeve890 1y ago
    >Find something better than Django Rails. QED.
    - paulddraper 1y ago
      Rails is a good project no doubt. Django comes batteries included for basic apps, including an admin.
      - aeve890 1y ago
        Counterpoint, you still need a 3rd party library to implement a sane rest API in Django. Same with async jobs.
        paulddraper 1y ago
        I would definitely not use Django to do REST API. It can work...but that's not what it was designed for.
  - deleted 1y ago
    [deleted]
- deleted 1y ago
  [deleted]
scrollaway 1y ago
Can someone explain why UV is so praised when Poetry achieved a lot of that good several years earlier? Maybe I missed the train but I’ve been using poetry since its first version and all the benefits people praise UV for have long been in my builds.
- mixmastamyk 1y ago
  Ruff was the gateway drug for me, much better than the black/isort/etc combo. Led me to try uv, which fixed a couple of egregious bugs in pip. Add speed and it’s a no brainer. I don’t think poetry has these advantages, and heard about bugs early on. Is that completely fair? Probably not. But it’s obvious astral tools have funding and a competent team.
- Game_Ender 1y ago
  Speed and simplicity. Now I can fetch one binary on a system and in seconds fetch everything needed to run a Python tool or work on a code base. I can do all that without having to even worry about virtual ends, or Python versions too.
- zahlman 1y ago
  Poetry was in some senses before its time; people were frustrated with pip, but didn't really understand the problems they were encountering, meanwhile the ecosystem had started trying to move away from doing everything with Setuptools for everyone. There's a ton that I'd love to explain here about the history of pyproject.toml etc. but the point is that Poetry had its own idea about what it would mean to be an all-in-one tool... and so did everyone else. Meanwhile, the main packaging people had been designing standards with the expectation of making a UNIX-style tool ecosystem work. Everyone seems to like uv's answer better, but I'm still a believer in composable toolchains, since I've already been using those forever. I actually was an early Poetry adopter for a few reasons. In particular, I would have been fine sticking with Setuptools for building projects if it had supported PEPS 517/518/621 promptly. 621 came later but Poetry's workaround was nicer than Setuptools' to me. And it was easier to use with the new pyproject.toml setup, and I really wanted to get away from the expectation of using setup.py even for pure-Python projects. But that was really it. The main selling point point of Poetry was (and is) that they offered a lockfile and dependency resolution, but these weren't really things I personally needed. So there was nothing really to outweigh the downsides: * The way Poetry does the actual installation is, as far as I can tell, not much different from what pip does. And there are a ton of problems with that model. * The early days of Poetry were very inconsistent in terms of installation and upgrade procedures. There was at least once that it seemed that the only thing that would work was a complete manual uninstall and reinstall, and I had to do research to figure out what I had to remove for the uninstallation as there was nothing provided to automate that. * In the end, Poetry didn't have PEP 621 support for about four years (https://github.com/python-poetry/roadmap/issues/3 https://github.com/python-poetry/roadmap/issues/3 ; the OP was already almost a year after PEP acceptance in https://discuss.python.org/t/_/5472/109 https://discuss.python.org/t/_/5472/109); there was this whole thing about how you were supposed to use pyproject.toml to describe the basic metadata of your project for packaging purposes, but if you used Poetry then you used Masonry to build, and that meant using a whole separate metadata configuration. Setuptools was slow in getting PEP 621 support off the ground (and really, PEP 621 itself was slow! It's hard to justify expecting anyone to edit pyproject.toml manually without PEP 621!), but Poetry was far slower still. I had already long given up on it at that point. So for me, Poetry was basically there to provide Masonry, and Masonry was still sub-par. I was still creating venvs manually, using `twine` to upload to PyPI etc. because that's just how I think. Writing something like `poetry shell` (or `uv run`) makes about as much sense to me as `git run-unit-tests` would.
frizlab 1y ago
Yeah yeah, in rust… You know rust is not the only language in which one can write good code, right?
- zx8080 1y ago
  Good point! What are the other peoper languages in addition to golang and maybe c++ for such tooling? However people... don't like (hate? fear?) c++ nowadays. And I doubt that golang fits well here. So what else? (I'm not pushing towards rust! It just seems rust got into some vacuum, however not fitting it fully like a square thing in a round hole.)
  - frizlab 1y ago
    That I’m sure of: Swift is extremely underrated and actually very close functionally to rust. I don’t know other languages, but I’ve heard a lot of good things about zig.
    - zx8080 1y ago
      > swift Not everyone would want to get into Apple vendor-locked language (formally not, but it's like saying Chromium is opensource and not related to Alphabet/Google) and its development environment. Especially after JetBrains closed its swift oriented ide [0]. I hate xcode, sorry, after developing for too many years in it. 0: https://www.jetbrains.com/objc/ https://www.jetbrains.com/objc/
      - frizlab 1y ago
        Swift has an open-source and fully supported first party LSP integration and VSCode plugin, as well as a very active and welcoming community (forums). Using Swift w/o Xcode is now 100% possible and very much recommended by some people when no Apple development-platform specifics are required (native iOS/macOS apps). The language was initially very much Apple-platforms oriented (had to be), but now that pretty much all the Apple stuff works well they moved beyond that. Finally where the language comes from does not impact whether you can write good code with it.

charcircuit 1y ago

The post didn't answer why it took over 4 years. Why couldn't everyone be flied to the same place and have it all figured out in a week instead of having the process drag on for years?

quotemstr 1y ago
An endless multiplication of veto points in a consensus culture is a failure mode. Funny thing is people embedded in such a culture will see its slowness as a feature and sneer at the world as it leaves them behind. Python was better with a BDFL. The best thing that could happen to Python right now would be for someone to fork it. Maybe just have Astral run away with the whole language. This lock file format should have taken a weekend, not four damn years.
- vinnymac 1y ago
  A governance model where “time-based decision thresholds” directly damaged veto power as time passes could be used to harness this sort of BDFL-only power in a consensus culture.
- zahlman 1y ago
  > Python was better with a BDFL. The problem is that the BDFL also didn't want to think about packaging, certainly not the issues that inspired Conda. I wouldn't mind seeing a PyPy-like fork done in Rust. Maybe take the opportunity to redesign the standard library, too.

pjmlp 1y ago

No worries, it is taking longer to be able to rely on C++ modules for portable code, having Valhala available on JVM, Android supporting anything beyond Java 17, or a JIT in CPython, some things take their time to finally become widespread, for various kinds of reasons.

deleted 1y ago

[deleted]

eqvinox 1y ago

I was hoping part of this delay was due to people arguing lock files are poor engineering to begin with, but alas, no mention of that. I guess we've just given up on any kind of package version flexibility.

lexicality 1y ago
> I guess we've just given up on any kind of package version flexibility. Presumably because decades of experience has demonstrated that humans are extremely bad at maintaining compatibility between releases and dealing with fallout from badly specified package versions is probably second only to NULL in terms of engineering time wasted? Or possibly it's just because a lot of the Python ecosystem doesn't even try and follow semver and you have no guarantee that any two versions are compatible with each other without checking the changelog and sacrificing a small chicken...
- BugsJustFindMe 1y ago
  > Or possibly it's just because a lot of the Python ecosystem doesn't even try and follow semver Even if they try, semver can only ever be a suggestion of the possibility of compatibility at best because people are not oracles and they misjudge the effects of changes all the time.
  - 9dev 1y ago
    The PHP ecosystem is almost universally semver and has been going strong for years now, without any major accidental breaking change related outages. A little discipline and commitment to backwards compatibility and it isn’t too hard, really?
    - withinboredom 1y ago
      PHP compatibility and its commitment (across the ecosystem) to backwards compatibility is actually pretty cool. If there is one thing PHP does right, it’s this.
  - 1y ago

huflungdung 1y ago

[dead]

JimDabell 1y ago

Unfortunately: > Some of uv's functionality cannot be expressed in the pylock.toml format; as such, uv will continue to use the uv.lock format within the project interface. > However, uv supports pylock.toml as an export target and in the uv pip CLI. — https://docs.astral.sh/uv/concepts/projects/layout/#the-lockfile https://docs.astral.sh/uv/concepts/projects/layout/#the-lock...

testdelacc1 1y ago
That is pretty unfortunate. Would have been cool if all of them could have used the same file. Of course there must have been really good reasons for this decision. I hope no one will hold it against the maintainers of any of the projects. Especially because it looks like it is easy to move between the two lock files.
sirodoht 1y ago
Which functionality of uv's cannot be expressed in the pylock.toml format?
- maleldil 1y ago
  More information here: https://github.com/astral-sh/uv/issues/12584 https://github.com/astral-sh/uv/issues/12584 > The biggest limitation is that there's no support for arbitrary entrypoints to the graph, because pylock.toml includes a fixed marker for each package entry rather than recording a graph of dependencies.

miiiiiike 1y ago

I've exited countless Python/Django threads discussing future plans. Year -1: The community has a problem. Year 0: Proposal to fix the problem. Year 1: A small but vocal subset of the Python/Django community pops up in every thread: "It's not actually a problem." or "It's not an issue that my project would ever encounter so limited resources shouldn't be expended on it." Year 2: People are choosing other solutions because Python/Django isn't addressing the problem. Year 3: We'll form a committee. The committee doesn't think it's a problem. Year 4: The community has a problem. Fine. Why doesn't the community write a Python Enhancement Proposal/Django Enhancement Proposal (PEP/DEP)? Years 5-10: PEP/DEP ignored. Year 11: The community has a problem. PEP/DEP implemented and released. Year 12-22: Major packages refuse to support the change. Year 23: Last version not supporting the change deprecated. Year 23+1 day: Fork of last deprecated version Python not supporting change created and released. I have 15 years of code in Python still running but spend a little more than 50% of my time in other stacks. I don't notice as many people arguing against basic features, like a REST API package in Django, in non-Python/Django communities. The precursor to a Django REST API package, content negotiation, has been a draft DEP since early 2014 (https://github.com/django/deps/blob/main/draft/content-negotiation.rst https://github.com/django/deps/blob/main/draft/content-negot...). That's going on 12 years of stalled progress for a feature that needed to be released in 2010. With Python/Django you learn not to wait because nothing is going to change. And yes, Python/Django are open source. And yes agin, I donate over $1,000/year to support F/OSS projects that I depend on.

noitpmeder 1y ago
[flagged]
- miiiiiike 1y ago
  It's not a rant. It's a man with several burned fingers and a drifter beard muttering warnings from a corner. How was the Python 2 to Python 3 migration for you? How many Python libraries are you responsible for packaging? Has that been fun for the past 15-20 years? How heavily do you rely on async? Almost a decade for Django+major libraries to support Python 3. More than a decade for Django async support. I'm not ungrateful. And, I wish that I believed that throwing money at the problem would solve it. Python has a large and diverse user base. That brings unique challenges that I don't see in other stacks.

truelson 1y ago

So… what is a good example of a consensus driven culture on something popular with a lot of opinions, some legacy use cases, that can get these things done quickly? This is a systems problem. Successful examples wanted.

miiiiiike 1y ago
Angular turned things around quickly. Corporate sure, but if you know anything about how Angular is used within Google that was a massive job.
gurjeet 1y ago
Postgres community is one example to look at, that I can think of. Linux may be other, but I'm not intimately aware of its inner workings. https://www.postgresql.org/community/ https://www.postgresql.org/community/ is a good start to get a feel of all things related to Postgres community.
mkoubaa 1y ago
You need a consensus driven culture with a final decision maker when that culture fails to reach a decision.
- testdelacc1 1y ago
  In the general case what you say is true. But look at the specific example of PEP 751 (the thread we’re in). Normally you’d designate some person who has to gain consensus and add the feature to python or to the standard library. Even if everyone isn’t on board, they’ll get on board when they upgrade python. PEP 751 isn’t a python feature, it’s a feature that will be implemented by 3 projects - PDM, pip and uv. Consensus isn’t optional or nice to have, it’s necessary. If any of the 3 maintainers felt their needs weren’t met they wouldn’t have implemented it. Some projects wait too long for consensus because they prioritise not rushing into a suboptimal solution or hurting people’s feelings. Sometimes it’s ok to just go ahead and implement something. Pep 751 is not one of those projects.
  - 1y ago

1aH27JHq 1y ago

[flagged]

eduction 1y ago

Why did we have to call them “lock files?” There is an existing thing known as a lock file for actual file locking. Call them literally anything else. Freeze file, version spec, dependency pin… There really are only two hard problems in computer science, as the saying goes. Cache invalidation and epithet manufacturing (cough).

PaulHoule 1y ago
Locks don't actually work in POSIX in real life anyway.
- procaryote 1y ago
  doesn't opening a file with O_CREAT|O_EXCL work in posix?
  - wpollock 1y ago
    POSIX only guarantees advisory locks; mandatory locks are an optional feature and are not supported in Linux. See for example <https://stackoverflow.com/questions/77931997/linux-mandatory-locking-for-file-locking https://stackoverflow.com/questions/77931997/linux-mandatory...>.
    - procaryote 1y ago
      But that is file locking, not lockfiles... Creating a file with the exclusive creation mode is a pretty common way to make a lock file and as far as I know that works well in posix.
arccy 1y ago
we should have called the other thing mutex files
zahlman 1y ago
Python is following the precedent of many other "language ecosystems" here.

blindriver 1y ago

This is why having a benevolent dictator sometimes results in better progress than committees. It’s a double edged sword obviously if the dictator has limited skill but having someone like a Steve Jobs or Linus clears the way for progress when things like “consensus” causes decisions to take years or die from inertia. I’ve seen this first hand at FAANGs where bureaucracy kills great ideas because bureaucrats in key areas don’t want to lift a finger to make changes. The big counter example is c++ which I feel is too productive and should slow down their decisions by a factor of 3.

riedel 1y ago

Interesting to see the seemingly canonical meaning of lockfile (semaphore vs package version lock) change over the years. I at least was curious, how one could specify a format for typical empty files.