8 ms·

Multiversion Python Thoughts

89 points by divbzero 2y ago

andrewchambers 2y ago
I feel like most of these problems just disappear if people would follow the same naming scheme sqlite3 uses. Just put the major version in the name and most tools just work out of the box with multiple versions.
- greener_grass 2y ago
  This is Rich Hickey's suggestion too https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk
- the_mitsuhiko 2y ago
  That’s why I called the library “jinja2”. It was a new major version. But people over the years really did not like it and it did not catch on much.
  - smitty1e 2y ago
    Once more, the wisdom of "Explicit is better than implicit" shines. Instead, we jump through hoops with our hair on fire to manage complexity. People.
  - TZubiri 2y ago
    Agree. The issue was breaking bc by releasing a new major under the same name.
  - rbanffy 2y ago
    Really, people should just update their software to use the newer libraries and fix whatever breaks. If you want to use functions from version 2, you should port the rest of the code to version 2.
- bobnamob 2y ago
  Of course coming with the _major_ caveat that the interface and behaviour[1] of the dependency is absolutely stable for the entirety of the "major version". I'm an advocate for this style of library/dependency development, unfortunately in my experience the average dependency doesn't have the discipline to pull it off. [1] https://www.hyrumslaw.com https://www.hyrumslaw.com
  - zo1 2y ago
    I don't want to "pull this off", nor do I want to expend the time/energy to do so. We're not Microsoft with effectively infinite budgets, nor are we Elasticsearch/Grafana Orgs with metric oodles of developer-hours, and 50k+ github star mindshares-worth of evangelists behind us to document and tutorialize every tiny little feature in a cookbook or doc website. Code changes, and it's kinda silly to expect interfaces to be locked in place as that'll stifle development for even small-ish features. Does that mean every minor version or commit will change fundamental or large parts of the codebase? Probably not, but it's a sliding scale and people seriously need to find something better to do than writing Yet Another Python Package Manager. I use the term "we" loosely here ofc in the context of this mini-rant.
    - andrewchambers 2y ago
      If you are breaking the api it isn't that hard to add a new one instead of butchering the old one. If it really can't be done then you aren't really shipping a library that is meant to be depended on.
- benrutter 2y ago
  I think this works ok if your library is something like Django or Pandas that people are building their project around. But it makes things exponentially more complex for libraries like pyarrow or fsspec that are normally subdependencies. Imagine trying to do things like import pyarrow14, then if that failed try pyarrow13, etc. Additionally, python doesn't have a good way of saying "I need one of the following different libraries as a dependency"
  - andrewchambers 2y ago
    I like the way python handled this situation - python3 for when you care, python for when you don't.
- dist-epoch 2y ago
  sqlite3 was released 20 years ago, in 2004. I'm not convinced all code written in 2004 which used sqlite3 would still work today against the latest version.
  - aragilar 2y ago
    If stuff was removed, then I would have expected them to bump the ABI? As they haven't done that, I would actually assume it would work absent evidence to the contrary?
thingsgg 2y ago
It's one thing to see if something like this is possible from a technical standpoint, but whether this is desirable for the ecosystem as a whole is a different question. I would argue that allowing multiple versions of packages in the dependency tree is bad. It removes incentives for maintainers to adhere to sane versioning standards like semver, and also the incentive to keep dependencies updated, because resolution will never be impossible for package users. Instead, they will get subtle bugs due to unknowingly relying on multiple versions of the same package in their application that are incompatible with each other. For lack of a better word, the single package version forces the ecosystem to keep "moving": if you want your package to be continued to be used, you better make sure it works with reasonably recent other packages from the ecosystem.
- the_mitsuhiko 2y ago
  > It removes incentives for maintainers to adhere to sane versioning standards like semver Semvers does not matter in this way. The issue with having a singular resolution — semver or not — is that you can only move your entire dependency tree at once. If you have a very core library then you are locked in unless you can move the entire ecosystem up which is incredibly hard.
- greatgib 2y ago
  Indeed, in my opinion it is the best way to finish in a cluster mess like the nodejs/npm ecosystem... And a very real issue is that young developers don't know anymore how to develop by limiting dependencies to the strict minimum. You have some projects with hundreds of dependencies without a real reason than lazyness or always using the new shiny thing.
  - jollyllama 2y ago
    Indeed, I blame npm for normalizing this kind of thing. It's no surprise that frontend devs wouldn't understand why it's bad, but Python devs should know better.
physicsguy 2y ago
Does this not fall over in the circumstance of linking against a C library (not specifically a Python extension) as many Python libraries do? For e.g. I write “library” of which v1 depends on somelib.so.1.0.0 and v2 depends on somelib.so.2.0.0 If somelib has some symbols clashing in the names this can cause real problems!
- the_mitsuhiko 2y ago
  They don’t show up in the global symbol namespace usually so it’s fine. It’s only an issue for some libraries that load globally so that one library can reference another c library.
- snatchpiesinger 2y ago
  Python dlopens binary wheels with RTLD_LOCAL on Linux, and I assume it does the equivalent on Windows. There were issues relatively recently with -ffast-math binary wheels in Python packages, as some versions of gcc generates a global constructor with that option that messes with the floating point environment, affecting the whole process regardless of symbol namespaces. It's mostly just an insanity of this option and gcc behavior though.
  - dist-epoch 2y ago
    Windows has a different design where the symbols are not merged, the .dll name is part of the resolution, so you can have 2 different .dll export the same symbol.
orf 2y ago
This comes up every now and again, and there are two fairly simple examples that I think show the complexities: One: Library A takes a callback function, and catches “request.HttpError” when invoking that callback. The callback throws an exception from a differing version of “request”, which is missing an attribute that the exception handler requires. What happens? How? Two: Library A has a function that returns a “request.Response” object. Library B has a function that accepts a “request.Response” object, and performs “isinstance”/type equality on the object. Library A and library B have differing and incompatible dependencies on “request”. What version of the request object is sent to library B from library A, and how does “isinstance”/“type” interact with it? Both of these resolve around class identities. In Python they are intrinsically linked to the defining module. Either you break this invariant and have two incompatible/different types have the same identity and introduce all kinds of bugs, or you don’t and also introduce all kinds of bugs - “yes, this is a request.Response object, but this method doesn’t exist on this request.Response object”, or “yes this is someone’s request.Response object, but it’s not your request.Response object” Getting different module imports to succeed is more than possible, getting them to work together is another thing entirely. One solution to this is the concept of visibility, which in Python is famously “not really a thing”. It’s safe to use incompatible versions of a library as long as the types are not visible - I.e no method returns a request.Response object, so the module is essentially a completely private implementation detail. This is how Rust handles this, I think. However is obviously fucked by exceptions, so it seems pretty intractable.
- the_mitsuhiko 2y ago
  That is not any different in Python than it is in Rust, Go or JavaScript. Yes: it's a problem if those different dependency versions shine through via public APIs. However there are plenty of cases where the dependency you are using in a specific version is isolated within your own internal API. I think if Python were to want to go down this path it should be isolated to explicit migration cases for specific libraries that want to opt themselves into multi-version resolution. I think it would enable the move of pretty core libraries in the ecosystem in backwards incompatible ways in a much smoother way than it is today.
  - 2y ago
TZubiri 2y ago
Pip, Venv, poetry, pipenv, now uv If you are still struggling with this in 2024, you are missing the actual challenges of the world.
- simiones 2y ago
  So the fact that there are 5 different tools that (attempt to?) fix this problem is a sign that it is not a problem, from your point of view?
  - TZubiri 2y ago
    It's a sign developers get stuck in a paper bags. Same thing with text editors, orms, frameworks. Imagine 2050, flying cars, talking robots, teleportation, and John Developer is going to be releasing solution 57 to a problem that was solved in the 90s
    - dist-epoch 2y ago
      What is the 90's solution to this problem - autoconf, apt, yum, oops, here we go again with multiple standards.
- dikei 2y ago
  Just like JS has npm, yarn, pnpm, bun... It seems the more users a language has; the more dev tools get written for it.
  - TZubiri 2y ago
    Maybe if you don't import stuff like left-pad and actually write some code, you wouldn't have to write a dissertation on package management.
    - dikei 2y ago
      Actually, I prefer spending my time writing right-pad
dikei 2y ago
One way is to publish 2 variants of your packages: one with the major version number appended to package name and one without the version number. Users who need to install multiple versions can use the first variant, while user that just want to follow the latest can use the second.
CraigJPerry 2y ago
Should this be facilitated? Should this work really be done? I’m thinking not. If I have a special case and I need to do this today, I’m not blocked from doing so (I’ll vendor one of the dependencies and change the name) - certainly a pain to do, especially if I have to go change imports in a c module as part of the dep but achievable and not a blocker for a source available dependency. However, if this becomes easily possible, well why shouldn’t I use it? The net result is MORE complexity in python packaging. More overheads for infra tools to accomadate.
- baq 2y ago
  You’ve never found yourself in a dependency resolution situation where there are no solutions to requirements. You need multiple versions of the same package in such cases. The alternative is just cheating: ignore requirements, install packages and hope for the best. Alas hope is not a process.
  - dikei 2y ago
    The usual way is to put conflicting versions in optional-dependencies, and then build one target for each conflicting set of deps. That'd work fine if the code path of one target doesn't touch the others, which is often the case. You'd obviously need to have tests for both targets, possibly using a flexible test runner like `nox` to setup separate test env for each target.
  - theelous3 2y ago
    I've never found myself in this situation where it can't be solved by being throughtful and taking the time to improve my code.
    - leetrout 2y ago
      These issues crop up in dependencies more than your code. Then you have to vendor one of the deps and edit it (and hopefully ship that back upstream and hope the maintainer will merge it).
  - 2y ago
pid-1 2y ago
I have done the following in the past: 1. pip install libfoo==1.x.x 2. pip install libfoo==2.x.x --target ~/libs/libfoo_v2 # vendor libfoo v2 3. import sys import libfoo original_sys_path = sys.path.copy() sys.path.insert(0, '~/libs/libfoo_v2') import libfoo as libfoo_v2 sys.path = original_sys_path There are caveats of course. But works for simple cases.
rbanffy 2y ago
One of the reasons I made pip-chill is to create an incentive to not bother with version numbers and just make your software always run against the latest versions. If something breaks, fix it. If it's too hard to do it, maybe you depend on too many things. Leftpad feelings here. Having your software depend on two different versions of a library is just asking for more pain. BTW, I still need to fix it to run on 3.12+ in a neat way. For now, it runs, but I don't like it.
- theelous3 2y ago
  This is a fun idea. I always do this currently with some non-core dependencies like black, linters, or dependencies I controll the versioning of. I can't imagine I'd use this on libraries that provide real functionality in prod code though. If you have an incident popping off and you find out your dependency resolution goblin has reared its little head at the same time, bad day ahead.
  - rbanffy 2y ago
    You can always replicate the precise configuration running on the server. I assume the software passed all tests before being deployed with those same exact versions.
- deleted 2y ago
  [deleted]
benavn 2y ago
It is kind of the same problem as for shared libraries. In the GNU universe the cleanest solution is to have multiple soversions. Transferred to jinja it would be jinja.so.1 and jinja.so.2. Maintaining fine grained symbol versioning is a pain and a massive amount of work for the package maintainer: https://invisible-island.net/ncurses/ncurses-mapsyms.html https://invisible-island.net/ncurses/ncurses-mapsyms.html Honestly, multiple installed versions like jinja1 and jinja2 sounds best to me.
aragilar 2y ago
What is the difference between this and and the approach taken by eggs/buildout (other than the time difference, and so some changes in APIs)? My impression (having never used buildout) was handling multiple versions made debugging hard, and there were lots of random issues unless you did everything correctly (and that setuptools and its variants switched away from the options to use multi-versioning because it was effectively a foot-gun)?
i-use-nixos-btw 2y ago
This would introduce more problems than it solves 99% of the time. The 1% of the time, it could be very handy. I haven’t used UV, but it says that it manages python as well as packages - I’m guessing like conda, python-venv, and of course nix does. If the C api is an issue, it sounds like you have control over it if you need it. You manage the python distribution, so could it be patched? This way it feels like you’d be able to establish not just what is being imported, but what is importing it - then redirect through a package router and grab the one you want. This may be particularly useful if you’re loading in shared libraries, because that is already a dumpster fire in python, and I imagine loading in different versions of the same thing would be quite awkward as-is.
xrd 2y ago
Another person here mentioned a comment "More overheads for infra tools to accomadate." I agree. I am using python as my main language these days, coming from JS and C++ and a bit of rust. The biggest problem I face is that the tools (editors, mainly) don't support the basic packaging tools. I use venv for everything, but when I try to use something else, I almost always get bitten. For example, asdf. I tried to use this tool. So awesome! It works great from the command line. But, when I try to use zed, it cannot figure out what to do, and I cannot find references in the zed github repository on the right way to setup pyproject.toml. And, emacs. Will uv work within emacs? Each of these packaging tools (and I'm thinking about the long history of nvm (node version manager), brew, and everything else) makes different assumptions about the right way to modify the path variable, or create aliases, or use shims (the definition of which varies with each tool) and I'm sure I'm missing other details. Does uv do the right thing mostly? I will say my experiences with python and the tooling has been more frustrating than the tools for JS. I use pnpm and it just works, and I understand the benefits. And, I can survive with just npm and yarn. But, to me, it is saying a lot that the python tooling feels more broken than JS. I mean, I lived through the webpack years, and I'm still using JS and have a generally favorable opinion of it.
- codethief 2y ago
  Could you elaborate on the issues you've been seeing in Zed? I've been using asdf as a version manager for Python for quite some time and haven't really had any issues.
  - xrd 2y ago
    You are using asdf and zed successfully? I'm so glad to hear this. I removed asdf because I could not get it to recognize the pyproject.toml file. This is working in harmony for you? With Zed and a venv (from python -m venv .venv for example), Zed properly recognized installed packages and provided type hinting and docs, but when I switched to asdf it did not seem to work. But, I was new to asdf and perhaps was using it incorrectly. I was always assuming that when I'm in the command line, running asdf to use the right python works because the path is correctly established. But, when I run zed, it launches without the path setup step, and things went badly. I'm just speculating, but I could not get type hinting and didn't know how to fix it.
deleted 2y ago
[deleted]
eqvinox 2y ago
The absence of any references to how other ecosystems do this (e.g. oldest widely in use: ELF soversion) strikes me as massive oversight? There's 40-50 years of history of people trying to do close cousins of this.
tpoacher 2y ago
Am I the only person here who thinks the easiest thing to do is to download the python version you need, compile, install in your preferred prefix like any other program and then run it?
- zahlman 2y ago
  I think that's quite easy, sure. But it's also irrelevant to the discussion. The issue the article is talking about (despite the title) isn't about managing multiple versions of Python. It's with trying to have multiple versions of a third-party library, within a single Python environment.
  - tpoacher 2y ago
    Ah you're right, thanks. Admittedly I only skimmed and relied on assumptions of the usual discussion around complaints on the python packaging ecosystem xD Though in principle, my preferred approach here would be similar. Manually install in a particular prefix, add it to the path (manually at runtime or programmatically via the sys module), and then import multiple versions under different namespaces...
    - zahlman 2y ago
      This approach was discussed elsewhere in the thread and you may want to be aware of its limitations (https://news.ycombinator.com/item?id=41706009 https://news.ycombinator.com/item?id=41706009).
      - tpoacher 2y ago
        Thanks! I won't start the same thread here (though thank you for pointing that one out), but when I've come across scenarios similar to those in that thread in my own projects (which, admittedly, while not trivial, were probably not enterprise-scale), the solution generally still involved making sure that one ensured the paths / global variables were suitably modified in the relevant modules before the relevant calls, to ensure you're using the correct namespaces expected by the recipient. Which may be tedious, but to me is not absurd; you're just sticking to the contract expected by the recipient. The bigger issue here (for me) is probably whether those contracts are visible/explicit or not, and how/whether the contract is enforced in the recipient library ... but I would hesitate to call this a multiversion / dependency problem.

89 points by divbzero 2y ago

I feel like most of these problems just disappear if people would follow the same naming scheme sqlite3 uses. Just put the major version in the name and most tools just work out of the box with multiple versions.

greener_grass 2y ago
This is Rich Hickey's suggestion too https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk
the_mitsuhiko 2y ago
That’s why I called the library “jinja2”. It was a new major version. But people over the years really did not like it and it did not catch on much.
- smitty1e 2y ago
  Once more, the wisdom of "Explicit is better than implicit" shines. Instead, we jump through hoops with our hair on fire to manage complexity. People.
- TZubiri 2y ago
  Agree. The issue was breaking bc by releasing a new major under the same name.
- rbanffy 2y ago
  Really, people should just update their software to use the newer libraries and fix whatever breaks. If you want to use functions from version 2, you should port the rest of the code to version 2.
bobnamob 2y ago
Of course coming with the _major_ caveat that the interface and behaviour[1] of the dependency is absolutely stable for the entirety of the "major version". I'm an advocate for this style of library/dependency development, unfortunately in my experience the average dependency doesn't have the discipline to pull it off. [1] https://www.hyrumslaw.com https://www.hyrumslaw.com
- zo1 2y ago
  I don't want to "pull this off", nor do I want to expend the time/energy to do so. We're not Microsoft with effectively infinite budgets, nor are we Elasticsearch/Grafana Orgs with metric oodles of developer-hours, and 50k+ github star mindshares-worth of evangelists behind us to document and tutorialize every tiny little feature in a cookbook or doc website. Code changes, and it's kinda silly to expect interfaces to be locked in place as that'll stifle development for even small-ish features. Does that mean every minor version or commit will change fundamental or large parts of the codebase? Probably not, but it's a sliding scale and people seriously need to find something better to do than writing Yet Another Python Package Manager. I use the term "we" loosely here ofc in the context of this mini-rant.
  - andrewchambers 2y ago
    If you are breaking the api it isn't that hard to add a new one instead of butchering the old one. If it really can't be done then you aren't really shipping a library that is meant to be depended on.
benrutter 2y ago
I think this works ok if your library is something like Django or Pandas that people are building their project around. But it makes things exponentially more complex for libraries like pyarrow or fsspec that are normally subdependencies. Imagine trying to do things like import pyarrow14, then if that failed try pyarrow13, etc. Additionally, python doesn't have a good way of saying "I need one of the following different libraries as a dependency"
- andrewchambers 2y ago
  I like the way python handled this situation - python3 for when you care, python for when you don't.
dist-epoch 2y ago
sqlite3 was released 20 years ago, in 2004. I'm not convinced all code written in 2004 which used sqlite3 would still work today against the latest version.
- aragilar 2y ago
  If stuff was removed, then I would have expected them to bump the ABI? As they haven't done that, I would actually assume it would work absent evidence to the contrary?

thingsgg 2y ago

It's one thing to see if something like this is possible from a technical standpoint, but whether this is desirable for the ecosystem as a whole is a different question. I would argue that allowing multiple versions of packages in the dependency tree is bad. It removes incentives for maintainers to adhere to sane versioning standards like semver, and also the incentive to keep dependencies updated, because resolution will never be impossible for package users. Instead, they will get subtle bugs due to unknowingly relying on multiple versions of the same package in their application that are incompatible with each other. For lack of a better word, the single package version forces the ecosystem to keep "moving": if you want your package to be continued to be used, you better make sure it works with reasonably recent other packages from the ecosystem.

the_mitsuhiko 2y ago
> It removes incentives for maintainers to adhere to sane versioning standards like semver Semvers does not matter in this way. The issue with having a singular resolution — semver or not — is that you can only move your entire dependency tree at once. If you have a very core library then you are locked in unless you can move the entire ecosystem up which is incredibly hard.
greatgib 2y ago
Indeed, in my opinion it is the best way to finish in a cluster mess like the nodejs/npm ecosystem... And a very real issue is that young developers don't know anymore how to develop by limiting dependencies to the strict minimum. You have some projects with hundreds of dependencies without a real reason than lazyness or always using the new shiny thing.
- jollyllama 2y ago
  Indeed, I blame npm for normalizing this kind of thing. It's no surprise that frontend devs wouldn't understand why it's bad, but Python devs should know better.

physicsguy 2y ago

Does this not fall over in the circumstance of linking against a C library (not specifically a Python extension) as many Python libraries do? For e.g. I write “library” of which v1 depends on somelib.so.1.0.0 and v2 depends on somelib.so.2.0.0 If somelib has some symbols clashing in the names this can cause real problems!

the_mitsuhiko 2y ago
They don’t show up in the global symbol namespace usually so it’s fine. It’s only an issue for some libraries that load globally so that one library can reference another c library.
snatchpiesinger 2y ago
Python dlopens binary wheels with RTLD_LOCAL on Linux, and I assume it does the equivalent on Windows. There were issues relatively recently with -ffast-math binary wheels in Python packages, as some versions of gcc generates a global constructor with that option that messes with the floating point environment, affecting the whole process regardless of symbol namespaces. It's mostly just an insanity of this option and gcc behavior though.
- dist-epoch 2y ago
  Windows has a different design where the symbols are not merged, the .dll name is part of the resolution, so you can have 2 different .dll export the same symbol.

orf 2y ago

This comes up every now and again, and there are two fairly simple examples that I think show the complexities: One: Library A takes a callback function, and catches “request.HttpError” when invoking that callback. The callback throws an exception from a differing version of “request”, which is missing an attribute that the exception handler requires. What happens? How? Two: Library A has a function that returns a “request.Response” object. Library B has a function that accepts a “request.Response” object, and performs “isinstance”/type equality on the object. Library A and library B have differing and incompatible dependencies on “request”. What version of the request object is sent to library B from library A, and how does “isinstance”/“type” interact with it? Both of these resolve around class identities. In Python they are intrinsically linked to the defining module. Either you break this invariant and have two incompatible/different types have the same identity and introduce all kinds of bugs, or you don’t and also introduce all kinds of bugs - “yes, this is a request.Response object, but this method doesn’t exist on this request.Response object”, or “yes this is someone’s request.Response object, but it’s not your request.Response object” Getting different module imports to succeed is more than possible, getting them to work together is another thing entirely. One solution to this is the concept of visibility, which in Python is famously “not really a thing”. It’s safe to use incompatible versions of a library as long as the types are not visible - I.e no method returns a request.Response object, so the module is essentially a completely private implementation detail. This is how Rust handles this, I think. However is obviously fucked by exceptions, so it seems pretty intractable.

the_mitsuhiko 2y ago
That is not any different in Python than it is in Rust, Go or JavaScript. Yes: it's a problem if those different dependency versions shine through via public APIs. However there are plenty of cases where the dependency you are using in a specific version is isolated within your own internal API. I think if Python were to want to go down this path it should be isolated to explicit migration cases for specific libraries that want to opt themselves into multi-version resolution. I think it would enable the move of pretty core libraries in the ecosystem in backwards incompatible ways in a much smoother way than it is today.
- 2y ago

TZubiri 2y ago

Pip, Venv, poetry, pipenv, now uv If you are still struggling with this in 2024, you are missing the actual challenges of the world.

simiones 2y ago
So the fact that there are 5 different tools that (attempt to?) fix this problem is a sign that it is not a problem, from your point of view?
- TZubiri 2y ago
  It's a sign developers get stuck in a paper bags. Same thing with text editors, orms, frameworks. Imagine 2050, flying cars, talking robots, teleportation, and John Developer is going to be releasing solution 57 to a problem that was solved in the 90s
  - dist-epoch 2y ago
    What is the 90's solution to this problem - autoconf, apt, yum, oops, here we go again with multiple standards.
dikei 2y ago
Just like JS has npm, yarn, pnpm, bun... It seems the more users a language has; the more dev tools get written for it.
- TZubiri 2y ago
  Maybe if you don't import stuff like left-pad and actually write some code, you wouldn't have to write a dissertation on package management.
  - dikei 2y ago
    Actually, I prefer spending my time writing right-pad

dikei 2y ago

One way is to publish 2 variants of your packages: one with the major version number appended to package name and one without the version number. Users who need to install multiple versions can use the first variant, while user that just want to follow the latest can use the second.

CraigJPerry 2y ago

Should this be facilitated? Should this work really be done? I’m thinking not. If I have a special case and I need to do this today, I’m not blocked from doing so (I’ll vendor one of the dependencies and change the name) - certainly a pain to do, especially if I have to go change imports in a c module as part of the dep but achievable and not a blocker for a source available dependency. However, if this becomes easily possible, well why shouldn’t I use it? The net result is MORE complexity in python packaging. More overheads for infra tools to accomadate.

baq 2y ago
You’ve never found yourself in a dependency resolution situation where there are no solutions to requirements. You need multiple versions of the same package in such cases. The alternative is just cheating: ignore requirements, install packages and hope for the best. Alas hope is not a process.
- dikei 2y ago
  The usual way is to put conflicting versions in optional-dependencies, and then build one target for each conflicting set of deps. That'd work fine if the code path of one target doesn't touch the others, which is often the case. You'd obviously need to have tests for both targets, possibly using a flexible test runner like `nox` to setup separate test env for each target.
- theelous3 2y ago
  I've never found myself in this situation where it can't be solved by being throughtful and taking the time to improve my code.
  - leetrout 2y ago
    These issues crop up in dependencies more than your code. Then you have to vendor one of the deps and edit it (and hopefully ship that back upstream and hope the maintainer will merge it).
- 2y ago

pid-1 2y ago

I have done the following in the past: 1. pip install libfoo==1.x.x 2. pip install libfoo==2.x.x --target ~/libs/libfoo_v2 # vendor libfoo v2 3. import sys import libfoo original_sys_path = sys.path.copy() sys.path.insert(0, '~/libs/libfoo_v2') import libfoo as libfoo_v2 sys.path = original_sys_path There are caveats of course. But works for simple cases.

rbanffy 2y ago

One of the reasons I made pip-chill is to create an incentive to not bother with version numbers and just make your software always run against the latest versions. If something breaks, fix it. If it's too hard to do it, maybe you depend on too many things. Leftpad feelings here. Having your software depend on two different versions of a library is just asking for more pain. BTW, I still need to fix it to run on 3.12+ in a neat way. For now, it runs, but I don't like it.

theelous3 2y ago
This is a fun idea. I always do this currently with some non-core dependencies like black, linters, or dependencies I controll the versioning of. I can't imagine I'd use this on libraries that provide real functionality in prod code though. If you have an incident popping off and you find out your dependency resolution goblin has reared its little head at the same time, bad day ahead.
- rbanffy 2y ago
  You can always replicate the precise configuration running on the server. I assume the software passed all tests before being deployed with those same exact versions.
deleted 2y ago
[deleted]

benavn 2y ago

It is kind of the same problem as for shared libraries. In the GNU universe the cleanest solution is to have multiple soversions. Transferred to jinja it would be jinja.so.1 and jinja.so.2. Maintaining fine grained symbol versioning is a pain and a massive amount of work for the package maintainer: https://invisible-island.net/ncurses/ncurses-mapsyms.html https://invisible-island.net/ncurses/ncurses-mapsyms.html Honestly, multiple installed versions like jinja1 and jinja2 sounds best to me.

aragilar 2y ago

What is the difference between this and and the approach taken by eggs/buildout (other than the time difference, and so some changes in APIs)? My impression (having never used buildout) was handling multiple versions made debugging hard, and there were lots of random issues unless you did everything correctly (and that setuptools and its variants switched away from the options to use multi-versioning because it was effectively a foot-gun)?

i-use-nixos-btw 2y ago

This would introduce more problems than it solves 99% of the time. The 1% of the time, it could be very handy. I haven’t used UV, but it says that it manages python as well as packages - I’m guessing like conda, python-venv, and of course nix does. If the C api is an issue, it sounds like you have control over it if you need it. You manage the python distribution, so could it be patched? This way it feels like you’d be able to establish not just what is being imported, but what is importing it - then redirect through a package router and grab the one you want. This may be particularly useful if you’re loading in shared libraries, because that is already a dumpster fire in python, and I imagine loading in different versions of the same thing would be quite awkward as-is.

xrd 2y ago

Another person here mentioned a comment "More overheads for infra tools to accomadate." I agree. I am using python as my main language these days, coming from JS and C++ and a bit of rust. The biggest problem I face is that the tools (editors, mainly) don't support the basic packaging tools. I use venv for everything, but when I try to use something else, I almost always get bitten. For example, asdf. I tried to use this tool. So awesome! It works great from the command line. But, when I try to use zed, it cannot figure out what to do, and I cannot find references in the zed github repository on the right way to setup pyproject.toml. And, emacs. Will uv work within emacs? Each of these packaging tools (and I'm thinking about the long history of nvm (node version manager), brew, and everything else) makes different assumptions about the right way to modify the path variable, or create aliases, or use shims (the definition of which varies with each tool) and I'm sure I'm missing other details. Does uv do the right thing mostly? I will say my experiences with python and the tooling has been more frustrating than the tools for JS. I use pnpm and it just works, and I understand the benefits. And, I can survive with just npm and yarn. But, to me, it is saying a lot that the python tooling feels more broken than JS. I mean, I lived through the webpack years, and I'm still using JS and have a generally favorable opinion of it.

codethief 2y ago
Could you elaborate on the issues you've been seeing in Zed? I've been using asdf as a version manager for Python for quite some time and haven't really had any issues.
- xrd 2y ago
  You are using asdf and zed successfully? I'm so glad to hear this. I removed asdf because I could not get it to recognize the pyproject.toml file. This is working in harmony for you? With Zed and a venv (from python -m venv .venv for example), Zed properly recognized installed packages and provided type hinting and docs, but when I switched to asdf it did not seem to work. But, I was new to asdf and perhaps was using it incorrectly. I was always assuming that when I'm in the command line, running asdf to use the right python works because the path is correctly established. But, when I run zed, it launches without the path setup step, and things went badly. I'm just speculating, but I could not get type hinting and didn't know how to fix it.

deleted 2y ago

[deleted]

eqvinox 2y ago

The absence of any references to how other ecosystems do this (e.g. oldest widely in use: ELF soversion) strikes me as massive oversight? There's 40-50 years of history of people trying to do close cousins of this.

tpoacher 2y ago

Am I the only person here who thinks the easiest thing to do is to download the python version you need, compile, install in your preferred prefix like any other program and then run it?

zahlman 2y ago
I think that's quite easy, sure. But it's also irrelevant to the discussion. The issue the article is talking about (despite the title) isn't about managing multiple versions of Python. It's with trying to have multiple versions of a third-party library, within a single Python environment.
- tpoacher 2y ago
  Ah you're right, thanks. Admittedly I only skimmed and relied on assumptions of the usual discussion around complaints on the python packaging ecosystem xD Though in principle, my preferred approach here would be similar. Manually install in a particular prefix, add it to the path (manually at runtime or programmatically via the sys module), and then import multiple versions under different namespaces...
  - zahlman 2y ago
    This approach was discussed elsewhere in the thread and you may want to be aware of its limitations (https://news.ycombinator.com/item?id=41706009 https://news.ycombinator.com/item?id=41706009).
    - tpoacher 2y ago
      Thanks! I won't start the same thread here (though thank you for pointing that one out), but when I've come across scenarios similar to those in that thread in my own projects (which, admittedly, while not trivial, were probably not enterprise-scale), the solution generally still involved making sure that one ensured the paths / global variables were suitably modified in the relevant modules before the relevant calls, to ensure you're using the correct namespaces expected by the recipient. Which may be tedious, but to me is not absurd; you're just sticking to the contract expected by the recipient. The bigger issue here (for me) is probably whether those contracts are visible/explicit or not, and how/whether the contract is enforced in the recipient library ... but I would hesitate to call this a multiversion / dependency problem.