3 ms·

This comes up every now and again, and there are two fairly simple examples that I think show the complexities: One: Library A takes a callback function, and

by orf 2y ago

This comes up every now and again, and there are two fairly simple examples that I think show the complexities: One: Library A takes a callback function, and catches “request.HttpError” when invoking that callback. The callback throws an exception from a differing version of “request”, which is missing an attribute that the exception handler requires. What happens? How? Two: Library A has a function that returns a “request.Response” object. Library B has a function that accepts a “request.Response” object, and performs “isinstance”/type equality on the object. Library A and library B have differing and incompatible dependencies on “request”. What version of the request object is sent to library B from library A, and how does “isinstance”/“type” interact with it? Both of these resolve around class identities. In Python they are intrinsically linked to the defining module. Either you break this invariant and have two incompatible/different types have the same identity and introduce all kinds of bugs, or you don’t and also introduce all kinds of bugs - “yes, this is a request.Response object, but this method doesn’t exist on this request.Response object”, or “yes this is someone’s request.Response object, but it’s not your request.Response object” Getting different module imports to succeed is more than possible, getting them to work together is another thing entirely. One solution to this is the concept of visibility, which in Python is famously “not really a thing”. It’s safe to use incompatible versions of a library as long as the types are not visible - I.e no method returns a request.Response object, so the module is essentially a completely private implementation detail. This is how Rust handles this, I think. However is obviously fucked by exceptions, so it seems pretty intractable.

the_mitsuhiko 2y ago
That is not any different in Python than it is in Rust, Go or JavaScript. Yes: it's a problem if those different dependency versions shine through via public APIs. However there are plenty of cases where the dependency you are using in a specific version is isolated within your own internal API. I think if Python were to want to go down this path it should be isolated to explicit migration cases for specific libraries that want to opt themselves into multi-version resolution. I think it would enable the move of pretty core libraries in the ecosystem in backwards incompatible ways in a much smoother way than it is today.
- orf 2y ago
  The problem with this is exceptions: they easily allow dependencies to escape, be that via uncaught exceptions or wrapped ones. Go and JavaScript have type systems and idioms far more amenable to this kind of thing (interfaces for Go, no real type identity + reliance on structural typing for JS) and rely a lot less on the kind of reflection common in Python (identity, class, etc). I guess there are some use cases for this, I just feel that the lack of ability to enforce visibility combined with the “rock and a hard place” identity trade-off limits the practical usefulness.
  - simiones 2y ago
    Exceptions are no different from Go's error types (and in general interface types in any language) from this point of view. If moduleA is doing something like `errors.Is(err, ModuleBError)` on an error that was returned from moduleC which uses a diffferent version of moduleB, you'll get the same issue.
    - orf 2y ago
      That’s interesting - is it common to do this instead of casting to an interface? It seems a lot more impactful with Python due to type equality being core to how exceptions are handled, even if there are similarities.
      - simiones 2y ago
        Well, the most common is of course `if err != nil`, which is unaffected. But in the very rare occasions that someone is actually handling errors in Go, `errors.Is` and `errors.As` are recommended over plain casts since they correctly handle composite exceptions Say a function returns `fmt.Errorf("Error while doing intermediate operation: %w", lowerLevelErr)`, where `lowerLevelErr` is `ModuleBError`. Then, if you do `if _, ok := err.(ModuleBError) {...}`, this will return false; but if you do `if errors.Is(err, ModuleBError)`, you will get the expected true. Regardless, the core problem would be the same: if your code can handle moduleB v1.5 errors but it's receiving moduleB v.17 errors, then it may not be able to handle them. This same thing happens with error values, Exceptions, and in fact any other case of two different implementations returned under the same interface. You even have this problem with C-style integer error codes: say in version 1.5, whenever you try to open a path that is not recognized, you return the int 404. But in 1.7, you return 404 for a missing file, but 407 if it's a missing dir. Any code that is checking for err > 0 will keep working exactly as well, but code which was checking for code 404 to fix missing dir paths is now broken, even though the types are all exactly the same.
snatchpiesinger 2y ago
You can have private and public dependencies. Private dependencies are the ones that don't show up on your interface at all. That is you don't return it, you don't throw it or catch it (other than passing through), you don't take callbacks that have it in their signature, etc... You can use private dependencies for the implementation. It should be safe to use multiple versions of the same library, as long as they are used as private dependencies of unrelated dependencies. It would require some tooling support to do it safely: 1. Being able to declare dependencies are "private" or "public". 2. Tooling to check that you don't use private dependencies in your interfaces. This requires type annotations to gain some confidence, but even then, exceptions are a problem that is hard to check for (in Python that is). In compiled languages there are additional compilications, like exported symbols. It is solveable in some controlled circumstances, but it's best to just not have this problem.
- orf 2y ago
  > you don't throw it or catch it Herein lies the issue: in this context exceptions can be thought of as the same as returns. So you actually need to catch/handle all possible exceptions in order to not leak private types. Also what does “except requests.HttpError” do in an outer context? It checks the class of an exception - so either it doesn’t catch some other modules version of requests.HttpError (confusion, invariants broken) or it does (confusion, invariants broken).
  - snatchpiesinger 2y ago
    It's fine as long as you catch all exceptions, and only produce ones that you document. Your users aren't supposed to know that you used `requests` at all.
    - orf 2y ago
      Sure, but who does this? And the typical pattern is to wrap exceptions, giving you access to the inner exception if you need more context. The requests HTTP exception contains the request and response object. Wrapping that would be a huge pain and a lot of code.

by orf 2y ago

the_mitsuhiko 2y ago

That is not any different in Python than it is in Rust, Go or JavaScript. Yes: it's a problem if those different dependency versions shine through via public APIs. However there are plenty of cases where the dependency you are using in a specific version is isolated within your own internal API. I think if Python were to want to go down this path it should be isolated to explicit migration cases for specific libraries that want to opt themselves into multi-version resolution. I think it would enable the move of pretty core libraries in the ecosystem in backwards incompatible ways in a much smoother way than it is today.

orf 2y ago
The problem with this is exceptions: they easily allow dependencies to escape, be that via uncaught exceptions or wrapped ones. Go and JavaScript have type systems and idioms far more amenable to this kind of thing (interfaces for Go, no real type identity + reliance on structural typing for JS) and rely a lot less on the kind of reflection common in Python (identity, class, etc). I guess there are some use cases for this, I just feel that the lack of ability to enforce visibility combined with the “rock and a hard place” identity trade-off limits the practical usefulness.
- simiones 2y ago
  Exceptions are no different from Go's error types (and in general interface types in any language) from this point of view. If moduleA is doing something like `errors.Is(err, ModuleBError)` on an error that was returned from moduleC which uses a diffferent version of moduleB, you'll get the same issue.
  - orf 2y ago
    That’s interesting - is it common to do this instead of casting to an interface? It seems a lot more impactful with Python due to type equality being core to how exceptions are handled, even if there are similarities.
    - simiones 2y ago
      Well, the most common is of course `if err != nil`, which is unaffected. But in the very rare occasions that someone is actually handling errors in Go, `errors.Is` and `errors.As` are recommended over plain casts since they correctly handle composite exceptions Say a function returns `fmt.Errorf("Error while doing intermediate operation: %w", lowerLevelErr)`, where `lowerLevelErr` is `ModuleBError`. Then, if you do `if _, ok := err.(ModuleBError) {...}`, this will return false; but if you do `if errors.Is(err, ModuleBError)`, you will get the expected true. Regardless, the core problem would be the same: if your code can handle moduleB v1.5 errors but it's receiving moduleB v.17 errors, then it may not be able to handle them. This same thing happens with error values, Exceptions, and in fact any other case of two different implementations returned under the same interface. You even have this problem with C-style integer error codes: say in version 1.5, whenever you try to open a path that is not recognized, you return the int 404. But in 1.7, you return 404 for a missing file, but 407 if it's a missing dir. Any code that is checking for err > 0 will keep working exactly as well, but code which was checking for code 404 to fix missing dir paths is now broken, even though the types are all exactly the same.

snatchpiesinger 2y ago

You can have private and public dependencies. Private dependencies are the ones that don't show up on your interface at all. That is you don't return it, you don't throw it or catch it (other than passing through), you don't take callbacks that have it in their signature, etc... You can use private dependencies for the implementation. It should be safe to use multiple versions of the same library, as long as they are used as private dependencies of unrelated dependencies. It would require some tooling support to do it safely: 1. Being able to declare dependencies are "private" or "public". 2. Tooling to check that you don't use private dependencies in your interfaces. This requires type annotations to gain some confidence, but even then, exceptions are a problem that is hard to check for (in Python that is). In compiled languages there are additional compilications, like exported symbols. It is solveable in some controlled circumstances, but it's best to just not have this problem.

orf 2y ago
> you don't throw it or catch it Herein lies the issue: in this context exceptions can be thought of as the same as returns. So you actually need to catch/handle all possible exceptions in order to not leak private types. Also what does “except requests.HttpError” do in an outer context? It checks the class of an exception - so either it doesn’t catch some other modules version of requests.HttpError (confusion, invariants broken) or it does (confusion, invariants broken).
- snatchpiesinger 2y ago
  It's fine as long as you catch all exceptions, and only produce ones that you document. Your users aren't supposed to know that you used `requests` at all.
  - orf 2y ago
    Sure, but who does this? And the typical pattern is to wrap exceptions, giving you access to the inner exception if you need more context. The requests HTTP exception contains the request and response object. Wrapping that would be a huge pain and a lot of code.