7 ms·

This whole problem is because in python you can not specify interfaces and say "I need a function called f that takes an x of type T and returns a U", so instea

This whole problem is because in python you can not specify interfaces and say "I need a function called f that takes an x of type T and returns a U", so instead you encode that in an indirect way, like "I know version x.y.z works so I'll just require that". Any other way risks runtime errors. And to people about to mention types in python: those are also checked at runtime. People keep using these hyper dynamic languages and then running into these robustness issues and scaling limitations brought on by their very dynamism. It makes me mad and sad.

teaearlgraycold 3y ago
Just use TypeScript when possible.
317070 3y ago
There are static type checkers, though: https://github.com/google/pytype https://github.com/google/pytype So you can specify interfaces (protocols) and check them in your installation process.
leni536 3y ago
Function interfaces are better, but not necessarily sufficient. Semantics of functions can also change in a breaking way.
LegionMammal978 3y ago
Since when has static typing eliminated backward-compatibility problems in interfaces? Not all visible runtime behavior can be encoded in the type system, unless you're using a proof assistant or formal-verification system.
- AlotOfReading 3y ago
  Even formal verification misses visible behaviors, the line is just a little farther down. Correct runtime visible behavior can also include side effects like the amount of time taken (e.g. constant time cryptography), or even the amount of heat generated [0]. You're not going to encode everything you could possibly want in a type system without modeling the entire universe, so draw an opinionated line somewhere reasonable and be happy with it. [0] https://news.ycombinator.com/item?id=39751509#39761349 https://news.ycombinator.com/item?id=39751509#39761349
hot_gril 3y ago
Dynamic typing can make importing libraries riskier, but the benefits can outweigh the costs. Also, it's not hard to just not break APIs (unless you're node-redis), and you should have tests anyway if you really care.
oivey 3y ago
It makes me pretty mad and sad that people think static languages solve this problem pretty much at all. If you do, I have a version of liblzma for you to install. If you do, do you release versions of your libraries without version numbers because the compiler will catch any mistakes?
- VS1999 3y ago
  It solves the issue of finding out a function signature changed at compile time instead of runtime, which is infinitely better. The real answer is that serious software developers don't just leave their packages on auto-update and rarely, if ever, update dependencies unless there's a good reason.
  - llm_trw 3y ago
    It doesn't solve the problem of the function body changing though.
  - oivey 3y ago
    It finds the most trivial mistake. It isn’t infinitely better because static typing is far from free.
- deleted 3y ago
  [deleted]
- o11c 3y ago
  Theoretically static languages don't solve this problem, but in practice, programmers writing packages in a static language don't gratuitously break their API every release or so. Which seems far too common in Python-land.
  - oivey 3y ago
    I’m not entirely sure that’s true, and I’m not sure it makes sense to extrapolate all dynamic languages from Python. Huge amounts of effort are expended on Linux distros ensuring that all the packages work together. Much of and maybe most of those packages are written in static languages. Many Python packages don’t have issues with things constantly breaking. I find NumPy, SciPy, the Scikits, and more to be rather stable. I can only think of making trivial fixes in the last few years. I have lots of exotic code using things like Numba that’s been long lived. I’m guessing Flask and Django are pretty stable at this point, but I don’t work on that side of things. Packages undergoing a lot of construction still are less nice. I think that might be the nature of all new things, though. The example at the beginning of this article, TensorFlow, is still a relatively new sort of package and is seeing tons of new development still. Packaging in Python in 2024 still sucks, which is a uniquely Python issue. Python’s slowness necessitating wrapping lots of platform specific binaries doesn’t help. Seemingly even major Python projects like TensorFlow have really only just started making an attempt to version their dependencies. In one of the issues in the article, the issue was TF pinning things way too specifically in the main project. One of the satellite projects had the opposite issue, not even setting min bounds. The Wild West of unpinned deps make it hard for upstream authors to even know they are breaking things. Many people know Python packaging sucks, but I don’t think they know how bad it really is. The slowness is also special to Python. Other languages like Julia and Clojure seem to be much better with these difficulties, and I think in large part this is due to early investments preventing the problems from festering. Rust vs C++ is a good comparison I think. Cargo is better than anything C++ has by far. In C++, it’s common to completely avoid dependencies altogether because the best you’ve had historically is the OS-specific package manager. The issue isn’t static vs dynamic. The issue is early investment in packaging and community uptake.
cardanome 3y ago
> And to people about to mention types in python: those are also checked at runtime. They are not checked at runtime at all. Type declarations are only used for static analyzing tools and not by the runtime. So types are checked BEFORE runtime by the tooling just like they would be in TypeScript or any other language that offers gradual typing. Yes, the dynamic nature of Python does make type safety and certain performance optimizations very difficult but then again it is the dynamic nature that allows for the high productivity of the language. A static language would be far less ergonomic to use for the typical prototyping and explorative programming done in Python.
- int_19h 3y ago
  > A static language would be far less ergonomic to use for the typical prototyping and explorative programming done in Python. A static language without type inference, sure. But that's not the only option. OCaml, for example, will infer object types for you based on what methods are called with what kinds of arguments inside the body.
- flakes 3y ago
  > They are not checked at runtime at all. Type declarations are only used for static analyzing tools and not by the runtime. This is the common usecase, but types certainly are used at runtime by many libraries. Frameworks like FastAPI use the type annotations to declare dependency injection which is resolved during application startup. In other cases like Pydantic, they are used to determine marshalling/unmarshalling strategies.
cqqxo4zV46cp 3y ago
What!? A function’s signature does not completely describe its behaviour. This doesn’t remotely address the problem. Have your preferences all you want, but this is blatantly a case of being blinded by some silly programming language culture war.
burntsushi 3y ago
No. It's a problem because you can only have one version of any given package in your dependency tree. You can't have `foo 2` and `foo 3` in your dependency tree. Without that limitation, there is a release valve of sorts where you can incur two different semver incompatible releases of the same package in your dependency tree in exchange for a working build. The hope is that it would be transitory state until all of your dependencies migrate. Rust, for example, has precisely this same problem, except that it is limited to public dependencies. For example, if `serde 2` were ever to be published, then there would likely be a period of immense pain where, effectively, everyone needs to migrate all at once. Even though `serde 1` and `serde 2` can both appear in the same dependency tree (unlike in Python), because it is a public dependency, everyone needs to be using the same version of the library or else the `Serialize` trait from `serde 1` will be considered distinct from the `Serialize` trait (or whatever) in `serde 2`. But if I, say published a `regex 2.0.0` tomorrow, then folks could migrate at their leisure. The only downside is that you'd have `regex 1` and `regex 2` in your dependency tree. Potentially for a long time until everyone migrated over. But your build would still work because it is uncommon for `regex` to be a public dependency. (Rust does have the semver trick[1] available to it as another release valve of sorts.) This problem is definitely not because of missing interfaces or whatever. [1]: https://github.com/dtolnay/semver-trick https://github.com/dtolnay/semver-trick
- Chris_Newton 3y ago
  It's a problem because you can only have one version of any given package in your dependency tree. You can't have `foo 2` and `foo 3` in your dependency tree. That does seem to be the fundamental problem with the Python model of dependency management. If your dependencies have transitive dependencies of their own but your dependency model is a tree and everything is clearly namespaced/versioned, you might end up with multiple versions of the same package installed, but at least they won’t conflict. If your dependency model is flat but each dependency bakes in its own transitive dependencies so they’re hidden from the rest of the system, for example via static linking, again you might end up with multiple versions of the same package (or some part of it) installed, but again they won’t conflict. But if your dependency model is flat and each dependency can require specific versions of its transitive dependencies to be installed as peers, you fundamentally can’t avoid the potential for unresolvable conflicts. A pragmatic improvement in the third case is, as others have suggested, to replace the SemVer-following mypackage 1.x.y and mypackage 2.x.y with separate top-level packages mypackage1 x.y and mypackage2 x.y. Now you have reintroduced namespaces and you can install mypackage1 and mypackage2 together as peers without conflict. Moreover, if increasing x and y faithfully represent minor and point releases, always using the latest versions of mypackage1 and mypackage2 should normally satisfy any other packages that depend on them, however many there are. Of course it doesn’t always work like that in practice. However, at least the problem is now reduced to manually adjusting versions to resolve conflicts where a package didn’t match its versions to its behaviour properly and/or Hyrum’s Law is relevant, which is probably much less work than before.
deleted 3y ago
[deleted]
eesmith 3y ago
I need a function which, given a graph, returns the graph diameter as an integer. I need another function which, given a graph, uses a faster but approximate method to return the graph diameter as an integer. I need a third function which, given a graph, returns the graph radius as an integer. All three of these functions have an identical type signature. Oh, now I need something which takes a regex pattern string and a haystack string, and returns 1 if the pattern is found in the haystack, otherwise 0. And the regex "|" pattern must match longest first, not left-right. And it needs to support "verbose" mode, which allows comments. And it supports backreference matches. How do you express that type? Now I need to numerically integrate some arbitrary function "f" in the range 0.0 to 1.0. Which of the many numeric integration methods should I use which prevents runtime issues like being unable to converge?

teaearlgraycold 3y ago

Just use TypeScript when possible.

317070 3y ago

There are static type checkers, though: https://github.com/google/pytype https://github.com/google/pytype So you can specify interfaces (protocols) and check them in your installation process.

leni536 3y ago

Function interfaces are better, but not necessarily sufficient. Semantics of functions can also change in a breaking way.

LegionMammal978 3y ago

Since when has static typing eliminated backward-compatibility problems in interfaces? Not all visible runtime behavior can be encoded in the type system, unless you're using a proof assistant or formal-verification system.

AlotOfReading 3y ago
Even formal verification misses visible behaviors, the line is just a little farther down. Correct runtime visible behavior can also include side effects like the amount of time taken (e.g. constant time cryptography), or even the amount of heat generated [0]. You're not going to encode everything you could possibly want in a type system without modeling the entire universe, so draw an opinionated line somewhere reasonable and be happy with it. [0] https://news.ycombinator.com/item?id=39751509#39761349 https://news.ycombinator.com/item?id=39751509#39761349

hot_gril 3y ago

Dynamic typing can make importing libraries riskier, but the benefits can outweigh the costs. Also, it's not hard to just not break APIs (unless you're node-redis), and you should have tests anyway if you really care.

oivey 3y ago

It makes me pretty mad and sad that people think static languages solve this problem pretty much at all. If you do, I have a version of liblzma for you to install. If you do, do you release versions of your libraries without version numbers because the compiler will catch any mistakes?

VS1999 3y ago
It solves the issue of finding out a function signature changed at compile time instead of runtime, which is infinitely better. The real answer is that serious software developers don't just leave their packages on auto-update and rarely, if ever, update dependencies unless there's a good reason.
- llm_trw 3y ago
  It doesn't solve the problem of the function body changing though.
- oivey 3y ago
  It finds the most trivial mistake. It isn’t infinitely better because static typing is far from free.
deleted 3y ago
[deleted]
o11c 3y ago
Theoretically static languages don't solve this problem, but in practice, programmers writing packages in a static language don't gratuitously break their API every release or so. Which seems far too common in Python-land.
- oivey 3y ago
  I’m not entirely sure that’s true, and I’m not sure it makes sense to extrapolate all dynamic languages from Python. Huge amounts of effort are expended on Linux distros ensuring that all the packages work together. Much of and maybe most of those packages are written in static languages. Many Python packages don’t have issues with things constantly breaking. I find NumPy, SciPy, the Scikits, and more to be rather stable. I can only think of making trivial fixes in the last few years. I have lots of exotic code using things like Numba that’s been long lived. I’m guessing Flask and Django are pretty stable at this point, but I don’t work on that side of things. Packages undergoing a lot of construction still are less nice. I think that might be the nature of all new things, though. The example at the beginning of this article, TensorFlow, is still a relatively new sort of package and is seeing tons of new development still. Packaging in Python in 2024 still sucks, which is a uniquely Python issue. Python’s slowness necessitating wrapping lots of platform specific binaries doesn’t help. Seemingly even major Python projects like TensorFlow have really only just started making an attempt to version their dependencies. In one of the issues in the article, the issue was TF pinning things way too specifically in the main project. One of the satellite projects had the opposite issue, not even setting min bounds. The Wild West of unpinned deps make it hard for upstream authors to even know they are breaking things. Many people know Python packaging sucks, but I don’t think they know how bad it really is. The slowness is also special to Python. Other languages like Julia and Clojure seem to be much better with these difficulties, and I think in large part this is due to early investments preventing the problems from festering. Rust vs C++ is a good comparison I think. Cargo is better than anything C++ has by far. In C++, it’s common to completely avoid dependencies altogether because the best you’ve had historically is the OS-specific package manager. The issue isn’t static vs dynamic. The issue is early investment in packaging and community uptake.

cardanome 3y ago

> And to people about to mention types in python: those are also checked at runtime. They are not checked at runtime at all. Type declarations are only used for static analyzing tools and not by the runtime. So types are checked BEFORE runtime by the tooling just like they would be in TypeScript or any other language that offers gradual typing. Yes, the dynamic nature of Python does make type safety and certain performance optimizations very difficult but then again it is the dynamic nature that allows for the high productivity of the language. A static language would be far less ergonomic to use for the typical prototyping and explorative programming done in Python.

int_19h 3y ago
> A static language would be far less ergonomic to use for the typical prototyping and explorative programming done in Python. A static language without type inference, sure. But that's not the only option. OCaml, for example, will infer object types for you based on what methods are called with what kinds of arguments inside the body.
flakes 3y ago
> They are not checked at runtime at all. Type declarations are only used for static analyzing tools and not by the runtime. This is the common usecase, but types certainly are used at runtime by many libraries. Frameworks like FastAPI use the type annotations to declare dependency injection which is resolved during application startup. In other cases like Pydantic, they are used to determine marshalling/unmarshalling strategies.

cqqxo4zV46cp 3y ago

What!? A function’s signature does not completely describe its behaviour. This doesn’t remotely address the problem. Have your preferences all you want, but this is blatantly a case of being blinded by some silly programming language culture war.

burntsushi 3y ago

No. It's a problem because you can only have one version of any given package in your dependency tree. You can't have `foo 2` and `foo 3` in your dependency tree. Without that limitation, there is a release valve of sorts where you can incur two different semver incompatible releases of the same package in your dependency tree in exchange for a working build. The hope is that it would be transitory state until all of your dependencies migrate. Rust, for example, has precisely this same problem, except that it is limited to public dependencies. For example, if `serde 2` were ever to be published, then there would likely be a period of immense pain where, effectively, everyone needs to migrate all at once. Even though `serde 1` and `serde 2` can both appear in the same dependency tree (unlike in Python), because it is a public dependency, everyone needs to be using the same version of the library or else the `Serialize` trait from `serde 1` will be considered distinct from the `Serialize` trait (or whatever) in `serde 2`. But if I, say published a `regex 2.0.0` tomorrow, then folks could migrate at their leisure. The only downside is that you'd have `regex 1` and `regex 2` in your dependency tree. Potentially for a long time until everyone migrated over. But your build would still work because it is uncommon for `regex` to be a public dependency. (Rust does have the semver trick[1] available to it as another release valve of sorts.) This problem is definitely not because of missing interfaces or whatever. [1]: https://github.com/dtolnay/semver-trick https://github.com/dtolnay/semver-trick

Chris_Newton 3y ago
It's a problem because you can only have one version of any given package in your dependency tree. You can't have `foo 2` and `foo 3` in your dependency tree. That does seem to be the fundamental problem with the Python model of dependency management. If your dependencies have transitive dependencies of their own but your dependency model is a tree and everything is clearly namespaced/versioned, you might end up with multiple versions of the same package installed, but at least they won’t conflict. If your dependency model is flat but each dependency bakes in its own transitive dependencies so they’re hidden from the rest of the system, for example via static linking, again you might end up with multiple versions of the same package (or some part of it) installed, but again they won’t conflict. But if your dependency model is flat and each dependency can require specific versions of its transitive dependencies to be installed as peers, you fundamentally can’t avoid the potential for unresolvable conflicts. A pragmatic improvement in the third case is, as others have suggested, to replace the SemVer-following mypackage 1.x.y and mypackage 2.x.y with separate top-level packages mypackage1 x.y and mypackage2 x.y. Now you have reintroduced namespaces and you can install mypackage1 and mypackage2 together as peers without conflict. Moreover, if increasing x and y faithfully represent minor and point releases, always using the latest versions of mypackage1 and mypackage2 should normally satisfy any other packages that depend on them, however many there are. Of course it doesn’t always work like that in practice. However, at least the problem is now reduced to manually adjusting versions to resolve conflicts where a package didn’t match its versions to its behaviour properly and/or Hyrum’s Law is relevant, which is probably much less work than before.

deleted 3y ago

[deleted]

eesmith 3y ago

I need a function which, given a graph, returns the graph diameter as an integer. I need another function which, given a graph, uses a faster but approximate method to return the graph diameter as an integer. I need a third function which, given a graph, returns the graph radius as an integer. All three of these functions have an identical type signature. Oh, now I need something which takes a regex pattern string and a haystack string, and returns 1 if the pattern is found in the haystack, otherwise 0. And the regex "|" pattern must match longest first, not left-right. And it needs to support "verbose" mode, which allows comments. And it supports backreference matches. How do you express that type? Now I need to numerically integrate some arbitrary function "f" in the range 0.0 to 1.0. Which of the many numeric integration methods should I use which prevents runtime issues like being unable to converge?