4 ms·

My love for python was critically hurt when I learned about typing.TYPE_CHECKING. For those unaware, due to the dynamic nature of Python, you declare a variabl

My love for python was critically hurt when I learned about typing.TYPE_CHECKING. For those unaware, due to the dynamic nature of Python, you declare a variable type like this foo: Type This might look like Typescript, but it isn't because "Type" is actually an object. In python classes and functions are first-class objects that you can pass around and assign to variables. The obvious problem of this is that you can only use as a type an object that in "normal python" would be available in the scope of that line, which means that you can't do this: def foo() -> Bar: return Bar() class Bar: pass Because "Bar" is defined AFTER foo() it isn't in the scope when foo() is declared. To get around this you use this weird string-like syntax: def foo() -> "Bar": return Bar() This already looks ugly enough that should make Pythonists ask "Python... what are you doing?" but it gets worse. If you have a cyclic reference between two files, something that works out of the box in statically typed languages like Java, and that works in Python when you aren't using type hints because every object is the same "type" until it quacks like a duck, that isn't going to work if you try to use type hints in python because you're going to end up with a cyclic import. More specifically, you don't need cyclic imports in Python normally because you don't need the types, but you HAVE to import the types to add type hints, which introduces cyclic imports JUST to add type hints. To get around this, the solution is to use this monstrosity: if typing.TYPE_CHECKING: import Foo from foo And that's code that only "runs" when the static type check is statically checking the types. Nobody wants Python 4 but this was such an incredibly convoluted way to add this feature, specially when you consider that it means every module now "over-imports" just to add type hints that they previously didn't have to. Every time I see it makes me think that if type checks are so important maybe we shouldn't be programming Python to begin with.

dandiep 1y ago
Isn’t this solved in 3.14/PEP-649?
- analog31 1y ago
  At last, Pi-thon.
- aw1621107 1y ago
  I want to say it (or something similar at least) was originally addressed by from __future__ import annotations back in Python 3.7/3.8 or thereabouts? I definitely remember having to use stringified types a while back but I haven't needed to for quite a while now.
  - sgarland 1y ago
    Yes, annotations allows you to use the declared types as they are, no strings.
    - FreakLegion 1y ago
      It turns them into thunks (formerly strings) automatically, an important detail if you're inspecting annotations at run time because the performance hit of resolving the actual type can be significant.
      - sgarland 1y ago
        TIL, thanks! It looks like 3.14 is also changing it so that all evaluations are lazy.
zahlman 1y ago
> that isn't going to work if you try to use type hints in python because you're going to end up with a cyclic import. More specifically, you don't need cyclic imports in Python normally because you don't need the types, but you HAVE to import the types to add type hints, which introduces cyclic imports JUST to add type hints. Yes, `typing.TYPE_CHECKING` is there so that you can conditionally avoid imports that are only needed for type annotations. And yes, importing modules can have side effects and performance implications. And yes, I agree it's ugly as sin. But Python does in fact allow for cyclic imports — as long as you're importing the modules themselves, rather than importing names `from` those modules. (By the way, the syntax is the other way around: `from ... import ...`.) https://stackoverflow.com/questions/744373/ https://stackoverflow.com/questions/744373/
ninetyninenine 1y ago
You're actually missing the benefit of this. It's actually a feature. With python, because types are part of python itself, they can thus be programmable. You can create a function that takes in a typehint and returns a new typehint. This is legal python. For example below I create a function that dynamically returns a type that restricts a Dictionary to have a specific key and value. from typing import TypedDict def make_typed_dict(name: str, required_key: str, value_type: type): return TypedDict(name, {required_key: value_type, id: int}) # Example UserDict = make_typed_dict("UserDict", "username", str) def foo(data: UserDict): print(data["username"]) With this power in theory you can create programs where types essentially can "prove" your program correct, and in theory eliminate unit tests. Languages like idris specialize in this. But it's not just rare/specialized languages that do this. Typescript, believe it or not, has programmable types that are so powerful that writing functions that return types like the one above are Actually VERY common place. I was a bit late to the game to typescript but I was shocked to see that it was taking cutting edge stuff from the typing world and making it popular among users. In practice, using types to prove programs to be valid in place of testing is actually a bit too tedious compared with tests so people don't go overboard with it. It is a much more safer route then testing, but much harder. Additionally as of now, the thing with python is that it really depends on how powerful the typechecker is on whether or not it can enforce and execute type level functions. It's certainly possible, it's just nobody has done it yet. I'd go further than this actually. Python is actually a potentially more powerfully typed language than TS. In TS, types are basically another language tacked onto javascript. Both languages are totally different and the typing language is very very limited. The thing with python is that the types and the language ARE the SAME thing. They live in the same universe. You complained about this, but there's a lot of power in that because basically types become turing complete and you can create a type that does anything including proving your whole program correct. Like I said that power depends on the typechecker. Someone needs to create a typechecker that can recognize type level functions and so far it hasn't happened yet. But if you want to play with a language that does this, I believe that language is Idris.
- quietbritishjim 1y ago
  That's not a benefit. That's a monstrosity. And, as you heavily imply in your post, type checkers won't be able to cope with it, eliminating one if the main benefits of type hints. Neither will IDEs / language servers, eliminating the other main benefit.
meander_water 1y ago
There's actually another issue with ForwardRefs. They don't work in the REPL. So this will work when run as a module: def foo() -> "Bar": return Bar() But will throw an error if copy pasted into a REPL. However, all of these issues should be fixed in 3.14 with PEP649 and PEP749: > At compile time, if the definition of an object includes annotations, the Python compiler will write the expressions computing the annotations into its own function. When run, the function will return the annotations dict. The Python compiler then stores a reference to this function in __annotate__ on the object. > This mechanism delays the evaluation of annotations expressions until the annotations are examined, which solves many circular reference problems.
- pcwelder 1y ago
  It doesn't throw error in the REPL though. Surely you meant to share some other example?
- meander_water 1y ago
  Please ignore my first assertion that the behavior between REPL and module is different. This would have been the case if the semantics of the original PEP649 spec had been implemented. But instead, PEP749 ensures that it is not [0]. My bad. [0] https://peps.python.org/pep-0749/#behavior-of-the-repl https://peps.python.org/pep-0749/#behavior-of-the-repl
Too 1y ago
This is trivial to solve by simply not having circular imports. Place the types in one file and the usage of it in others. This has many benefits, like forcing you to think about the dependencies and layers of your architecture. Here is a good read about why, from F# that has the same limitation https://fsharpforfunandprofit.com/posts/cyclic-dependencies/ https://fsharpforfunandprofit.com/posts/cyclic-dependencies/ As others already mentioned, importing __annotations__ also works.
- _flux 1y ago
  If the type is a class with methods, then this method doesn't work, though adding intermediate interface classes (possibly with Generic types) might help in most cases. Python static type system isn't quite the same level as F#. > Well, these complaints are unfounded. "You're holding it wrong." I've also coded quite a bit of OCaml and it had the same limitation (which is where F# picked it up in the first place), and while the issue can be worked around, it still seemed to creep up at times. Rust, also with some virtual OCaml ancestry, went completely the opposite way. My view is that while in principle it's a nice property that you can read and and understand a piece of code by starting from the top and going to the bottom (and a REPL is going to do exactly that), in practice it's not the ultimate nice property to uphold.
  - Too 1y ago
    > If the type is a class with methods, then this method doesn't work Use typing.Self
    - _flux 1y ago
      I meant if you have two classes that need to refer to each other. But good pointer anyway, I hadn't noticed it, thanks!
      - alfons_foobar 1y ago
        I ran into some code recently where this pattern caused me so much headache - class A has an attribute which is an instance of class B, and class B has a "parent" attribute (which points to the instance of class A that class B is an attribute of): class Foo: def __init__(self, bar): self.bar = bar class Bar: def __init__(self, foo): self.foo = foo Obviously both called into each other to do $THINGS... Pure madness. So my suggestion: Try not to have interdependent classes :D
alfons_foobar 1y ago
Agreed that this "hack" is very ugly! On the other hand, I tend to take it as a hint that I should look at my module structure, and see if I can avoid the cyclic import (even if before adding type hints there was no error, there still already was a "semantic dependency"...)
JackSlateur 1y ago
from __future__ import annotations
actinium226 1y ago
> If you have a cyclic reference between two files, Don't have cyclic references between two files. It makes testing very difficult, because in order to test something in one file, you need to import the other one, even though it has nothing to do with the test. It makes the code more difficult to read, because you're importing these two files in places where you only need one of them, and it's not immediately clear why you're importing the second one. And it's not very satisfying to learn that you you're importing the second one not because you "need" it but because the circular import forces you to do so. Every single time you have cyclic references, what you really have are two pieces of code that rely on a third piece of code, so take that third piece, separate it out, and have the first two pieces of code depend on the third piece. Now things can be tested, imports can be made sanely, and life is much better.
- AlienRobot 1y ago
  Using the typical "Rust-killer" example: if you have a linked list where the List in list.py returns a Node type and Node in node.py takes a List in its constructor, you already have a cyclic reference.

dandiep 1y ago

Isn’t this solved in 3.14/PEP-649?

analog31 1y ago
At last, Pi-thon.
aw1621107 1y ago
I want to say it (or something similar at least) was originally addressed by from __future__ import annotations back in Python 3.7/3.8 or thereabouts? I definitely remember having to use stringified types a while back but I haven't needed to for quite a while now.
- sgarland 1y ago
  Yes, annotations allows you to use the declared types as they are, no strings.
  - FreakLegion 1y ago
    It turns them into thunks (formerly strings) automatically, an important detail if you're inspecting annotations at run time because the performance hit of resolving the actual type can be significant.
    - sgarland 1y ago
      TIL, thanks! It looks like 3.14 is also changing it so that all evaluations are lazy.

zahlman 1y ago

> that isn't going to work if you try to use type hints in python because you're going to end up with a cyclic import. More specifically, you don't need cyclic imports in Python normally because you don't need the types, but you HAVE to import the types to add type hints, which introduces cyclic imports JUST to add type hints. Yes, `typing.TYPE_CHECKING` is there so that you can conditionally avoid imports that are only needed for type annotations. And yes, importing modules can have side effects and performance implications. And yes, I agree it's ugly as sin. But Python does in fact allow for cyclic imports — as long as you're importing the modules themselves, rather than importing names `from` those modules. (By the way, the syntax is the other way around: `from ... import ...`.) https://stackoverflow.com/questions/744373/ https://stackoverflow.com/questions/744373/

ninetyninenine 1y ago

You're actually missing the benefit of this. It's actually a feature. With python, because types are part of python itself, they can thus be programmable. You can create a function that takes in a typehint and returns a new typehint. This is legal python. For example below I create a function that dynamically returns a type that restricts a Dictionary to have a specific key and value. from typing import TypedDict def make_typed_dict(name: str, required_key: str, value_type: type): return TypedDict(name, {required_key: value_type, id: int}) # Example UserDict = make_typed_dict("UserDict", "username", str) def foo(data: UserDict): print(data["username"]) With this power in theory you can create programs where types essentially can "prove" your program correct, and in theory eliminate unit tests. Languages like idris specialize in this. But it's not just rare/specialized languages that do this. Typescript, believe it or not, has programmable types that are so powerful that writing functions that return types like the one above are Actually VERY common place. I was a bit late to the game to typescript but I was shocked to see that it was taking cutting edge stuff from the typing world and making it popular among users. In practice, using types to prove programs to be valid in place of testing is actually a bit too tedious compared with tests so people don't go overboard with it. It is a much more safer route then testing, but much harder. Additionally as of now, the thing with python is that it really depends on how powerful the typechecker is on whether or not it can enforce and execute type level functions. It's certainly possible, it's just nobody has done it yet. I'd go further than this actually. Python is actually a potentially more powerfully typed language than TS. In TS, types are basically another language tacked onto javascript. Both languages are totally different and the typing language is very very limited. The thing with python is that the types and the language ARE the SAME thing. They live in the same universe. You complained about this, but there's a lot of power in that because basically types become turing complete and you can create a type that does anything including proving your whole program correct. Like I said that power depends on the typechecker. Someone needs to create a typechecker that can recognize type level functions and so far it hasn't happened yet. But if you want to play with a language that does this, I believe that language is Idris.

quietbritishjim 1y ago
That's not a benefit. That's a monstrosity. And, as you heavily imply in your post, type checkers won't be able to cope with it, eliminating one if the main benefits of type hints. Neither will IDEs / language servers, eliminating the other main benefit.

meander_water 1y ago

There's actually another issue with ForwardRefs. They don't work in the REPL. So this will work when run as a module: def foo() -> "Bar": return Bar() But will throw an error if copy pasted into a REPL. However, all of these issues should be fixed in 3.14 with PEP649 and PEP749: > At compile time, if the definition of an object includes annotations, the Python compiler will write the expressions computing the annotations into its own function. When run, the function will return the annotations dict. The Python compiler then stores a reference to this function in __annotate__ on the object. > This mechanism delays the evaluation of annotations expressions until the annotations are examined, which solves many circular reference problems.

pcwelder 1y ago
It doesn't throw error in the REPL though. Surely you meant to share some other example?
meander_water 1y ago
Please ignore my first assertion that the behavior between REPL and module is different. This would have been the case if the semantics of the original PEP649 spec had been implemented. But instead, PEP749 ensures that it is not [0]. My bad. [0] https://peps.python.org/pep-0749/#behavior-of-the-repl https://peps.python.org/pep-0749/#behavior-of-the-repl

Too 1y ago

This is trivial to solve by simply not having circular imports. Place the types in one file and the usage of it in others. This has many benefits, like forcing you to think about the dependencies and layers of your architecture. Here is a good read about why, from F# that has the same limitation https://fsharpforfunandprofit.com/posts/cyclic-dependencies/ https://fsharpforfunandprofit.com/posts/cyclic-dependencies/ As others already mentioned, importing __annotations__ also works.

_flux 1y ago
If the type is a class with methods, then this method doesn't work, though adding intermediate interface classes (possibly with Generic types) might help in most cases. Python static type system isn't quite the same level as F#. > Well, these complaints are unfounded. "You're holding it wrong." I've also coded quite a bit of OCaml and it had the same limitation (which is where F# picked it up in the first place), and while the issue can be worked around, it still seemed to creep up at times. Rust, also with some virtual OCaml ancestry, went completely the opposite way. My view is that while in principle it's a nice property that you can read and and understand a piece of code by starting from the top and going to the bottom (and a REPL is going to do exactly that), in practice it's not the ultimate nice property to uphold.
- Too 1y ago
  > If the type is a class with methods, then this method doesn't work Use typing.Self
  - _flux 1y ago
    I meant if you have two classes that need to refer to each other. But good pointer anyway, I hadn't noticed it, thanks!
    - alfons_foobar 1y ago
      I ran into some code recently where this pattern caused me so much headache - class A has an attribute which is an instance of class B, and class B has a "parent" attribute (which points to the instance of class A that class B is an attribute of): class Foo: def __init__(self, bar): self.bar = bar class Bar: def __init__(self, foo): self.foo = foo Obviously both called into each other to do $THINGS... Pure madness. So my suggestion: Try not to have interdependent classes :D

alfons_foobar 1y ago

Agreed that this "hack" is very ugly! On the other hand, I tend to take it as a hint that I should look at my module structure, and see if I can avoid the cyclic import (even if before adding type hints there was no error, there still already was a "semantic dependency"...)

JackSlateur 1y ago

from __future__ import annotations

actinium226 1y ago

> If you have a cyclic reference between two files, Don't have cyclic references between two files. It makes testing very difficult, because in order to test something in one file, you need to import the other one, even though it has nothing to do with the test. It makes the code more difficult to read, because you're importing these two files in places where you only need one of them, and it's not immediately clear why you're importing the second one. And it's not very satisfying to learn that you you're importing the second one not because you "need" it but because the circular import forces you to do so. Every single time you have cyclic references, what you really have are two pieces of code that rely on a third piece of code, so take that third piece, separate it out, and have the first two pieces of code depend on the third piece. Now things can be tested, imports can be made sanely, and life is much better.

AlienRobot 1y ago
Using the typical "Rust-killer" example: if you have a linked list where the List in list.py returns a Node type and Node in node.py takes a List in its constructor, you already have a cyclic reference.