9 ms·

Why if TYPE_CHECKING?

28 points by BerislavLopac 3y ago

monicaaa 3y ago
[dead]
diarrhea 3y ago
One question comes to mind: isn't cyclic dependencies something to be resolved at the root cause (remove the cycle)? It's an issue that causes various pains down the line, like type checking in this case.
- kkirsche 3y ago
  Broadly speaking, yes, but for some situations like database models with bi-directional relationships that can cause unnecessary maintenance burden simply for the benefit of type checking which this allows you to work around since the cyclical nature is only caused by typing not runtime semantics
- wzdd 3y ago
  The issue is that, often, without type annotations there would be no cyclic dependency (because any type matching the implicit interface would work). So by introducing type annotations (which means extra imports), you end up having to refactor quite a lot of code and the end game is a one-type-per-module-with-separate-interface-definition system of the sort encountered in, well, statically-typed languages. If you don't do this you end up with the 'Type in string' annotations mentioned in the article and TYPE_CHECKING-gated imports. Both of which feel kind of hackish. It's fine to have implicit circular dependencies in Python because duck typing means that every function is effectively generic. Type annotations eliminate this and turn Python into a fundamentally different language. Type annotations aren't just extra boilerplate in function signatures, they also have these sorts of knock-on effects.
  - RhysU 3y ago
    I read 'Type in string' like a forward declaration. I find uses for them within a single module. They're not always hacks.
  - gpderetta 3y ago
    Aside from type checking being a clusterfuck in python, wouldn't checking against a protocol be the right thing?
- maxcoder4 3y ago
  In python it's very hard to avoid cyclic dependencies. Something as simple as a parent-child link between two classes is a cyclic dependency, and if the classes are in different files you have a car like in the blog post. This comes up especially hard when typing your codebase, because you have to name every type, and in python implementation is the interface, i.e. you can't just include a type definition file.
  - remram 3y ago
    > parent-child link between two classes is a cyclic dependency What do you mean? Where is the cycle?
    - coryrc 3y ago
      class Parent: children : List[Child] class Child: __init__(self, parent: Parent):
      - remram 3y ago
        Ah yes, it might be what they mean. I assumed inheritance. Thanks.
    - diarrhea 3y ago
      I also wonder. If a child class knows its concrete parent (apart from where inheritance is declared), and not just references it via super(), things have already gone badly wrong.
    - MereInterest 3y ago
      The parent holds a reference to the child. The child holds a reference to the parent.
      - remram 3y ago
        Why does the parent hold a reference to the child? Are you talking about a specific pattern or implementation detail in the interpreter? Because I would not call this "dependency"...
- m000 3y ago
  Removing cycles may require extensive refactoring of the code, sometimes resulting in unintuitive layouts. All, just to solve the cyclic-import errors. At the core of the problem is the fact that there's no thing as "partial import" in Python. When you do "from M import A" in Python, all the contents of M are evaluated and a reference to A is added to the current namespace. So, cyclic dependencies arise sooner or later, unless you adopt some very un-pythonic style for your codebase (e.g. one file per class).
L3viathan 3y ago
> However, Python doesn’t have the compile-time check, because it’s an interpreted language that is dynamically-typed, which means its only real place to check is at runtime Pet peeve: Python is compiled (into bytecode), so it could theoretically do checks at compile time. The "dynamically-typed" part is correct and is the real reason.
- remram 3y ago
  TypeScript is also interpreted, and does type checking without running code. Their argument is very weird.
  - SlimTim10 3y ago
    That's an odd thing to say. The distinction between compiled and interpreted languages is very fuzzy, but TypeScript is clearly on the compiled side.
    - gpderetta 3y ago
      It is not so much that it is fuzzy, it is that the two definitions are not exclusive. A compilation translates a language to another, whether it is native code, internal bytecode, portable bytecode, or another source language. Compilation might also type check and optimize depending on the language. Interpretation executes a language; normally bytecode is executed, but pure source level interpreters exist. Native code is "interpreted" by the cpu itself of course. For example JAVA is certainly a compiled language, but it is typically compiled to bytecode and then interpreted or JITed (which is a combined interpreter/compiler).
    - remram 3y ago
      Likewise, the Python typing annotation language could very well be on the compiled side... Someone took the interpreted JavaScript language, added typing syntax on top, and typing can be checked statically. The Python team took the interpreted Python language, added typing syntax on top, and here we are reading a claim that it can't be checked statically because it's interpreted. You understand my puzzlement.
albertzeyer 3y ago
You should also add this: from __future__ import annotations Then you will not get this error (NameError: name 'Sequence' is not defined) because it will not need the reference to `Sequence` for the annotation but the annotation is simply a string. Using `if TYPE_CHECKING` is also useful when you want to speed up the module load time by not importing unnecessary modules (unnecessary at module load time). And then, also to resolve import cycles, where you still want to annotate the function argument types. One problem though: If this is a type/object which is not needed for module load time (there it's just used to annotate some function arg type), but which will be needed inside the function, type checkers and IDEs (e.g. PyCharm) will not show any problem, but then at runtime, you will the NameError once it tries to use the type/object. Example module: from __future__ import annotations from typing import TYPE_CHECKING if TYPE_CHECKING: import torch def agi_model(query: torch.Tensor) -> torch.Tensor: torch.setup_secret_agi() ... This module will load just fine, because `torch` is only needed for type checking at module load time. However, once you call this `agi_model` function, it will fail because `torch` is actually unknown at runtime. However, no IDE will show any problem with this code. (Edit: Eclipse/PyDev handles it correctly now, see answer below.) Then you would add another `import torch` inside that `agi_model` function. However, then the IDE might even complain that this hides the outer scope `torch`.
- fabioz 3y ago
  > However, no IDE will show any problem with this code. That's not entirely True... I just checked it here and Eclipse/PyDev does flag this correctly -- since version 11.0.0 ;) See: https://www.pydev.org/history_pydev.html https://www.pydev.org/history_pydev.html (Imports found inside a typing.TYPE_CHECKING will be considered undefined if the scope that uses it requires it to be available when not type-checking).
- guappa 3y ago
  Adding that will break every single runtime typechecking module. Which is why in the end it was never included in python proper. If you don't do runtime typechecking in the entire program, sure.
  - 3y ago
bruce343434 3y ago
Professionals should stop putting up with unprofessional tools and languages. This? This whole mess is not professional. If you want types, do not use python. Python is not a typed language. You can't just bolt on a type system, as is being live demonstrated here.
- vbezhenar 3y ago
  TypeScript is a good example of perfectly "bolted on" type system.
  - bruce343434 3y ago
    But it is its own thing that compiles down to javascript, and is otherwise syntactically and semantically "very very similar" to javascript. It IS always statically typechecked and has good, or at least well defined, semantics for typing. Typescript does not change base javascript. It really is a well separated layer on top of javascript. The way python does it is different. There's no "typethon". The python grammar was simply extended to allow any random expression as a "type". In fact: x: print("hi") = 4 is valid python, and it does print "hi" without even so much as a warning. Personally, I don't think "print(hi)" should be a valid type.
    - tomn 3y ago
      This feels like splitting hairs -- python and typescript are equivalent, as they are both dynamically typed languages with type annotations. The fact that the typescript implementation does not run typescript directly, and cPython does not type check before running the code, are both implementation details. Typescript is just a better type system on top of javascript than python's type system is on top of untyped python, because typescript is a cohesive design rather than a collection or random stuff accumulated over time. There's no reason that couldn't have happened for python, it just didn't.
    - lijok 3y ago
      > Personally, I don't think "print(hi)" should be a valid type. And it isn't: a.py:1: error: Invalid type comment or annotation [valid-type] a.py:1: note: Suggestion: use print[...] instead of print(...) Found 1 error in 1 file (checked 1 source file)
avgcorrection 3y ago
I rarely butt into practices for languages that I don’t use.[1] But having a flag for “are we type-checking” is a, well, huge red flag. [1] Except some light scripting
- Ensorceled 3y ago
  Yeah, the article explains in depth why we are this stage with Python ...
  - avgcorrection 3y ago
    There is always a history. Which does not imply a good reason.
    - Ensorceled 3y ago
      It's a work in progress, the article didn't say the current situation is ideal, nor am I.
      - avgcorrection 3y ago
        Yeah you’re not saying anything.
- leoff 3y ago
  doesn't Typescript have something similar? ``` import type {foo} from ... ```
  - rundev 3y ago
    Not really. `import type` just means that this import is guaranteed to be removed when compiling to JS. TS already fully erases types during compilation, and this is just a way to guarantee that the imports of types are guaranteed to be erased as well. E.g. for `import { SomeType } from "backend-server"`, you don't want to include your entire server code in the frontend just because you imported the response type of some JSON API. `import type` neatly solves this issue, and it even enforces that you can't import anything but types from your backend. I also want to state that TS already removes regular `import`s that only import types. `import type` is mostly used to help bundlers. https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-8.html https://www.typescriptlang.org/docs/handbook/release-notes/t...
ericvsmith 3y ago
PEP 649 “Deferred Evaluation Of Annotations Using Descriptors” should result in vastly reducing the need for TYPE_CHECKING. https://peps.python.org/pep-0649/ https://peps.python.org/pep-0649/
- durumu 3y ago
  I think in practice that increases the need for TYPE_CHECKING. If you use deferred annotations, and you don't import the types you use, type checkers and IDEs will complain, and go to definition won't work. But if you import the types outside of a TYPE_CHECKING block it's wasteful, since you don't actually need the import for your code to run. So you need TYPE_CHECKING to satisfy your linter without impacting runtime performance.
- guappa 3y ago
  It will also break any runtime typecheck module that you might want to use.
spinningslate 3y ago
Small but important correction required in point 3 of the TL;DR: > Python doesn’t care about types at runtime This should say "Python doesn't care about type annotations at runtime." Python _does_ care about types at runtime - but it doesn't use type annotations to compute them. That doesn't detract from what's otherwise a really clear and helpful post. Python is living through a schizophrenic period in its evolution, and it's causing some problems. I hope it gets ironed out, though don't underestimate the difficulty. Python remains my go-to for many problems: a joy to use (for me) in many cases. But I definitely feel the need for static typing as codebase size increases. Having type annotations is good, but not being able to rely on them (i.e. static typing is not sound) adds to cognitive load and detracts from a confident development experience.
- the_af 3y ago
  I agree with the gist of your comment. I will add that, in my experience, having type annotations that are unreliable, cumbersome, with so many edge cases and requiring this kind of magic (e.g. "if TYPE_CHECKING", etc) which is beyond most "casual" Python users effectively means type annotations are a very hard sell for most non-hardcore team mates. I've been in a position to try improving processes and code quality of the Python codebase, and introducing mypy and type annotations has always been a very hard sell. The team just won't accept the benefits, given the difficulties in using it. If only Python made it easier and more reliable, but the current state is a mess. I come from the static typing world, so of course I see the benefits. But most/all of my team mates don't, and so they see this as some really odd hoops I want them to jump through, for minimal benefit (in their eyes).
  - durumu 3y ago
    I think type annotations are extremely valuable even if you don't use mypy. On my team we basically use them as comments that are understood by the IDE, but aren't enforced. Of course comments can lie, but appeasing mypy really is a lot of work, and I would generally rather have a slightly dishonest but mostly correct type hint than some TypeVar monstrosity or, worse, Any.
mgaunard 3y ago
most of the Python code I've seen with type annotations appears to be significantly worse than the code without them. You're in Python. Either embrace it or use another language.
- n3150n 3y ago
  Could you expand on it? It is truly surprising for me that anyone would find code with type annotations to be significantly worse, for any reasons whatsoever. On the contrary, I joyfully read and write code with type annotations. It is obviously very useful knowing which object types a function expects and which it will return.
  - mgaunard 3y ago
    verbosity tends to make code worse, not better.
kissgyorgy 3y ago
> However, Python doesn’t have the compile-time check Yes it does. All Python source code is parsed and compiled into bytecode. SyntaxError is raised before runtime.
- gpderetta 3y ago
  I think the author is obviously talking about compile time type checks here.
- maweki 3y ago
  It's arguable whether a SyntaxError happens as a compile-time check. It's true that it is raised before runtime. But a document that can be parsed according to the syntax definition is a necessary precondition to all checks of any static program property. Syntactic correctness in itself is not a static property of a program. A program is, by definition, syntactically correct. Without a correct syntax there is no way to assign meaning or execute or interpret, etc.

28 points by BerislavLopac 3y ago

[dead]

One question comes to mind: isn't cyclic dependencies something to be resolved at the root cause (remove the cycle)? It's an issue that causes various pains down the line, like type checking in this case.

kkirsche 3y ago
Broadly speaking, yes, but for some situations like database models with bi-directional relationships that can cause unnecessary maintenance burden simply for the benefit of type checking which this allows you to work around since the cyclical nature is only caused by typing not runtime semantics
wzdd 3y ago
The issue is that, often, without type annotations there would be no cyclic dependency (because any type matching the implicit interface would work). So by introducing type annotations (which means extra imports), you end up having to refactor quite a lot of code and the end game is a one-type-per-module-with-separate-interface-definition system of the sort encountered in, well, statically-typed languages. If you don't do this you end up with the 'Type in string' annotations mentioned in the article and TYPE_CHECKING-gated imports. Both of which feel kind of hackish. It's fine to have implicit circular dependencies in Python because duck typing means that every function is effectively generic. Type annotations eliminate this and turn Python into a fundamentally different language. Type annotations aren't just extra boilerplate in function signatures, they also have these sorts of knock-on effects.
- RhysU 3y ago
  I read 'Type in string' like a forward declaration. I find uses for them within a single module. They're not always hacks.
- gpderetta 3y ago
  Aside from type checking being a clusterfuck in python, wouldn't checking against a protocol be the right thing?
maxcoder4 3y ago
In python it's very hard to avoid cyclic dependencies. Something as simple as a parent-child link between two classes is a cyclic dependency, and if the classes are in different files you have a car like in the blog post. This comes up especially hard when typing your codebase, because you have to name every type, and in python implementation is the interface, i.e. you can't just include a type definition file.
- remram 3y ago
  > parent-child link between two classes is a cyclic dependency What do you mean? Where is the cycle?
  - coryrc 3y ago
    class Parent: children : List[Child] class Child: __init__(self, parent: Parent):
    - remram 3y ago
      Ah yes, it might be what they mean. I assumed inheritance. Thanks.
  - diarrhea 3y ago
    I also wonder. If a child class knows its concrete parent (apart from where inheritance is declared), and not just references it via super(), things have already gone badly wrong.
  - MereInterest 3y ago
    The parent holds a reference to the child. The child holds a reference to the parent.
    - remram 3y ago
      Why does the parent hold a reference to the child? Are you talking about a specific pattern or implementation detail in the interpreter? Because I would not call this "dependency"...
m000 3y ago
Removing cycles may require extensive refactoring of the code, sometimes resulting in unintuitive layouts. All, just to solve the cyclic-import errors. At the core of the problem is the fact that there's no thing as "partial import" in Python. When you do "from M import A" in Python, all the contents of M are evaluated and a reference to A is added to the current namespace. So, cyclic dependencies arise sooner or later, unless you adopt some very un-pythonic style for your codebase (e.g. one file per class).

L3viathan 3y ago

> However, Python doesn’t have the compile-time check, because it’s an interpreted language that is dynamically-typed, which means its only real place to check is at runtime Pet peeve: Python is compiled (into bytecode), so it could theoretically do checks at compile time. The "dynamically-typed" part is correct and is the real reason.

remram 3y ago
TypeScript is also interpreted, and does type checking without running code. Their argument is very weird.
- SlimTim10 3y ago
  That's an odd thing to say. The distinction between compiled and interpreted languages is very fuzzy, but TypeScript is clearly on the compiled side.
  - gpderetta 3y ago
    It is not so much that it is fuzzy, it is that the two definitions are not exclusive. A compilation translates a language to another, whether it is native code, internal bytecode, portable bytecode, or another source language. Compilation might also type check and optimize depending on the language. Interpretation executes a language; normally bytecode is executed, but pure source level interpreters exist. Native code is "interpreted" by the cpu itself of course. For example JAVA is certainly a compiled language, but it is typically compiled to bytecode and then interpreted or JITed (which is a combined interpreter/compiler).
  - remram 3y ago
    Likewise, the Python typing annotation language could very well be on the compiled side... Someone took the interpreted JavaScript language, added typing syntax on top, and typing can be checked statically. The Python team took the interpreted Python language, added typing syntax on top, and here we are reading a claim that it can't be checked statically because it's interpreted. You understand my puzzlement.

albertzeyer 3y ago

You should also add this: from __future__ import annotations Then you will not get this error (NameError: name 'Sequence' is not defined) because it will not need the reference to `Sequence` for the annotation but the annotation is simply a string. Using `if TYPE_CHECKING` is also useful when you want to speed up the module load time by not importing unnecessary modules (unnecessary at module load time). And then, also to resolve import cycles, where you still want to annotate the function argument types. One problem though: If this is a type/object which is not needed for module load time (there it's just used to annotate some function arg type), but which will be needed inside the function, type checkers and IDEs (e.g. PyCharm) will not show any problem, but then at runtime, you will the NameError once it tries to use the type/object. Example module: from __future__ import annotations from typing import TYPE_CHECKING if TYPE_CHECKING: import torch def agi_model(query: torch.Tensor) -> torch.Tensor: torch.setup_secret_agi() ... This module will load just fine, because `torch` is only needed for type checking at module load time. However, once you call this `agi_model` function, it will fail because `torch` is actually unknown at runtime. However, no IDE will show any problem with this code. (Edit: Eclipse/PyDev handles it correctly now, see answer below.) Then you would add another `import torch` inside that `agi_model` function. However, then the IDE might even complain that this hides the outer scope `torch`.

fabioz 3y ago
> However, no IDE will show any problem with this code. That's not entirely True... I just checked it here and Eclipse/PyDev does flag this correctly -- since version 11.0.0 ;) See: https://www.pydev.org/history_pydev.html https://www.pydev.org/history_pydev.html (Imports found inside a typing.TYPE_CHECKING will be considered undefined if the scope that uses it requires it to be available when not type-checking).
guappa 3y ago
Adding that will break every single runtime typechecking module. Which is why in the end it was never included in python proper. If you don't do runtime typechecking in the entire program, sure.
- 3y ago

bruce343434 3y ago

Professionals should stop putting up with unprofessional tools and languages. This? This whole mess is not professional. If you want types, do not use python. Python is not a typed language. You can't just bolt on a type system, as is being live demonstrated here.

vbezhenar 3y ago
TypeScript is a good example of perfectly "bolted on" type system.
- bruce343434 3y ago
  But it is its own thing that compiles down to javascript, and is otherwise syntactically and semantically "very very similar" to javascript. It IS always statically typechecked and has good, or at least well defined, semantics for typing. Typescript does not change base javascript. It really is a well separated layer on top of javascript. The way python does it is different. There's no "typethon". The python grammar was simply extended to allow any random expression as a "type". In fact: x: print("hi") = 4 is valid python, and it does print "hi" without even so much as a warning. Personally, I don't think "print(hi)" should be a valid type.
  - tomn 3y ago
    This feels like splitting hairs -- python and typescript are equivalent, as they are both dynamically typed languages with type annotations. The fact that the typescript implementation does not run typescript directly, and cPython does not type check before running the code, are both implementation details. Typescript is just a better type system on top of javascript than python's type system is on top of untyped python, because typescript is a cohesive design rather than a collection or random stuff accumulated over time. There's no reason that couldn't have happened for python, it just didn't.
  - lijok 3y ago
    > Personally, I don't think "print(hi)" should be a valid type. And it isn't: a.py:1: error: Invalid type comment or annotation [valid-type] a.py:1: note: Suggestion: use print[...] instead of print(...) Found 1 error in 1 file (checked 1 source file)

avgcorrection 3y ago

I rarely butt into practices for languages that I don’t use.[1] But having a flag for “are we type-checking” is a, well, huge red flag. [1] Except some light scripting

Ensorceled 3y ago
Yeah, the article explains in depth why we are this stage with Python ...
- avgcorrection 3y ago
  There is always a history. Which does not imply a good reason.
  - Ensorceled 3y ago
    It's a work in progress, the article didn't say the current situation is ideal, nor am I.
    - avgcorrection 3y ago
      Yeah you’re not saying anything.
leoff 3y ago
doesn't Typescript have something similar? ``` import type {foo} from ... ```
- rundev 3y ago
  Not really. `import type` just means that this import is guaranteed to be removed when compiling to JS. TS already fully erases types during compilation, and this is just a way to guarantee that the imports of types are guaranteed to be erased as well. E.g. for `import { SomeType } from "backend-server"`, you don't want to include your entire server code in the frontend just because you imported the response type of some JSON API. `import type` neatly solves this issue, and it even enforces that you can't import anything but types from your backend. I also want to state that TS already removes regular `import`s that only import types. `import type` is mostly used to help bundlers. https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-8.html https://www.typescriptlang.org/docs/handbook/release-notes/t...

ericvsmith 3y ago

PEP 649 “Deferred Evaluation Of Annotations Using Descriptors” should result in vastly reducing the need for TYPE_CHECKING. https://peps.python.org/pep-0649/ https://peps.python.org/pep-0649/

durumu 3y ago
I think in practice that increases the need for TYPE_CHECKING. If you use deferred annotations, and you don't import the types you use, type checkers and IDEs will complain, and go to definition won't work. But if you import the types outside of a TYPE_CHECKING block it's wasteful, since you don't actually need the import for your code to run. So you need TYPE_CHECKING to satisfy your linter without impacting runtime performance.
guappa 3y ago
It will also break any runtime typecheck module that you might want to use.

spinningslate 3y ago

Small but important correction required in point 3 of the TL;DR: > Python doesn’t care about types at runtime This should say "Python doesn't care about type annotations at runtime." Python _does_ care about types at runtime - but it doesn't use type annotations to compute them. That doesn't detract from what's otherwise a really clear and helpful post. Python is living through a schizophrenic period in its evolution, and it's causing some problems. I hope it gets ironed out, though don't underestimate the difficulty. Python remains my go-to for many problems: a joy to use (for me) in many cases. But I definitely feel the need for static typing as codebase size increases. Having type annotations is good, but not being able to rely on them (i.e. static typing is not sound) adds to cognitive load and detracts from a confident development experience.

the_af 3y ago
I agree with the gist of your comment. I will add that, in my experience, having type annotations that are unreliable, cumbersome, with so many edge cases and requiring this kind of magic (e.g. "if TYPE_CHECKING", etc) which is beyond most "casual" Python users effectively means type annotations are a very hard sell for most non-hardcore team mates. I've been in a position to try improving processes and code quality of the Python codebase, and introducing mypy and type annotations has always been a very hard sell. The team just won't accept the benefits, given the difficulties in using it. If only Python made it easier and more reliable, but the current state is a mess. I come from the static typing world, so of course I see the benefits. But most/all of my team mates don't, and so they see this as some really odd hoops I want them to jump through, for minimal benefit (in their eyes).
- durumu 3y ago
  I think type annotations are extremely valuable even if you don't use mypy. On my team we basically use them as comments that are understood by the IDE, but aren't enforced. Of course comments can lie, but appeasing mypy really is a lot of work, and I would generally rather have a slightly dishonest but mostly correct type hint than some TypeVar monstrosity or, worse, Any.

mgaunard 3y ago

most of the Python code I've seen with type annotations appears to be significantly worse than the code without them. You're in Python. Either embrace it or use another language.

n3150n 3y ago
Could you expand on it? It is truly surprising for me that anyone would find code with type annotations to be significantly worse, for any reasons whatsoever. On the contrary, I joyfully read and write code with type annotations. It is obviously very useful knowing which object types a function expects and which it will return.
- mgaunard 3y ago
  verbosity tends to make code worse, not better.

kissgyorgy 3y ago

> However, Python doesn’t have the compile-time check Yes it does. All Python source code is parsed and compiled into bytecode. SyntaxError is raised before runtime.

gpderetta 3y ago
I think the author is obviously talking about compile time type checks here.
maweki 3y ago
It's arguable whether a SyntaxError happens as a compile-time check. It's true that it is raised before runtime. But a document that can be parsed according to the syntax definition is a necessary precondition to all checks of any static program property. Syntactic correctness in itself is not a static property of a program. A program is, by definition, syntactically correct. Without a correct syntax there is no way to assign meaning or execute or interpret, etc.