5 ms·

The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code. The reason we're stuck on mypy at work is be

The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code. The reason we're stuck on mypy at work is because it's the only type checker that has a plugin for Django that properly manages to type check its crazy runtime generated methods. I wish more python tooling took the TS approach of "what's in the wild IS the language", as opposed to a "we only typecheck the constructs we think you SHOULD be using".

mjr00 1y ago
> The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code. Or in this case, writing it in Rust... mypy is written in Python. People have forgotten that Python is really, really slow for CPU-intensive operations. Python's performance may not matter when you're writing web service code and the bottlenecks are database I/O and network calls, but for a tool that's loading up files, parsing into an AST, etc, it's no surprise that Rust/C/even Go would be an order of magnitude or two faster than Python. uv and ruff have been fantastic for me. ty is definitely not production ready (I see several bizarre issues on a test codebase, such as claiming `datetime.UTC` doesn't exist) but I trust that Astral will match the "crazy reality" of real Python (which I agree, is very crazy).
- _carljm 1y ago
  (ty developer here) Currently we default to our oldest supported Python version, in which `datetime.UTC` really doesn't exist! Use `--python-version 3.12` on the CLI, or add a `ty.toml` with e.g. ``` [environment] python-version = "3.12" ``` And we'll find `datetime.UTC`. We've discussed that this is probably the wrong default, and plan to change it.
  - mjr00 1y ago
    aha makes sense! Yeah it'd be nice if you could divine the intended python version from the uv configuration/`.python-version`. Thanks for all your hard work, looking forward to the full release!
  - miki123211 1y ago
    I realize this might be hard from a technical / architecture standpoint, but it would be great if "does not exist" and "does not exist in this version of Python" were two different errors. If I saw something like "datetime.UTC doesn't exist", I'd immediately think "wait, was that datetime.utc", not "ooh it got added in 3.11, I need to change my Python version"
    - _carljm 1y ago
      I agree that would be nice; probably not near the top of our list right now (and not trivial to implement), but it makes sense. Thanks for the suggestion.
      - jychang 1y ago
        Nontrivial way to do it is dynamically scan the python 3.12 namespace, and add these warnings. Is there any big downside to do it the boring way, hardcode a list and compare the error to the list?
        _carljm 1y ago
        This information is already maintained via `if sys.version_info >= (...):` conditionals in typeshed stubs. I don't think this is important enough to justify maintaining the same information in a duplicate way.
  - HelloNurse 1y ago
    Defaulting is wrong: what is checked is the aggregate of actual user code, standard library for a given Python version and installed packages. It has to be the same environment as when the program is run, leaving conservative approximations (checking types with the oldest supported library versions and hoping newer ones are OK) to the user.
    - dcreager 1y ago
      Yes, if you have a Python version specifed in pyproject.toml, for instance, we respect that, and that's what we use to type-check your code. The default being discussed here is what we fall back on if that project metadata isn't available.
      - 0xffff2 1y ago
        Could you check what version of `python` is in the PATH and use that as the default?
- dcreager 1y ago
  > such as claiming `datetime.UTC` doesn't exist) This is a known issue — we're currently defaulting to a conservative Python version, and `datetime.UTC` really doesn't exist until Python 3.11! https://docs.python.org/3/library/datetime.html#datetime.UTC https://docs.python.org/3/library/datetime.html#datetime.UTC We will probably change the default to "most recent supported Python version", but as mentioned elsewhere, this is very early and we're still working out these kinds of kinks!
  - zo1 1y ago
    You should be doing this dynamically based on the version of python you are running against, so that you don't have to hardcode or make such "conservative" choices by hand.
    - lacasito25 1y ago
      I think they probably know that, this is alpha software, no need to be condescending.
      - lionkor 1y ago
        They said they will default to some newer version, which indicates they are not planning to do this dynamically.
      - zarathustreal 1y ago
        Criticism isn’t necessarily condescending. “You should be doing x because y” is just a plain assertion, it doesn’t imply any moral judgement or opinion of the author.
      - johnisgood 1y ago
        How is it condescending in any way? I found it to be a constructive criticism; i.e. useful help.
        mahogany 1y ago
- deleted 1y ago
  [deleted]
- davidfstr 1y ago
  mypy is compiled using mypyc. It does not run as Python code.
  - mzl 1y ago
    The semantics of Python makes it problematic to run at speed, it is not just about interpreted vs compiled code. Give the high levels of dynamic behaviors that are allowed, a Jit (like pypy) has a higher chance of getting decent performance if the code has an underlying behavior that can be extracted.
    - Sinidir 1y ago
      mypy is also written in a style conducive to speed ups when compiling with mypyc
- miki123211 1y ago
  Python is slow for some CPU-intensive operations. There are some extremely CPU-intensive low-level operations that you can easily write in C and expose as a Python API, like what Numpy and Pandas do. You can then write really efficient algorithms in pure Python. As long as those low-level operations are fast, those Python-only algorithms will also be fast. I don't think this is necessarily "cheating" or "just calling disguised C functions." As an example, you can write an efficient linear regression algorithm with Numpy, even though there's nothing in Numpy that supports linear regression specifically, it's just one of the ways a Python programmer can arrange Numpy's low-level primitives. If you invent some new numerical algorithm to solve some esoteric problem in chemistry, you may be able to implement it efficiently in Python too, even if you're literally the first person ever writing it in any language. The actual problem is that it's hard for people to get an intuition of which Python operations can be made fast and which can't, AST and file manipulation are sadly in the latter group.
  - fastball 1y ago
    That is a confusing way to look at it. Python is slow, C is fast. If your python code is calling functions that were not written in Python (even if it is indirectly thru a library you are using), that is not "pure python".
  - francasso 1y ago
    That works in numerical libraries because you can encapsulate the loops into basic operations that you then lower to C. In a domain like type checking it's not nearly as easy/doable.
  - maleldil 1y ago
    > As long as those low-level operations are fast, those Python-only algorithms will also be fast. Only if you spend more time on the C implementations than on Python. If you have pure Python loops, you'll be slow. You need quite high-level components and minimal Python glue for it to be fast.
- shiandow 1y ago
  CPU intensive is not quite the right metric. What python is slow at is all the extra administration that comes with basic stuff like accessing attributes and function calls. This gives somewhat counterintuitive results where declaring and summing a whole list of integers in memory can be faster than a simple for loop with an iterator. But yeah writing stuff in a different (compiled) language is often better if that means the python interpreter doesn't need to go through as many steps.
johnfn 1y ago
In defense of mypy et al, Typescript had some of the greatest minds of our generation working for a decade+ on properly typing every insane form found in every random Javascript file. Microsoft has funded a team of great developers to hammer away at every obscure edge case imaginable. No other python checker can compare to the resources that TS had.
- smithkl42 1y ago
  And in the process, they ended up creating an extremely powerful type system that ~nobody outside of that original team can (fully) understand.
  - dathinab 1y ago
    IMHO they created type annotations, not a type system and how you use the type annotations to indicate a type system is inconsistent and incomplete (e.g. NoneType vs. None for inconsistency and a lot of mess related to mataclasses (e.g. Enum) and supporting type annotations for them for incomplete) the fact that even today something as fundamental as enums have issues with type checking _which are not just type checker incompetence_ is I think a good way to highlight what mess it is or that `Annotated[]` was only added in 3.9 and has a ton of visual overhead even through its essential for a lot of clean definitions in modern python code (where for backwards compatibility there is often some other way, which can be de-facto wrongly typed but shouldn't be type linted, have fun type checkers).
    - simonw 1y ago
      Parent comment was talking about TypeScript, not Python.
  - johnfn 1y ago
    How many people understand the intricacies of any complex language or type system? Though I think one of the great things about TS is that you need to understand none of it in order to do `npm install @types/lodash` and get all the benefits.
- 1y ago
HideousKojima 1y ago
>The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code. Nah, that's just part of the parade of excuses that comes out any time existing software solutions get smoked by a newcomer in performance, or when existing software gets more slow and bloated. Here's one of many examples: https://m.youtube.com/watch?v=GC-0tCy4P1U&pp=0gcJCdgAo7VqN5tD https://m.youtube.com/watch?v=GC-0tCy4P1U&pp=0gcJCdgAo7VqN5t...
- Redoubts 1y ago
  Sure, but so far this has been a true criticism of every python type checker that isnt mypy that is production ready today
- dathinab 1y ago
  the thing is most (all) of the type checkers including e.g. mypy _do not_ support most crazy python ... not because they don't want to or because it's to slow but because it's not really viable without fully executing module loading in a sandbox, which might seem viable until you realize that you still need to type check `__main__` modules etc. and that its a common trend in python to do configs by loading python modules and grabbing the module locals as keys of the config or loading some things might actually idk. initialize a GPU driver :sob: So it's kinda 100% guaranteed not possible to do fully correct type checking for all project :smh: But also python is one of the slowest popular languages (and with a large margin to any not also "one of slowest" languages). Only by moving hot code into C++/Rust is it fast, which often is good enough, but a type checker is exactly this kind of software where this approach stops working.
  - renmillar 1y ago
    Python's static checking capabilities could significantly improve both tracing and compilation efficiency. The language features that currently limit type checkers are likely the same ones making efficient compilation difficult. Perhaps we'll eventually see a Python 3.40 with complete JIT compilation, functioning similarly to Julia but retaining Python's extensive ecosystem that makes it essential in certain domains.
amelius 1y ago
> I wish more python tooling And not directly related, but I wish more python modules did proper checks with Valgrind before shipping.
- lyu07282 1y ago
  The CPython API is such a dumpster fire, even writing very simple modules the reference counting is very difficult to do correctly. The majority of python modules written in C are probably leaking memory somewhere but nobody knows.
  - amelius 1y ago
    My problem is that debugging a segfault in a Python system is impossible because of all the noise generated by Python modules that never bothered to clean up their Valgrind output.
    - badmintonbaseba 1y ago
      Some tricks that worked for me in the past: 1. use rr for debugging your binary wheel, you can set up watchpoints, and reverse step/continue from the segfault. 2. compile and run your wheel with sanitizers (ASAN, UBSAN). I rarely use valgrind so I can't comment on that.
TheTaytay 1y ago
Even Typescript is rewriting their compiler in Go. I think that the bottleneck is _actually_ the language sometimes. (And uv and ruff have basically proved that at this point)
- dathinab 1y ago
  through algorithmic improvements can also go a long way and if you are one of the first type checker which have to figure out the mess the python type annotation system is you will vast a lot of time on figuring that out instead of refactoring it's architecture to allow for algorithmic improvements which brings us to another python issue, python is quite bad at such huge refactoring even with type checkers but yeah python is by far the slowest widely used language, and for some use cases you can side step it by placing most hot code in C++/Rust extension modules, (or don't care because you are much much more network latency bound) but a type checker probably doesn't belong into that category
davedx 1y ago
Can mypy type check SQLAlchemy somehow? That's what caused me to give up on Python type checking recently
- lukaslalinsky 1y ago
  SQLAlchemy 2.x has direct support for mypy, it works out of the box, no longer needing mypy plugins. Many things in SQLAlchemy as are still dynamic and can't be type checked, but the native support works great where it can.
  - davedx 1y ago
    I’ll give that a try. Thanks
collinmanderson 1y ago
I also mentioned this up-thread, but https://pypi.org/project/django-types/ https://pypi.org/project/django-types/ is compatible with pyright without plugins, so it should theoretically work with ty. It's not quite as good as the mypy-django plugin but it still catches a lot.

mjr00 1y ago

> The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code. Or in this case, writing it in Rust... mypy is written in Python. People have forgotten that Python is really, really slow for CPU-intensive operations. Python's performance may not matter when you're writing web service code and the bottlenecks are database I/O and network calls, but for a tool that's loading up files, parsing into an AST, etc, it's no surprise that Rust/C/even Go would be an order of magnitude or two faster than Python. uv and ruff have been fantastic for me. ty is definitely not production ready (I see several bizarre issues on a test codebase, such as claiming `datetime.UTC` doesn't exist) but I trust that Astral will match the "crazy reality" of real Python (which I agree, is very crazy).

_carljm 1y ago
(ty developer here) Currently we default to our oldest supported Python version, in which `datetime.UTC` really doesn't exist! Use `--python-version 3.12` on the CLI, or add a `ty.toml` with e.g. ``` [environment] python-version = "3.12" ``` And we'll find `datetime.UTC`. We've discussed that this is probably the wrong default, and plan to change it.
- mjr00 1y ago
  aha makes sense! Yeah it'd be nice if you could divine the intended python version from the uv configuration/`.python-version`. Thanks for all your hard work, looking forward to the full release!
- miki123211 1y ago
  I realize this might be hard from a technical / architecture standpoint, but it would be great if "does not exist" and "does not exist in this version of Python" were two different errors. If I saw something like "datetime.UTC doesn't exist", I'd immediately think "wait, was that datetime.utc", not "ooh it got added in 3.11, I need to change my Python version"
  - _carljm 1y ago
    I agree that would be nice; probably not near the top of our list right now (and not trivial to implement), but it makes sense. Thanks for the suggestion.
    - jychang 1y ago
      Nontrivial way to do it is dynamically scan the python 3.12 namespace, and add these warnings. Is there any big downside to do it the boring way, hardcode a list and compare the error to the list?
      - _carljm 1y ago
        This information is already maintained via `if sys.version_info >= (...):` conditionals in typeshed stubs. I don't think this is important enough to justify maintaining the same information in a duplicate way.
- HelloNurse 1y ago
  Defaulting is wrong: what is checked is the aggregate of actual user code, standard library for a given Python version and installed packages. It has to be the same environment as when the program is run, leaving conservative approximations (checking types with the oldest supported library versions and hoping newer ones are OK) to the user.
  - dcreager 1y ago
    Yes, if you have a Python version specifed in pyproject.toml, for instance, we respect that, and that's what we use to type-check your code. The default being discussed here is what we fall back on if that project metadata isn't available.
    - 0xffff2 1y ago
      Could you check what version of `python` is in the PATH and use that as the default?
dcreager 1y ago
> such as claiming `datetime.UTC` doesn't exist) This is a known issue — we're currently defaulting to a conservative Python version, and `datetime.UTC` really doesn't exist until Python 3.11! https://docs.python.org/3/library/datetime.html#datetime.UTC https://docs.python.org/3/library/datetime.html#datetime.UTC We will probably change the default to "most recent supported Python version", but as mentioned elsewhere, this is very early and we're still working out these kinds of kinks!
- zo1 1y ago
  You should be doing this dynamically based on the version of python you are running against, so that you don't have to hardcode or make such "conservative" choices by hand.
  - lacasito25 1y ago
    I think they probably know that, this is alpha software, no need to be condescending.
    - lionkor 1y ago
      They said they will default to some newer version, which indicates they are not planning to do this dynamically.
    - zarathustreal 1y ago
      Criticism isn’t necessarily condescending. “You should be doing x because y” is just a plain assertion, it doesn’t imply any moral judgement or opinion of the author.
    - johnisgood 1y ago
      How is it condescending in any way? I found it to be a constructive criticism; i.e. useful help.
      - mahogany 1y ago
deleted 1y ago
[deleted]
davidfstr 1y ago
mypy is compiled using mypyc. It does not run as Python code.
- mzl 1y ago
  The semantics of Python makes it problematic to run at speed, it is not just about interpreted vs compiled code. Give the high levels of dynamic behaviors that are allowed, a Jit (like pypy) has a higher chance of getting decent performance if the code has an underlying behavior that can be extracted.
  - Sinidir 1y ago
    mypy is also written in a style conducive to speed ups when compiling with mypyc
miki123211 1y ago
Python is slow for some CPU-intensive operations. There are some extremely CPU-intensive low-level operations that you can easily write in C and expose as a Python API, like what Numpy and Pandas do. You can then write really efficient algorithms in pure Python. As long as those low-level operations are fast, those Python-only algorithms will also be fast. I don't think this is necessarily "cheating" or "just calling disguised C functions." As an example, you can write an efficient linear regression algorithm with Numpy, even though there's nothing in Numpy that supports linear regression specifically, it's just one of the ways a Python programmer can arrange Numpy's low-level primitives. If you invent some new numerical algorithm to solve some esoteric problem in chemistry, you may be able to implement it efficiently in Python too, even if you're literally the first person ever writing it in any language. The actual problem is that it's hard for people to get an intuition of which Python operations can be made fast and which can't, AST and file manipulation are sadly in the latter group.
- fastball 1y ago
  That is a confusing way to look at it. Python is slow, C is fast. If your python code is calling functions that were not written in Python (even if it is indirectly thru a library you are using), that is not "pure python".
- francasso 1y ago
  That works in numerical libraries because you can encapsulate the loops into basic operations that you then lower to C. In a domain like type checking it's not nearly as easy/doable.
- maleldil 1y ago
  > As long as those low-level operations are fast, those Python-only algorithms will also be fast. Only if you spend more time on the C implementations than on Python. If you have pure Python loops, you'll be slow. You need quite high-level components and minimal Python glue for it to be fast.
shiandow 1y ago
CPU intensive is not quite the right metric. What python is slow at is all the extra administration that comes with basic stuff like accessing attributes and function calls. This gives somewhat counterintuitive results where declaring and summing a whole list of integers in memory can be faster than a simple for loop with an iterator. But yeah writing stuff in a different (compiled) language is often better if that means the python interpreter doesn't need to go through as many steps.

johnfn 1y ago

In defense of mypy et al, Typescript had some of the greatest minds of our generation working for a decade+ on properly typing every insane form found in every random Javascript file. Microsoft has funded a team of great developers to hammer away at every obscure edge case imaginable. No other python checker can compare to the resources that TS had.

smithkl42 1y ago
And in the process, they ended up creating an extremely powerful type system that ~nobody outside of that original team can (fully) understand.
- dathinab 1y ago
  IMHO they created type annotations, not a type system and how you use the type annotations to indicate a type system is inconsistent and incomplete (e.g. NoneType vs. None for inconsistency and a lot of mess related to mataclasses (e.g. Enum) and supporting type annotations for them for incomplete) the fact that even today something as fundamental as enums have issues with type checking _which are not just type checker incompetence_ is I think a good way to highlight what mess it is or that `Annotated[]` was only added in 3.9 and has a ton of visual overhead even through its essential for a lot of clean definitions in modern python code (where for backwards compatibility there is often some other way, which can be de-facto wrongly typed but shouldn't be type linted, have fun type checkers).
  - simonw 1y ago
    Parent comment was talking about TypeScript, not Python.
- johnfn 1y ago
  How many people understand the intricacies of any complex language or type system? Though I think one of the great things about TS is that you need to understand none of it in order to do `npm install @types/lodash` and get all the benefits.
1y ago

HideousKojima 1y ago

>The way these type checkers get fast is usually by not supporting the crazy rich reality of realworld python code. Nah, that's just part of the parade of excuses that comes out any time existing software solutions get smoked by a newcomer in performance, or when existing software gets more slow and bloated. Here's one of many examples: https://m.youtube.com/watch?v=GC-0tCy4P1U&pp=0gcJCdgAo7VqN5tD https://m.youtube.com/watch?v=GC-0tCy4P1U&pp=0gcJCdgAo7VqN5t...

Redoubts 1y ago
Sure, but so far this has been a true criticism of every python type checker that isnt mypy that is production ready today
dathinab 1y ago
the thing is most (all) of the type checkers including e.g. mypy _do not_ support most crazy python ... not because they don't want to or because it's to slow but because it's not really viable without fully executing module loading in a sandbox, which might seem viable until you realize that you still need to type check `__main__` modules etc. and that its a common trend in python to do configs by loading python modules and grabbing the module locals as keys of the config or loading some things might actually idk. initialize a GPU driver :sob: So it's kinda 100% guaranteed not possible to do fully correct type checking for all project :smh: But also python is one of the slowest popular languages (and with a large margin to any not also "one of slowest" languages). Only by moving hot code into C++/Rust is it fast, which often is good enough, but a type checker is exactly this kind of software where this approach stops working.
- renmillar 1y ago
  Python's static checking capabilities could significantly improve both tracing and compilation efficiency. The language features that currently limit type checkers are likely the same ones making efficient compilation difficult. Perhaps we'll eventually see a Python 3.40 with complete JIT compilation, functioning similarly to Julia but retaining Python's extensive ecosystem that makes it essential in certain domains.

amelius 1y ago

> I wish more python tooling And not directly related, but I wish more python modules did proper checks with Valgrind before shipping.

lyu07282 1y ago
The CPython API is such a dumpster fire, even writing very simple modules the reference counting is very difficult to do correctly. The majority of python modules written in C are probably leaking memory somewhere but nobody knows.
- amelius 1y ago
  My problem is that debugging a segfault in a Python system is impossible because of all the noise generated by Python modules that never bothered to clean up their Valgrind output.
  - badmintonbaseba 1y ago
    Some tricks that worked for me in the past: 1. use rr for debugging your binary wheel, you can set up watchpoints, and reverse step/continue from the segfault. 2. compile and run your wheel with sanitizers (ASAN, UBSAN). I rarely use valgrind so I can't comment on that.

TheTaytay 1y ago

Even Typescript is rewriting their compiler in Go. I think that the bottleneck is _actually_ the language sometimes. (And uv and ruff have basically proved that at this point)

dathinab 1y ago
through algorithmic improvements can also go a long way and if you are one of the first type checker which have to figure out the mess the python type annotation system is you will vast a lot of time on figuring that out instead of refactoring it's architecture to allow for algorithmic improvements which brings us to another python issue, python is quite bad at such huge refactoring even with type checkers but yeah python is by far the slowest widely used language, and for some use cases you can side step it by placing most hot code in C++/Rust extension modules, (or don't care because you are much much more network latency bound) but a type checker probably doesn't belong into that category

davedx 1y ago

Can mypy type check SQLAlchemy somehow? That's what caused me to give up on Python type checking recently

lukaslalinsky 1y ago
SQLAlchemy 2.x has direct support for mypy, it works out of the box, no longer needing mypy plugins. Many things in SQLAlchemy as are still dynamic and can't be type checked, but the native support works great where it can.
- davedx 1y ago
  I’ll give that a try. Thanks

collinmanderson 1y ago

I also mentioned this up-thread, but https://pypi.org/project/django-types/ https://pypi.org/project/django-types/ is compatible with pyright without plugins, so it should theoretically work with ty. It's not quite as good as the mypy-django plugin but it still catches a lot.