9 ms·

:wave: Looks like you found the not-so-secret repository we're using to prepare for a broader announcement :) Please be aware this is pre-alpha software. The

by zanie 1y ago

:wave: Looks like you found the not-so-secret repository we're using to prepare for a broader announcement :) Please be aware this is pre-alpha software. The current version is 0.0.0a6 and the releases so far are all in service of validating our release process. We're excited to get this in people's hands, but want to set the expectation that we still have a lot of work left to do before this is production ready. Stay tuned for more for news in the near future! (... I work at Astral)

theLiminator 1y ago
Curious if this means it'll be released as a separate binary than ruff? I personally feel like having it within ruff is much nicer for ensuring that we have a consistent set of dependencies that play nicely with each other. Though I guess because a type checker doesn't mutate the files maybe that's not a real concern (vs formatting/linting with --fix).
- zanie 1y ago
  It'll be separate (at least to start) — we want to be able to iterate on it rapidly. Long-term, a consistent toolchain is definitely important and something we're thinking about.
- skwashd 1y ago
  +1 for (eventually) baking it ty into ruff. In my mind static type checking is a form of linting. For years I pushed black for formatting code. Once formatting was baked into ruff I ditched black. Having fewer dependencies to track and update simplifies my life and shortens my dependabot queue.
12_throw_away 1y ago
If you can say - are there any thoughts about implementing plugins / extension capabilities to keep type checking working even with libraries that aren't otherwise typecheckable? (where "not otherwise typecheckable" means types that can't be expressed with stubs - e.g., Django, dataclasses pre-PEP-681, pytest fixtures, etc.)
- dcreager 1y ago
  At least for the moment, we aren't planning on a plugin architecture. We do recognize that there are some popular libraries and code patterns that aren't easily (or at all) typeable with the current state of the typing spec. We feel it would be more useful to help drive changes to the typing spec where we can, so that other type checkers can also benefit; and/or implement workarounds for the most popular libraries directly in ty, so that a library author doesn't have to rely on their downstream consumers installing a particular set of plugins to get good type-checker results. (It's also more difficult to support plugins effectively in a type checker like ty, than in a linter like ruff, since any non-trivial use case would likely require deep changes to how we represent types and how we implement type inference. That's not something that lends itself to a couple of simple hook extension points.)
  - tyrion 1y ago
    Helping improve the spec and all is great, but being 100% honest, as a user, I would rather have a type checker I can bend to my needs. As you said, some code patterns in a dynamic language like Python are difficult, or even impossible, to type-check without custom code. Type checkers are becoming more popular than ever, and this implicitly means that these code patterns are are going to be discouraged. On one hand, I believe the dynamism of Python is core to the language. On the other, I would never want to write any collaborative piece of software without a type checker anymore. Therefore, to get the benefits of a type checker, I am occasionally forced to write worse code just to please it. Considering how fast uv and ruff took off, I am sure you are aware of the impact your project could have. I understand that supporting plugins is hard. However, if you are considering adding support for some popular libraries, IMHO, it would be really beneficial for the community if you could evaluate the feasibility of implementing things in a somewhat generic way, which could be then maybe leveraged by third-party authors. In any case, thanks for all the amazing work.
    - jez 1y ago
      Out of curiosity, do you have experience with other languages that have type system plugins that you’d hope be used as inspiration for something in Python? I don’t have any such experience (short of a macro system, which requires code generation or runtime support) and it always makes me curious when people ask for type system plugins whether this is a standard feature in a type system I’ve never used.
digdugdirk 1y ago
Cool! Out of curiosity, what's the bedrock that's used to determine what the fundamental python AST objects are? I'm wondering what the "single source of truth" is, if you will. Is this all based off a spec that python provides? If so, what does that look like? Or do you "recode" the python language in rust, then use rust features to parse the python files? Regardless of how it's done - This is a really fascinating project, and I'm really glad you guys are doing it!
- dcreager 1y ago
  There is a formal grammar defined in the CPython repo, implemented in a language called ASDL: https://github.com/python/cpython/blob/main/Parser/Python.asdl https://github.com/python/cpython/blob/main/Parser/Python.as... ty uses the same AST and parser as ruff. We don't use the ASDL grammar directly, because we store a few syntax nodes differently internally than how they're represented upstream. Our parser is hand-written in Rust. At first, our AST was also entirely hand-written, though we're moving in the direction of auto-generating more of it from a declarative grammar. https://github.com/astral-sh/ruff/issues/15655 https://github.com/astral-sh/ruff/issues/15655 https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_parser https://github.com/astral-sh/ruff/tree/main/crates/ruff_pyth... https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_ast/ast.toml https://github.com/astral-sh/ruff/blob/main/crates/ruff_pyth...
  - zanie 1y ago
    ditto! but we gave impressively non-overlapping answers
- zanie 1y ago
  As in, how are we parsing the Python code into an AST? CPython uses a generated parser. The grammar is defined in https://github.com/python/cpython/blob/main/Grammar/python.gram https://github.com/python/cpython/blob/main/Grammar/python.g... which is used to generate the specification at https://docs.python.org/3/reference/grammar.html#full-grammar-specification https://docs.python.org/3/reference/grammar.html#full-gramma... We use a hand-written parser, in Rust, based on the specification. We've written that previously at https://astral.sh/blog/ruff-v0.4.0#a-hand-written-parser https://astral.sh/blog/ruff-v0.4.0#a-hand-written-parser
BewareTheYiga 1y ago
For pre-alpha software it's working fantastic for my project. I thought I type annotated it well, but Ty had quite a lot of feedback for me. Great job and I can't wait until this is released.
- IshKebab 1y ago
  Had you checked it with Pyright previously?
  - BewareTheYiga 1y ago
    I've only used Pylance standard type checking in vscode. I have not used pyright as a stand alone package.
    - ddorian43 1y ago
      Use Pyright or basedpyright for now.
      - likeon 1y ago
        pylance is pyright
        taldo 1y ago
        pylance is pyright + proprietary stuff on top, see https://docs.basedpyright.com/latest/ https://docs.basedpyright.com/latest/ for a fork with some of pylance's features added.
ZiiS 1y ago
Pointlessmy anal; but 0.0.0a6 is very strongly indicative of the sixth alpha release. Pre-alpha are much better as .dev releases.
- usr9012809 1y ago
  > Pre-alpha are much better as .dev releases. No, they are correctly using semantic versioning to indicate pre-alpha releases. https://github.com/astral-sh/ty/releases https://github.com/astral-sh/ty/releases https://semver.org/ https://semver.org/
  - kstrauser 1y ago
    Python doesn’t use plain semver: https://peps.python.org/pep-0440/ https://peps.python.org/pep-0440/
    - zahlman 1y ago
      The reference Python implementation, written in C, doesn't use semver. But other projects in the Python ecosystem are generally assumed to unless stated otherwise. For example, Setuptools does (but not pip: https://pip.pypa.io/en/stable/development/release-process/ https://pip.pypa.io/en/stable/development/release-process/).
    - deleted 1y ago
      [deleted]
- fastball 1y ago
  It is the sixth alpha release. They haven't yet released a stable version – this is their sixth alpha release before that. What am I missing here?
  - ZiiS 1y ago
    I was replying to a post that said "Please be aware this is pre-alpha software." presumably trying to make a distinction from "alpha software".
davedx 1y ago
What might, possibly, redeem Python in my eyes as a potential language for making production applications (something that today, it is most certainly not) would be if the type checker worked across the broader ecosystem of common Python packages. For example, as my recent struggles showed, SQLAlchemy breaks `pyright` in all kinds of ways. Compared with how other 'dynamic' ORMs like Prisma interact with types, it's just a disaster and makes type checking applications that use it almost pointless. How does Ty play with SQLAlchemy?
- jakewins 1y ago
  My experience is this is nearly impossible, the solution is new packages written after typing was introduced. I don’t know about SQLAlchemy, but for libraries like pandas I just don’t see how it can be done, and so people are actively replacing them with modern typed alternatives
  - davedx 1y ago
    Ha. I just finished a huge rewrite at work from sync SQLAlchemy to async SQLAlchemy, because the async version uses a totally different API (core queries) to sync. So this implies if I want type checking I need to use a different ORM and start again? I love how Python makes me so much faster due to its dynamic nature! Move fast, break things!
    - mapcars 1y ago
      I don't agree that dynamic nature makes things necessarily faster, if you compare Python to C or Java it is true, but if you compare to Typescript it is not. With a decent typing system and a good editor that makes use of it (and AI-assistants nowadays) the prototyping can actually be both faster and more stable.
      - networked 1y ago
        I think davedx was being sarcastic. Python's dynamic nature cost them time.
opem 1y ago
Finally the missing puzzle piece from the astral toolchain is here! <3

by zanie 1y ago

theLiminator 1y ago

Curious if this means it'll be released as a separate binary than ruff? I personally feel like having it within ruff is much nicer for ensuring that we have a consistent set of dependencies that play nicely with each other. Though I guess because a type checker doesn't mutate the files maybe that's not a real concern (vs formatting/linting with --fix).

zanie 1y ago
It'll be separate (at least to start) — we want to be able to iterate on it rapidly. Long-term, a consistent toolchain is definitely important and something we're thinking about.
skwashd 1y ago
+1 for (eventually) baking it ty into ruff. In my mind static type checking is a form of linting. For years I pushed black for formatting code. Once formatting was baked into ruff I ditched black. Having fewer dependencies to track and update simplifies my life and shortens my dependabot queue.

12_throw_away 1y ago

If you can say - are there any thoughts about implementing plugins / extension capabilities to keep type checking working even with libraries that aren't otherwise typecheckable? (where "not otherwise typecheckable" means types that can't be expressed with stubs - e.g., Django, dataclasses pre-PEP-681, pytest fixtures, etc.)

dcreager 1y ago
At least for the moment, we aren't planning on a plugin architecture. We do recognize that there are some popular libraries and code patterns that aren't easily (or at all) typeable with the current state of the typing spec. We feel it would be more useful to help drive changes to the typing spec where we can, so that other type checkers can also benefit; and/or implement workarounds for the most popular libraries directly in ty, so that a library author doesn't have to rely on their downstream consumers installing a particular set of plugins to get good type-checker results. (It's also more difficult to support plugins effectively in a type checker like ty, than in a linter like ruff, since any non-trivial use case would likely require deep changes to how we represent types and how we implement type inference. That's not something that lends itself to a couple of simple hook extension points.)
- tyrion 1y ago
  Helping improve the spec and all is great, but being 100% honest, as a user, I would rather have a type checker I can bend to my needs. As you said, some code patterns in a dynamic language like Python are difficult, or even impossible, to type-check without custom code. Type checkers are becoming more popular than ever, and this implicitly means that these code patterns are are going to be discouraged. On one hand, I believe the dynamism of Python is core to the language. On the other, I would never want to write any collaborative piece of software without a type checker anymore. Therefore, to get the benefits of a type checker, I am occasionally forced to write worse code just to please it. Considering how fast uv and ruff took off, I am sure you are aware of the impact your project could have. I understand that supporting plugins is hard. However, if you are considering adding support for some popular libraries, IMHO, it would be really beneficial for the community if you could evaluate the feasibility of implementing things in a somewhat generic way, which could be then maybe leveraged by third-party authors. In any case, thanks for all the amazing work.
  - jez 1y ago
    Out of curiosity, do you have experience with other languages that have type system plugins that you’d hope be used as inspiration for something in Python? I don’t have any such experience (short of a macro system, which requires code generation or runtime support) and it always makes me curious when people ask for type system plugins whether this is a standard feature in a type system I’ve never used.

digdugdirk 1y ago

Cool! Out of curiosity, what's the bedrock that's used to determine what the fundamental python AST objects are? I'm wondering what the "single source of truth" is, if you will. Is this all based off a spec that python provides? If so, what does that look like? Or do you "recode" the python language in rust, then use rust features to parse the python files? Regardless of how it's done - This is a really fascinating project, and I'm really glad you guys are doing it!

dcreager 1y ago
There is a formal grammar defined in the CPython repo, implemented in a language called ASDL: https://github.com/python/cpython/blob/main/Parser/Python.asdl https://github.com/python/cpython/blob/main/Parser/Python.as... ty uses the same AST and parser as ruff. We don't use the ASDL grammar directly, because we store a few syntax nodes differently internally than how they're represented upstream. Our parser is hand-written in Rust. At first, our AST was also entirely hand-written, though we're moving in the direction of auto-generating more of it from a declarative grammar. https://github.com/astral-sh/ruff/issues/15655 https://github.com/astral-sh/ruff/issues/15655 https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_parser https://github.com/astral-sh/ruff/tree/main/crates/ruff_pyth... https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_ast/ast.toml https://github.com/astral-sh/ruff/blob/main/crates/ruff_pyth...
- zanie 1y ago
  ditto! but we gave impressively non-overlapping answers
zanie 1y ago
As in, how are we parsing the Python code into an AST? CPython uses a generated parser. The grammar is defined in https://github.com/python/cpython/blob/main/Grammar/python.gram https://github.com/python/cpython/blob/main/Grammar/python.g... which is used to generate the specification at https://docs.python.org/3/reference/grammar.html#full-grammar-specification https://docs.python.org/3/reference/grammar.html#full-gramma... We use a hand-written parser, in Rust, based on the specification. We've written that previously at https://astral.sh/blog/ruff-v0.4.0#a-hand-written-parser https://astral.sh/blog/ruff-v0.4.0#a-hand-written-parser

BewareTheYiga 1y ago

For pre-alpha software it's working fantastic for my project. I thought I type annotated it well, but Ty had quite a lot of feedback for me. Great job and I can't wait until this is released.

IshKebab 1y ago
Had you checked it with Pyright previously?
- BewareTheYiga 1y ago
  I've only used Pylance standard type checking in vscode. I have not used pyright as a stand alone package.
  - ddorian43 1y ago
    Use Pyright or basedpyright for now.
    - likeon 1y ago
      pylance is pyright
      - taldo 1y ago
        pylance is pyright + proprietary stuff on top, see https://docs.basedpyright.com/latest/ https://docs.basedpyright.com/latest/ for a fork with some of pylance's features added.

ZiiS 1y ago

Pointlessmy anal; but 0.0.0a6 is very strongly indicative of the sixth alpha release. Pre-alpha are much better as .dev releases.

usr9012809 1y ago
> Pre-alpha are much better as .dev releases. No, they are correctly using semantic versioning to indicate pre-alpha releases. https://github.com/astral-sh/ty/releases https://github.com/astral-sh/ty/releases https://semver.org/ https://semver.org/
- kstrauser 1y ago
  Python doesn’t use plain semver: https://peps.python.org/pep-0440/ https://peps.python.org/pep-0440/
  - zahlman 1y ago
    The reference Python implementation, written in C, doesn't use semver. But other projects in the Python ecosystem are generally assumed to unless stated otherwise. For example, Setuptools does (but not pip: https://pip.pypa.io/en/stable/development/release-process/ https://pip.pypa.io/en/stable/development/release-process/).
  - deleted 1y ago
    [deleted]
fastball 1y ago
It is the sixth alpha release. They haven't yet released a stable version – this is their sixth alpha release before that. What am I missing here?
- ZiiS 1y ago
  I was replying to a post that said "Please be aware this is pre-alpha software." presumably trying to make a distinction from "alpha software".

davedx 1y ago

What might, possibly, redeem Python in my eyes as a potential language for making production applications (something that today, it is most certainly not) would be if the type checker worked across the broader ecosystem of common Python packages. For example, as my recent struggles showed, SQLAlchemy breaks `pyright` in all kinds of ways. Compared with how other 'dynamic' ORMs like Prisma interact with types, it's just a disaster and makes type checking applications that use it almost pointless. How does Ty play with SQLAlchemy?

jakewins 1y ago
My experience is this is nearly impossible, the solution is new packages written after typing was introduced. I don’t know about SQLAlchemy, but for libraries like pandas I just don’t see how it can be done, and so people are actively replacing them with modern typed alternatives
- davedx 1y ago
  Ha. I just finished a huge rewrite at work from sync SQLAlchemy to async SQLAlchemy, because the async version uses a totally different API (core queries) to sync. So this implies if I want type checking I need to use a different ORM and start again? I love how Python makes me so much faster due to its dynamic nature! Move fast, break things!
  - mapcars 1y ago
    I don't agree that dynamic nature makes things necessarily faster, if you compare Python to C or Java it is true, but if you compare to Typescript it is not. With a decent typing system and a good editor that makes use of it (and AI-assistants nowadays) the prototyping can actually be both faster and more stable.
    - networked 1y ago
      I think davedx was being sarcastic. Python's dynamic nature cost them time.

opem 1y ago

Finally the missing puzzle piece from the astral toolchain is here! <3