7 ms·

Shape typing in Python

131 points by alexmolas 2y ago

jmduke 2y ago
Having migrated my application's Python and JS codebases to their typed siblings respectively last year, my 2c is that Python typing feels good and worthwhile when you're in the standard lib, but _awful_ (and net-negative) once you leave "normal Python" for the shores of third-party packages, particularly ones that lean heavily on duck typing (Django and BeautifulSoup both come to mind.) This is where some of the stuff in the TypeScript ecosystem really shines, IMHO — being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/ https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards.
- lemagedurage 2y ago
  That's fair. Though there's a wave of newer Python packages with great typing support. E.g. packages from https://github.com/encode https://github.com/encode
- amelius 2y ago
  Maybe we can use LLMs to automatically bring these third party libs up to par? Could be a nice showcase project for Copilot.
  - 12_throw_away 2y ago
    > Maybe we can use LLMs to automatically bring these third party libs up to par? So, I actually tried this. I tried to use copilot to help generate type stubs for a third party library, hoping to be pleasantly surprised. Copilot generated reasonable-looking type stubs that were not close enough to correct to be of any value. Even with the full source code in context, it failed to "reason" correctly about any of the hard stuff (unions, generics, overloads, variadics, quasi-structured mappings, weird internal proxy types, state-dependent responses, etc. etc.). In my experience, bolting types onto a duck-typed API always produces somewhat kludgy results that won't be as nice as a system designed around static typing. So _of course_ an LLM can't solve that problem any more than adding type stubs can. But really, the answer to "will LLMs fix $hard_problem for us?" is almost always "no", because $hard_problem can rarely be solved by just writing some code.
    - golergka 2y ago
      What about GPT-4? I think that coupled with a decent RAG and agent systems, it would do a good job.
      - 12_throw_away 2y ago
        Ok, how does one set up "GPT-4 coupled with a decent RAG and agent systems"?
- vforgione 2y ago
  While I don’t disagree, there are a number of additional stub libraries you can install that provide typing for libraries that don’t already have them. I personally find [django-types](https://pypi.org/project/django-types/ https://pypi.org/project/django-types/) to be really well constructed and useful.
- jddj 2y ago
  If I remember correctly, Typescript felt the same way for quite a long time
  - nurettin 2y ago
    It did, especially in the late 2013 and early 2014s. But then the type repositories quickly caught up. Python package authors usually shy away from such endeavours, especially those who use kwargs in order to configure large classes. pygann comes to mind.
    - maleldil 2y ago
      Python has also seen a sizable movement towards using types as part of the design, such as Typer, FastAPI and Pydantic. Existing mainstream libraries are also slowly adopting types, such as pandas and numpy (including `numpy.typing`). For the latter cases, it's not easy because typed APIs require different principles than dynamic/duck-typed ones. Still, I think it's safe to say that the community is trending towards more typing over time, especially greenfield projects. Personally, all my new projects are 100% typed, with type-safe wrappers around untyped libraries. For what it's worth, since Python 3.12 (or with typing_extensions for earlier versions), it's also possible to use Unpack and TypedDict to type kwargs.
    - nerdponx 2y ago
      What is pygann? It's not on PyPI.
      - nurettin 2y ago
        it was pygad
    - int_19h 2y ago
      Python was always much more dynamically typed than JS, and it also became the prevailing approach in the ecosystem.
- eloisius 2y ago
  I agree. Prior to the introduction of types in Python, I thought I wanted it. Now I hate them. It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me. Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1. You see extraneous code like x: int = 1 in Python now. Third party libs have bonkers types. This function signature is ridiculous: sqlalchemy.orm.relationship(argument: _RelationshipArgumentType[Any] | None = None, secondary: _RelationshipSecondaryArgument | None = None, *, uselist: bool | None = None, collection_class: Type[Collection[Any]] | Callable[[], Collection[Any]] | None = None, primaryjoin: _RelationshipJoinConditionArgument | None = None, secondaryjoin: _RelationshipJoinConditionArgument | None = None, back_populates: str | None = None, order_by: _ORMOrderByArgument = False, backref: ORMBackrefArgument | None = None, overlaps: str | None = None, post_update: bool = False, cascade: str = 'save-update, merge', viewonly: bool = False, init: _NoArg | bool = _NoArg.NO_ARG, repr: _NoArg | bool = _NoArg.NO_ARG, default: _NoArg | _T = _NoArg.NO_ARG, default_factory: _NoArg | Callable[[], _T] = _NoArg.NO_ARG, compare: _NoArg | bool = _NoArg.NO_ARG, kw_only: _NoArg | bool = _NoArg.NO_ARG, lazy: _LazyLoadArgumentType = 'select', passive_deletes: Literal['all'] | bool = False, passive_updates: bool = True, active_history: bool = False, enable_typechecks: bool = True, foreign_keys: _ORMColCollectionArgument | None = None, remote_side: _ORMColCollectionArgument | None = None, join_depth: int | None = None, comparator_factory: Type[RelationshipProperty.Comparator[Any]] | None = None, single_parent: bool = False, innerjoin: bool = False, distinct_target_key: bool | None = None, load_on_pending: bool = False, query_class: Type[Query[Any]] | None = None, info: _InfoType | None = None, omit_join: Literal[None, False] = None, sync_backref: bool | None = None, **kw: Any) → Relationship[Any] https://docs.sqlalchemy.org/en/20/orm/relationship_api.html#sqlalchemy.orm.relationship https://docs.sqlalchemy.org/en/20/orm/relationship_api.html#...
  - yrro 2y ago
    I wouldn't mind all of that if the SQLAlchemy documentation would hide all the types until I mouse over them. Ditto for vim!
  - lolinder 2y ago
    > It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me. Maybe try using an IDE? Without one any language's type system will feel more frustrating than it's worth, since you won't get inline error messages either. > Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1. This is called type inference, and as far as I can tell this level of basic type inference is supported by the major python type checkers. If you're seeing people explicitly annotate types on local variables that's a cultural problem with people who are unaccustomed to using types. As for that function signature, it would be bonkers with or without types. The types themselves look pretty straightforward, the problem is just that they formatted it all on one line and have a ridiculous number of keyword arguments.
- DanielVZ 2y ago
  I think you are missing stub libraries. After installing them its been a breeze
- traverseda 2y ago
  > being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/ https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards. Alright, but there's nothing stopping you from having a completely typesafe ORM in python, is there? Sure, there's isn't really one that everyone uses yet, but the python community tends to be a bit more cautious and slower to adopt big changes like that.
  - jmduke 2y ago
    I'm talking about practical limitations, not academic ones. You're not incorrect (and libraries like FastAPI and Pydantic make me confident that the benefits of type-safety will grow throughout the ecosystem) but I am talking about from the perspective of someone considering whether or not to adopt typing within their Python project today.
    - ghnws 2y ago
      What harm do you think typing a function, for example, would do? I'm genuinely qurious because I just can't see where the issue is.
- networked 2y ago
  My experience has been different: last year I started writing Python again after a long break, and I am yet to regret using types pervasively. If some library has no type definitions, I prefer to have my typed code interact with its untyped code. It is still better than having no types at all. You can sometimes get some useful type safety by annotating your functions with the untyped library's classes. Since then, I have used established libraries like Beautiful Soup, Jinja, Pillow, platformdirs, psutil, python-dateutil, redis-py, and xmltodict with either official or third-party types. I remember their types being useful to varying degrees and not a problem. I have replaced Requests with the very similar but typed and optionally async HTTPX. My most objectionable experience with types in Python so far has been having to write root = cast( lxml.etree._Element, # noqa: SLF001 html5.parse(html, return_root=True), ) when I used types-lxml with https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser. In return I have been able to catch some bugs early and to "fearlessly"refactor code with few or no unit tests, only integration tests. The style I have arrived at is close to https://kobzol.github.io/rust/python/2023/05/20/writing-python-like-its-rust.html https://kobzol.github.io/rust/python/2023/05/20/writing-pyth.... Admittedly, I don't use Django. Maybe I won't like typed Django if I do. My choice of type checker is Pyright in non-strict mode. It seems to usually, though not always, catch more and more subtle type errors than mypy. I understand that for Django, mypy with a Django plugin is preferred.
  - andenacitelli 2y ago
    You can also use something like stubgen to generate function definition signatures for dependencies for mypy to validate, then make your own changes to those files with better types if you wish. I don’t think it’s very scalable, and having the library itself or a stubs package come with types is the only “good”-feeling route, but you at least have a somewhat decent path to still getting it decent without any intervention on the library’s part. It may even be sufficient, if (like in most situations) you only use a few functions from a library (which may in turn call others, but you only care about the ones your code directly touches), and therefore only need to type those ones.
- technocratius 2y ago
  Which language do you refer to when you speak of python's typed sibling?
  - js2 2y ago
    I take jmduke to mean Python with type annotations.
- eternityforest 2y ago
  Using types properly is always annoying. Seeing MyPy report no problems makes it worth it. I find myself doing a lot of isinstance() and raise TypeError, but that's still a huge win, protecting everything after I've asserted the duck type is what it should be. I also use beartype for runtime protection. Typescript is pretty amazing though. I really like how integrated the ecosystem is.
naddeo 2y ago
I was surprised to see the example in the blog. Python actually has come pretty far with types but the blog's example doesn't really highlight it. For structural static typing, something like this is nicer as an example from typing import Protocol, Tuple, TypedDict class Foo(TypedDict): foo: str bar: int baz: Tuple[str, int] baaz: Tuple[float, ...] class Functionality(Protocol): def do(self, it: Foo): ... class MyFunctionality: # not explicitly implemented def do(self, it: Foo): ... class DoIt: def execute(self, it: Foo, func: Functionality): ... doit = DoIt().execute({ # Type checks "foo": "foo", "bar": 7, "baz": ("str", 2), "baaz": (1.0, 2.0)}, MyFunctionality()) Protocols and TypedDicts let you do nice structural stuff, similar typescript (though not as feature complete). Types are good enough on python that I would never consider a project without them, and I work with pandas and numpy a lot. You change your workflow a little bit so that you end up quarantining the code that interfaces with third-party libraries that don't have good type support behind your own functions that do. There are other pretty cool type things as well. Python is definitely in much better shape than it was. Combine all of that with Pyright's ability to do more advanced type inference and its like a whole new language experience.
- bormaj 2y ago
  I would supplement this by suggesting `pydantic` models instead of `TypedDict`s. This library has become a core utility for me as it greatly improves the developer experience with typing/validation/serialization support.
  - maleldil 2y ago
    They fulfil different roles. pydantic models would be an alternative to dataclasses, attrs or data-centric classes in general. TypedDict is used when you're stuck with dicts and can't convert them to a specific class.
    - catlover76 2y ago
      [dead]
lmeyerov 2y ago
Where I really want this is pandas. The community has been smoothing the basic typing story over the last couple of years, which helps with deprecations & basic API misuses. However, I'm excited for shape/dependent typing over dataframe column names, as that would get more into our typical case of data & logic errors.
- ninja3925 2y ago
  Column misnaming/typo is indeed a problem in pandas. I think a powerful IDE could do the trick though.
  - lmeyerov 2y ago
    Sort of... the IDE would want the mypy (or otherwise) typings to surface that. Internally, the dataframe library should make it easier for the IDE to see that, vs today's norm of tracking just "Any" / "Index" / "Series" / ... .
- sampo 2y ago
  I wish timezone-naive and timezone-aware Timestamps would be different types.
  - Hasnep 2y ago
    See this post for a comparison of Python datetime libraries. The datetype, heiclockter and whenever libraries have different types for them. https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls/ https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls...
- thenobsta 2y ago
  Pandera helps with some of this. Check it out -- https://pandera.readthedocs.io/en/stable/ https://pandera.readthedocs.io/en/stable/ We've used it to great effect.
  - lmeyerov 2y ago
deleted 2y ago
[deleted]
davnn 2y ago
There are libraries that support shape checking and I‘ve written a package to combine beartype’s runtime typechecks with jaxtyping‘s shapechecks, see https://github.com/davnn/safecheck https://github.com/davnn/safecheck Once accustomed to shape checking it‘s quite a boost in productivity for us, no more fiddling around with invalid dimensions.
LegNeato 2y ago
You might be interested in `dfdx`, shape checked deep learning in rust: https://github.com/coreylowman/dfdx https://github.com/coreylowman/dfdx.
arjvik 2y ago
I tried to do this but got bitten by the fact that Python didn't support varardic type arguments... somehow never thought to just pass the arguments as a tuple! I'll try and put together a Numpy/JAX wrapper for this, because I've been looking for something that does compile-time shape checking properly for a long time! Still wish Python's type system was as powerful as Typescript's... it's got potential that it just doesn't live up to.
swingingFlyFish 2y ago
I've been using Python since 2.x and I have to admit which each new release I toy with leaving this language behind. Type hints are the latest warts in what should have been a beautiful language. I tend to agree with Charles Lesleifer and others here. Python3 is a mess. https://charlesleifer.com/blog/what-happened/ https://charlesleifer.com/blog/what-happened/
- walthamstow 2y ago
  That blog post is all very sensible until the drive-by on f-strings at the end, I can't begin to understand the rationale for hating them
xrd 2y ago
I'm a bit new to Python but using it a lot the last few weeks. People here are suggesting that without an ide typing in Python doesn't make sense. I'm finding that as an emacs user this feels true. Is anyone using emacs primarily and if so, do you have suggestions on what to do to benefit from Python typing? Also, I've been struggling with python in the repl. Is there a way to see types in a more dynamic way there? Obviously autocomplete works but I wish a function call would suggest the type. I assume I'm not using things correctly.

131 points by alexmolas 2y ago

Having migrated my application's Python and JS codebases to their typed siblings respectively last year, my 2c is that Python typing feels good and worthwhile when you're in the standard lib, but _awful_ (and net-negative) once you leave "normal Python" for the shores of third-party packages, particularly ones that lean heavily on duck typing (Django and BeautifulSoup both come to mind.) This is where some of the stuff in the TypeScript ecosystem really shines, IMHO — being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/ https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards.

lemagedurage 2y ago
That's fair. Though there's a wave of newer Python packages with great typing support. E.g. packages from https://github.com/encode https://github.com/encode
amelius 2y ago
Maybe we can use LLMs to automatically bring these third party libs up to par? Could be a nice showcase project for Copilot.
- 12_throw_away 2y ago
  > Maybe we can use LLMs to automatically bring these third party libs up to par? So, I actually tried this. I tried to use copilot to help generate type stubs for a third party library, hoping to be pleasantly surprised. Copilot generated reasonable-looking type stubs that were not close enough to correct to be of any value. Even with the full source code in context, it failed to "reason" correctly about any of the hard stuff (unions, generics, overloads, variadics, quasi-structured mappings, weird internal proxy types, state-dependent responses, etc. etc.). In my experience, bolting types onto a duck-typed API always produces somewhat kludgy results that won't be as nice as a system designed around static typing. So _of course_ an LLM can't solve that problem any more than adding type stubs can. But really, the answer to "will LLMs fix $hard_problem for us?" is almost always "no", because $hard_problem can rarely be solved by just writing some code.
  - golergka 2y ago
    What about GPT-4? I think that coupled with a decent RAG and agent systems, it would do a good job.
    - 12_throw_away 2y ago
      Ok, how does one set up "GPT-4 coupled with a decent RAG and agent systems"?
vforgione 2y ago
While I don’t disagree, there are a number of additional stub libraries you can install that provide typing for libraries that don’t already have them. I personally find [django-types](https://pypi.org/project/django-types/ https://pypi.org/project/django-types/) to be really well constructed and useful.
jddj 2y ago
If I remember correctly, Typescript felt the same way for quite a long time
- nurettin 2y ago
  It did, especially in the late 2013 and early 2014s. But then the type repositories quickly caught up. Python package authors usually shy away from such endeavours, especially those who use kwargs in order to configure large classes. pygann comes to mind.
  - maleldil 2y ago
    Python has also seen a sizable movement towards using types as part of the design, such as Typer, FastAPI and Pydantic. Existing mainstream libraries are also slowly adopting types, such as pandas and numpy (including `numpy.typing`). For the latter cases, it's not easy because typed APIs require different principles than dynamic/duck-typed ones. Still, I think it's safe to say that the community is trending towards more typing over time, especially greenfield projects. Personally, all my new projects are 100% typed, with type-safe wrappers around untyped libraries. For what it's worth, since Python 3.12 (or with typing_extensions for earlier versions), it's also possible to use Unpack and TypedDict to type kwargs.
  - nerdponx 2y ago
    What is pygann? It's not on PyPI.
    - nurettin 2y ago
      it was pygad
  - int_19h 2y ago
    Python was always much more dynamically typed than JS, and it also became the prevailing approach in the ecosystem.
eloisius 2y ago
I agree. Prior to the introduction of types in Python, I thought I wanted it. Now I hate them. It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me. Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1. You see extraneous code like x: int = 1 in Python now. Third party libs have bonkers types. This function signature is ridiculous: sqlalchemy.orm.relationship(argument: _RelationshipArgumentType[Any] | None = None, secondary: _RelationshipSecondaryArgument | None = None, *, uselist: bool | None = None, collection_class: Type[Collection[Any]] | Callable[[], Collection[Any]] | None = None, primaryjoin: _RelationshipJoinConditionArgument | None = None, secondaryjoin: _RelationshipJoinConditionArgument | None = None, back_populates: str | None = None, order_by: _ORMOrderByArgument = False, backref: ORMBackrefArgument | None = None, overlaps: str | None = None, post_update: bool = False, cascade: str = 'save-update, merge', viewonly: bool = False, init: _NoArg | bool = _NoArg.NO_ARG, repr: _NoArg | bool = _NoArg.NO_ARG, default: _NoArg | _T = _NoArg.NO_ARG, default_factory: _NoArg | Callable[[], _T] = _NoArg.NO_ARG, compare: _NoArg | bool = _NoArg.NO_ARG, kw_only: _NoArg | bool = _NoArg.NO_ARG, lazy: _LazyLoadArgumentType = 'select', passive_deletes: Literal['all'] | bool = False, passive_updates: bool = True, active_history: bool = False, enable_typechecks: bool = True, foreign_keys: _ORMColCollectionArgument | None = None, remote_side: _ORMColCollectionArgument | None = None, join_depth: int | None = None, comparator_factory: Type[RelationshipProperty.Comparator[Any]] | None = None, single_parent: bool = False, innerjoin: bool = False, distinct_target_key: bool | None = None, load_on_pending: bool = False, query_class: Type[Query[Any]] | None = None, info: _InfoType | None = None, omit_join: Literal[None, False] = None, sync_backref: bool | None = None, **kw: Any) → Relationship[Any] https://docs.sqlalchemy.org/en/20/orm/relationship_api.html#sqlalchemy.orm.relationship https://docs.sqlalchemy.org/en/20/orm/relationship_api.html#...
- yrro 2y ago
  I wouldn't mind all of that if the SQLAlchemy documentation would hide all the types until I mouse over them. Ditto for vim!
- lolinder 2y ago
  > It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me. Maybe try using an IDE? Without one any language's type system will feel more frustrating than it's worth, since you won't get inline error messages either. > Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1. This is called type inference, and as far as I can tell this level of basic type inference is supported by the major python type checkers. If you're seeing people explicitly annotate types on local variables that's a cultural problem with people who are unaccustomed to using types. As for that function signature, it would be bonkers with or without types. The types themselves look pretty straightforward, the problem is just that they formatted it all on one line and have a ridiculous number of keyword arguments.
DanielVZ 2y ago
I think you are missing stub libraries. After installing them its been a breeze
traverseda 2y ago
> being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/ https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards. Alright, but there's nothing stopping you from having a completely typesafe ORM in python, is there? Sure, there's isn't really one that everyone uses yet, but the python community tends to be a bit more cautious and slower to adopt big changes like that.
- jmduke 2y ago
  I'm talking about practical limitations, not academic ones. You're not incorrect (and libraries like FastAPI and Pydantic make me confident that the benefits of type-safety will grow throughout the ecosystem) but I am talking about from the perspective of someone considering whether or not to adopt typing within their Python project today.
  - ghnws 2y ago
    What harm do you think typing a function, for example, would do? I'm genuinely qurious because I just can't see where the issue is.
networked 2y ago
My experience has been different: last year I started writing Python again after a long break, and I am yet to regret using types pervasively. If some library has no type definitions, I prefer to have my typed code interact with its untyped code. It is still better than having no types at all. You can sometimes get some useful type safety by annotating your functions with the untyped library's classes. Since then, I have used established libraries like Beautiful Soup, Jinja, Pillow, platformdirs, psutil, python-dateutil, redis-py, and xmltodict with either official or third-party types. I remember their types being useful to varying degrees and not a problem. I have replaced Requests with the very similar but typed and optionally async HTTPX. My most objectionable experience with types in Python so far has been having to write root = cast( lxml.etree._Element, # noqa: SLF001 html5.parse(html, return_root=True), ) when I used types-lxml with https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser. In return I have been able to catch some bugs early and to "fearlessly"refactor code with few or no unit tests, only integration tests. The style I have arrived at is close to https://kobzol.github.io/rust/python/2023/05/20/writing-python-like-its-rust.html https://kobzol.github.io/rust/python/2023/05/20/writing-pyth.... Admittedly, I don't use Django. Maybe I won't like typed Django if I do. My choice of type checker is Pyright in non-strict mode. It seems to usually, though not always, catch more and more subtle type errors than mypy. I understand that for Django, mypy with a Django plugin is preferred.
- andenacitelli 2y ago
  You can also use something like stubgen to generate function definition signatures for dependencies for mypy to validate, then make your own changes to those files with better types if you wish. I don’t think it’s very scalable, and having the library itself or a stubs package come with types is the only “good”-feeling route, but you at least have a somewhat decent path to still getting it decent without any intervention on the library’s part. It may even be sufficient, if (like in most situations) you only use a few functions from a library (which may in turn call others, but you only care about the ones your code directly touches), and therefore only need to type those ones.
technocratius 2y ago
Which language do you refer to when you speak of python's typed sibling?
- js2 2y ago
  I take jmduke to mean Python with type annotations.
eternityforest 2y ago
Using types properly is always annoying. Seeing MyPy report no problems makes it worth it. I find myself doing a lot of isinstance() and raise TypeError, but that's still a huge win, protecting everything after I've asserted the duck type is what it should be. I also use beartype for runtime protection. Typescript is pretty amazing though. I really like how integrated the ecosystem is.

naddeo 2y ago

I was surprised to see the example in the blog. Python actually has come pretty far with types but the blog's example doesn't really highlight it. For structural static typing, something like this is nicer as an example from typing import Protocol, Tuple, TypedDict class Foo(TypedDict): foo: str bar: int baz: Tuple[str, int] baaz: Tuple[float, ...] class Functionality(Protocol): def do(self, it: Foo): ... class MyFunctionality: # not explicitly implemented def do(self, it: Foo): ... class DoIt: def execute(self, it: Foo, func: Functionality): ... doit = DoIt().execute({ # Type checks "foo": "foo", "bar": 7, "baz": ("str", 2), "baaz": (1.0, 2.0)}, MyFunctionality()) Protocols and TypedDicts let you do nice structural stuff, similar typescript (though not as feature complete). Types are good enough on python that I would never consider a project without them, and I work with pandas and numpy a lot. You change your workflow a little bit so that you end up quarantining the code that interfaces with third-party libraries that don't have good type support behind your own functions that do. There are other pretty cool type things as well. Python is definitely in much better shape than it was. Combine all of that with Pyright's ability to do more advanced type inference and its like a whole new language experience.

bormaj 2y ago
I would supplement this by suggesting `pydantic` models instead of `TypedDict`s. This library has become a core utility for me as it greatly improves the developer experience with typing/validation/serialization support.
- maleldil 2y ago
  They fulfil different roles. pydantic models would be an alternative to dataclasses, attrs or data-centric classes in general. TypedDict is used when you're stuck with dicts and can't convert them to a specific class.
  - catlover76 2y ago
    [dead]

lmeyerov 2y ago

Where I really want this is pandas. The community has been smoothing the basic typing story over the last couple of years, which helps with deprecations & basic API misuses. However, I'm excited for shape/dependent typing over dataframe column names, as that would get more into our typical case of data & logic errors.

ninja3925 2y ago
Column misnaming/typo is indeed a problem in pandas. I think a powerful IDE could do the trick though.
- lmeyerov 2y ago
  Sort of... the IDE would want the mypy (or otherwise) typings to surface that. Internally, the dataframe library should make it easier for the IDE to see that, vs today's norm of tracking just "Any" / "Index" / "Series" / ... .
sampo 2y ago
I wish timezone-naive and timezone-aware Timestamps would be different types.
- Hasnep 2y ago
  See this post for a comparison of Python datetime libraries. The datetype, heiclockter and whenever libraries have different types for them. https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls/ https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls...
thenobsta 2y ago
Pandera helps with some of this. Check it out -- https://pandera.readthedocs.io/en/stable/ https://pandera.readthedocs.io/en/stable/ We've used it to great effect.
- lmeyerov 2y ago

deleted 2y ago

[deleted]

davnn 2y ago

There are libraries that support shape checking and I‘ve written a package to combine beartype’s runtime typechecks with jaxtyping‘s shapechecks, see https://github.com/davnn/safecheck https://github.com/davnn/safecheck Once accustomed to shape checking it‘s quite a boost in productivity for us, no more fiddling around with invalid dimensions.

LegNeato 2y ago

You might be interested in `dfdx`, shape checked deep learning in rust: https://github.com/coreylowman/dfdx https://github.com/coreylowman/dfdx.

arjvik 2y ago

I tried to do this but got bitten by the fact that Python didn't support varardic type arguments... somehow never thought to just pass the arguments as a tuple! I'll try and put together a Numpy/JAX wrapper for this, because I've been looking for something that does compile-time shape checking properly for a long time! Still wish Python's type system was as powerful as Typescript's... it's got potential that it just doesn't live up to.

swingingFlyFish 2y ago

I've been using Python since 2.x and I have to admit which each new release I toy with leaving this language behind. Type hints are the latest warts in what should have been a beautiful language. I tend to agree with Charles Lesleifer and others here. Python3 is a mess. https://charlesleifer.com/blog/what-happened/ https://charlesleifer.com/blog/what-happened/

walthamstow 2y ago
That blog post is all very sensible until the drive-by on f-strings at the end, I can't begin to understand the rationale for hating them

xrd 2y ago

I'm a bit new to Python but using it a lot the last few weeks. People here are suggesting that without an ide typing in Python doesn't make sense. I'm finding that as an emacs user this feels true. Is anyone using emacs primarily and if so, do you have suggestions on what to do to benefit from Python typing? Also, I've been struggling with python in the repl. Is there a way to see types in a more dynamic way there? Obviously autocomplete works but I wish a function call would suggest the type. I assume I'm not using things correctly.