7 ms·

The Python type system is pretty bad, but it's still 100x better than not using types. We are heavy users of the (Rust) type system at Svix, and it's been a god

by tasn 2y ago

The Python type system is pretty bad, but it's still 100x better than not using types. We are heavy users of the (Rust) type system at Svix, and it's been a godsend. I wrote about it here https://www.svix.com/blog/strong-typing-hill-to-die-on/ https://www.svix.com/blog/strong-typing-hill-to-die-on/ We also use Python in some places, including the shitty Python type-system (and some cool hackery to make SQLAlchemy feel very typed and work nicely with Pydantic).

youdontknowjuli 2y ago
> (and some cool hackery to make SQLAlchemy feel very typed and work nicely with Pydantic). Sounds interesting. Can you elaborate on the cool hackery? We introduced SQLModel recently but struggle in a few cases (e.g. multi-level joins). Do you know reference projects for SQLAlchemy and pydantic?
- tasn 2y ago
  My info is maybe a bit dated, as it's been a while since we wrote this hackery. We also adopted SQLModel at some point but we had to patch it to work well (I think some of my contributions are now in upstream). As for some of the hacks: def c(prop: t.Any) -> sa.Column: # type: ignore return prop To make it possible to access sqlmodel properties as columns for doing things like `in_` but still maintaining type safety. Added types ourselves to the base model like this: __table__: t.ClassVar[sa.Table] Added functions that help with typing like this: @classmethod async def _fetch_one(cls: t.Type[BaseT], db: BaseReadOnlySqlSession, query: Select) -> t.Optional[BaseT]: try: return (await db.execute(query)).scalar_one() except NoResultFound: return None and stuff like this for relationships: def ezrelationship( model: t.Type[T_], id_our: t.Union[str, sa.Column], # type: ignore id_other: t.Optional[t.Union[t.Any, sa.Column]] = None, # type: ignore ) -> T_: if id_other is None: id_other = model.id return sqlm.Relationship(sa_relationship=relationship(model, primaryjoin=f"foreign({id_our}) == {id_other}")) def ezrelationship_back( id_our: t.Union[str, sa.Column], # type: ignore id_other: t.Union[str, sa.Column], # type: ignore ) -> t.Any: model, only_id2 = id_other.split(".") return sqlm.Relationship( sa_relationship=relationship( model, primaryjoin=f"foreign({id_our}) == {id_other}_id", back_populates=only_id2, ) ) I hope this helps, I don't have time to find all the stuff, but we also hacked on SQLAlchemy a bit, and in other places.
Myrmornis 2y ago
Can you give some examples of how the Python type system is disappointing you?
- colemannerd 2y ago
  default values! Since type hints are *hints*, it is difficult to set default values for complicated types. For instance, if you have lists, dicts, sets in the type signature, without a library like pydantic, it is difficult and non-standard. This becomes even more problematic when you start doing more complicated data structures. The configuration in this library starts to show the problems. https://koxudaxi.github.io/datamodel-code-generator/custom_template/ https://koxudaxi.github.io/datamodel-code-generator/custom_t... The issue very much is a lack of a standard for the entire language; rather than it not being possible.
  - alfons_foobar 2y ago
    I might be dense, but I don't understand what that has to do with type hints... To my eyes, the problem of choosing useful defaults for complicated types/datastructures is independent of whether I add type hints for them. I think I am missing something...
- CJefferson 2y ago
  Mainly, the seems to be no way, in a dynamic language, to dynamically check if functions get the right types. To me, this means I don't really understand the python type hinting at all, as adding hints to just one or two functions provides no value to me at all. I assume I must be not using them usefully, as I've tried adding type hints to some projects and they just seemed to do nothing useful.
  - Spivak 2y ago
    Type hints alone don't do this, but you can use Pydantic to accomplish what you want. In Python type hints aren't enforced anywhere at runtime. They're for a type-checker to validate your source. https://docs.pydantic.dev/latest/concepts/validation_decorator/ https://docs.pydantic.dev/latest/concepts/validation_decorat...
belorn 2y ago
Looking at that blog post, I find it illustrative in how people who like strong types and people who dislike strong types are addressing different form of bugs. If the main types of issues comes from bugs like 1 + "2" == 12", then strong types is a big help. It also enables many developers who spend the majority of time in a programming editor to quickly get automatic help with such bugs. The other side is those people who do not find those kind of bugs annoying, or they simply don't get hit by such bugs at a rate that is high enough to warrant using a strong type system. Developers who spend their time prototyping in ipython also get less out of the strong types. The bugs that those developers are concerned about are design bugs, like finding out why a bunch of small async programs reading from a message buss may stall once every second Friday, and where the bug may be a dependency of a dependency of a dependency that do not use a socket timeout. Types are similar not going to help those who spend the wast majority of time on bugs where someone finally says "This design could never have worked".
- tasn 2y ago
  I think you're missing the point of the blog a bit, as the `1 + "2" == "12"` type of issues wasn't it. It definitely also sucks and much more common than you make it sound (especially when refactoring) but it's definitely not that. Anyhow, no need to rehash the same arguments, there was a long thread here on HN about the post, you can read some of it here: https://news.ycombinator.com/item?id=37764326 https://news.ycombinator.com/item?id=37764326
- satvikpendem 2y ago
  It's not about the bugs, it's about designing the layout of the program in types first (ie, laying out all of the data structures required) such that the actual coding of the functionality is fairly trivial. This is known as type driven development: https://blog.ploeh.dk/2015/08/10/type-driven-development/ https://blog.ploeh.dk/2015/08/10/type-driven-development/
  - slowking2 2y ago
    At work, I find type hints useful as basically enforced documentation and as a weak sort of test, but few type systems offer decent basic support for the sort of things you would need to do type driven programming in scientific/numerical work. Things like making sure matrices have compatible dimensions, handling units, and constraining the range of a numerical variable would be a solid minimum. I've read that F# has units, Ada and Pascal have ranges as types (my understanding is these are runtime enforced mostly), Rust will land const generics that might be useful for matrix type stuff some time soon. Does any language support all 3 of these things well together? Do you basically need fully dependent types for this? Obviously, with discipline you can work to enforce all these things at runtime, but I'd like it if there was a language that made all 3 of these things straightforward.
ansgri 2y ago
If it’s 100x better than no types, then probably 10x better than C++ type system. It takes some time to unlearn using dicts everywhere, but then namedtuples become your best friend and noticeably improve maintainability. Probably the only place where python type system feels inadequate is describing json-like data near the point of its (de)serialization.
- rcfox 2y ago
  Pretty much anywhere you're tempted to use a namedtuple, you should be using a dataclass[0] instead. And typing JSON-like data is possible with TypedDict[1]. [0] https://docs.python.org/3/library/dataclasses.html https://docs.python.org/3/library/dataclasses.html [1] https://docs.python.org/3/library/typing.html#typing.TypedDict https://docs.python.org/3/library/typing.html#typing.TypedDi...
  - ansgri 2y ago
    Why? I thought one should prefer immutability. As for typed dicts.. yes, I’m mostly stuck on old python versions, nice reminder.
    - maleldil 2y ago
      You can use TypedDict from `typing_extensions` if your version doesn't have it. You can use a lot of the newer stuff from there, too, especially if you enable `__future__.annotations`. How old is your Python, though? TypedDict is from 3.8. That was 5 years ago.
    - throwaway2037 2y ago
      You can use: > @dataclass(frozen=True) to create an immutable data class.
      - maleldil 2y ago
        While that works (and I use it extensively), it's a bit hacky. You have to use `object.__setattr__` to set attributes in `__init__` or `__post_init__`, which looks so wrong.
VBprogrammer 2y ago
I had a play with Dart a while back. It felt like Python with types designed in from the outset. Would quite like to use it more seriously. It's in that funny position though where it is in danger of becoming synonymous with Flutter. Like Ruby and Rails.
hbrn 2y ago
> Writing software without types lets you go at full speed. Full speed towards the cliff. Isn't it strange that back when Python (or Ruby) didn't even have type hints (not type checkers, type hints!), it would easily outperform pretty much every heavily typed language? Somehow when types weren't an option we weren't going towards the cliff, but now that they are, not using them means jumping off a cliff? Something doesn't add up.
- dinosaurdynasty 2y ago
  It's because the nature of typing has changed drastically over the last decade or so, in well known languages, going from C++/Java's `FancyObject *fancyObject = new FancyObject()` (which was definitely annoying to type, and was seen as a way to "tell the compiler how to arrange memory" as opposed to "how do we ensure constraints hold?") to modern TypeScript, where large well-typed programs can be written with barely a type annotation in sight. There's also a larger understanding that as programs get larger and larger, they get harder to maintain and more importantly refactor, and good types help with this much more than brittle unit tests do. (You can also eliminate a lot of busywork tests with types.)
  - hbrn 2y ago
    Large programs are harder to maintain because people don't have the balls to break them into smaller ones with proper boundaries. They prefer incremental bandaids like type hints or unit tests that make it easier to deal with the big ball of mud, instead of not building the ball in the first place.
  - liontwist 2y ago
    No it hasn’t? C++ type system has hardly changed (until concepts) and is one of the most powerful available. A certain generation of devs thought types were academic nonsense and then relearned the existence of those features in other languages. Now they are zealots about using them.
    - josephg 2y ago
junyjujfjf 2y ago
> The Python type system is pretty bad Coming from the perspective of a religious python hater, their type hints are better than what you give credit for: Supports generics, nominative, structural, unions, bottom type, and literals. What is missing is mainstream adoption in libraries which is a matter of time.
- maleldil 2y ago
  > What is missing is mainstream adoption in libraries which is a matter of time. I don't think that's a big problem anymore. Between typeshed and typing's overall momentum, most libraries have at least decent typing and those that don't often have typed alternatives.
  - Chris_Newton 2y ago
    I don't think that's a big problem anymore. ORMs have entered the chat… These sometimes use a lot of dynamic modification, such as adding implicit ID fields or adding properties to navigate a relationship with another type that is defined in code only from the other side. It can also be awkward to deal with “not null” database fields if the way the ORM model classes are defined means fields are nullable as far as the Python type hints are concerned, yet the results of an actual database query should never have a null value there. Guarding against None every time you refer to one of them is tedious. I’m not exactly the world’s loudest advocate for ORMs anyway, but on projects that also try to take type safety seriously, they do seem to be a bit of a dark corner within the Python ecosystem.
    - never_inline 2y ago
      The approach I have found to work is isolate the logic which deals with ORM models, and convert them to/from typed models (eg pydantic) at the function boundary. with sqlalchemy mapped_column, its less of an issue. django, otoh, seems too much magic for static type. (happy to be proven wrong).
- lmm 2y ago
  Optional typing is always a castle built on sand. I don't see Python typing ever becoming reliable, because there's no way you can retrofit the entire ecosystem that thoroughly.
kristianp 2y ago
Isn't the rust type system fairly off-topic here? Python is a dynamic language, Rust is on the other end of the scale. The Rust Evangelism Strike Force used to be more subtle! (joke)
baranul 2y ago
For the Python people, it seems a matter of habit and culture. When a person has gone down a certain direction for so long, it can be really hard to change. Think that's why it is a good idea to be exposed to other languages earlier on, where the person would have seen other type systems and other ways of doing things. There wouldn't be so much trauma and drama, when confronted with types or differences.

by tasn 2y ago

youdontknowjuli 2y ago

> (and some cool hackery to make SQLAlchemy feel very typed and work nicely with Pydantic). Sounds interesting. Can you elaborate on the cool hackery? We introduced SQLModel recently but struggle in a few cases (e.g. multi-level joins). Do you know reference projects for SQLAlchemy and pydantic?

tasn 2y ago
My info is maybe a bit dated, as it's been a while since we wrote this hackery. We also adopted SQLModel at some point but we had to patch it to work well (I think some of my contributions are now in upstream). As for some of the hacks: def c(prop: t.Any) -> sa.Column: # type: ignore return prop To make it possible to access sqlmodel properties as columns for doing things like `in_` but still maintaining type safety. Added types ourselves to the base model like this: __table__: t.ClassVar[sa.Table] Added functions that help with typing like this: @classmethod async def _fetch_one(cls: t.Type[BaseT], db: BaseReadOnlySqlSession, query: Select) -> t.Optional[BaseT]: try: return (await db.execute(query)).scalar_one() except NoResultFound: return None and stuff like this for relationships: def ezrelationship( model: t.Type[T_], id_our: t.Union[str, sa.Column], # type: ignore id_other: t.Optional[t.Union[t.Any, sa.Column]] = None, # type: ignore ) -> T_: if id_other is None: id_other = model.id return sqlm.Relationship(sa_relationship=relationship(model, primaryjoin=f"foreign({id_our}) == {id_other}")) def ezrelationship_back( id_our: t.Union[str, sa.Column], # type: ignore id_other: t.Union[str, sa.Column], # type: ignore ) -> t.Any: model, only_id2 = id_other.split(".") return sqlm.Relationship( sa_relationship=relationship( model, primaryjoin=f"foreign({id_our}) == {id_other}_id", back_populates=only_id2, ) ) I hope this helps, I don't have time to find all the stuff, but we also hacked on SQLAlchemy a bit, and in other places.

Myrmornis 2y ago

Can you give some examples of how the Python type system is disappointing you?

colemannerd 2y ago
default values! Since type hints are *hints*, it is difficult to set default values for complicated types. For instance, if you have lists, dicts, sets in the type signature, without a library like pydantic, it is difficult and non-standard. This becomes even more problematic when you start doing more complicated data structures. The configuration in this library starts to show the problems. https://koxudaxi.github.io/datamodel-code-generator/custom_template/ https://koxudaxi.github.io/datamodel-code-generator/custom_t... The issue very much is a lack of a standard for the entire language; rather than it not being possible.
- alfons_foobar 2y ago
  I might be dense, but I don't understand what that has to do with type hints... To my eyes, the problem of choosing useful defaults for complicated types/datastructures is independent of whether I add type hints for them. I think I am missing something...
CJefferson 2y ago
Mainly, the seems to be no way, in a dynamic language, to dynamically check if functions get the right types. To me, this means I don't really understand the python type hinting at all, as adding hints to just one or two functions provides no value to me at all. I assume I must be not using them usefully, as I've tried adding type hints to some projects and they just seemed to do nothing useful.
- Spivak 2y ago
  Type hints alone don't do this, but you can use Pydantic to accomplish what you want. In Python type hints aren't enforced anywhere at runtime. They're for a type-checker to validate your source. https://docs.pydantic.dev/latest/concepts/validation_decorator/ https://docs.pydantic.dev/latest/concepts/validation_decorat...

belorn 2y ago

Looking at that blog post, I find it illustrative in how people who like strong types and people who dislike strong types are addressing different form of bugs. If the main types of issues comes from bugs like 1 + "2" == 12", then strong types is a big help. It also enables many developers who spend the majority of time in a programming editor to quickly get automatic help with such bugs. The other side is those people who do not find those kind of bugs annoying, or they simply don't get hit by such bugs at a rate that is high enough to warrant using a strong type system. Developers who spend their time prototyping in ipython also get less out of the strong types. The bugs that those developers are concerned about are design bugs, like finding out why a bunch of small async programs reading from a message buss may stall once every second Friday, and where the bug may be a dependency of a dependency of a dependency that do not use a socket timeout. Types are similar not going to help those who spend the wast majority of time on bugs where someone finally says "This design could never have worked".

tasn 2y ago
I think you're missing the point of the blog a bit, as the `1 + "2" == "12"` type of issues wasn't it. It definitely also sucks and much more common than you make it sound (especially when refactoring) but it's definitely not that. Anyhow, no need to rehash the same arguments, there was a long thread here on HN about the post, you can read some of it here: https://news.ycombinator.com/item?id=37764326 https://news.ycombinator.com/item?id=37764326
satvikpendem 2y ago
It's not about the bugs, it's about designing the layout of the program in types first (ie, laying out all of the data structures required) such that the actual coding of the functionality is fairly trivial. This is known as type driven development: https://blog.ploeh.dk/2015/08/10/type-driven-development/ https://blog.ploeh.dk/2015/08/10/type-driven-development/
- slowking2 2y ago
  At work, I find type hints useful as basically enforced documentation and as a weak sort of test, but few type systems offer decent basic support for the sort of things you would need to do type driven programming in scientific/numerical work. Things like making sure matrices have compatible dimensions, handling units, and constraining the range of a numerical variable would be a solid minimum. I've read that F# has units, Ada and Pascal have ranges as types (my understanding is these are runtime enforced mostly), Rust will land const generics that might be useful for matrix type stuff some time soon. Does any language support all 3 of these things well together? Do you basically need fully dependent types for this? Obviously, with discipline you can work to enforce all these things at runtime, but I'd like it if there was a language that made all 3 of these things straightforward.

ansgri 2y ago

If it’s 100x better than no types, then probably 10x better than C++ type system. It takes some time to unlearn using dicts everywhere, but then namedtuples become your best friend and noticeably improve maintainability. Probably the only place where python type system feels inadequate is describing json-like data near the point of its (de)serialization.

rcfox 2y ago
Pretty much anywhere you're tempted to use a namedtuple, you should be using a dataclass[0] instead. And typing JSON-like data is possible with TypedDict[1]. [0] https://docs.python.org/3/library/dataclasses.html https://docs.python.org/3/library/dataclasses.html [1] https://docs.python.org/3/library/typing.html#typing.TypedDict https://docs.python.org/3/library/typing.html#typing.TypedDi...
- ansgri 2y ago
  Why? I thought one should prefer immutability. As for typed dicts.. yes, I’m mostly stuck on old python versions, nice reminder.
  - maleldil 2y ago
    You can use TypedDict from `typing_extensions` if your version doesn't have it. You can use a lot of the newer stuff from there, too, especially if you enable `__future__.annotations`. How old is your Python, though? TypedDict is from 3.8. That was 5 years ago.
  - throwaway2037 2y ago
    You can use: > @dataclass(frozen=True) to create an immutable data class.
    - maleldil 2y ago
      While that works (and I use it extensively), it's a bit hacky. You have to use `object.__setattr__` to set attributes in `__init__` or `__post_init__`, which looks so wrong.

VBprogrammer 2y ago

I had a play with Dart a while back. It felt like Python with types designed in from the outset. Would quite like to use it more seriously. It's in that funny position though where it is in danger of becoming synonymous with Flutter. Like Ruby and Rails.

hbrn 2y ago

> Writing software without types lets you go at full speed. Full speed towards the cliff. Isn't it strange that back when Python (or Ruby) didn't even have type hints (not type checkers, type hints!), it would easily outperform pretty much every heavily typed language? Somehow when types weren't an option we weren't going towards the cliff, but now that they are, not using them means jumping off a cliff? Something doesn't add up.

dinosaurdynasty 2y ago
It's because the nature of typing has changed drastically over the last decade or so, in well known languages, going from C++/Java's `FancyObject *fancyObject = new FancyObject()` (which was definitely annoying to type, and was seen as a way to "tell the compiler how to arrange memory" as opposed to "how do we ensure constraints hold?") to modern TypeScript, where large well-typed programs can be written with barely a type annotation in sight. There's also a larger understanding that as programs get larger and larger, they get harder to maintain and more importantly refactor, and good types help with this much more than brittle unit tests do. (You can also eliminate a lot of busywork tests with types.)
- hbrn 2y ago
  Large programs are harder to maintain because people don't have the balls to break them into smaller ones with proper boundaries. They prefer incremental bandaids like type hints or unit tests that make it easier to deal with the big ball of mud, instead of not building the ball in the first place.
- liontwist 2y ago
  No it hasn’t? C++ type system has hardly changed (until concepts) and is one of the most powerful available. A certain generation of devs thought types were academic nonsense and then relearned the existence of those features in other languages. Now they are zealots about using them.
  - josephg 2y ago

junyjujfjf 2y ago

> The Python type system is pretty bad Coming from the perspective of a religious python hater, their type hints are better than what you give credit for: Supports generics, nominative, structural, unions, bottom type, and literals. What is missing is mainstream adoption in libraries which is a matter of time.

maleldil 2y ago
> What is missing is mainstream adoption in libraries which is a matter of time. I don't think that's a big problem anymore. Between typeshed and typing's overall momentum, most libraries have at least decent typing and those that don't often have typed alternatives.
- Chris_Newton 2y ago
  I don't think that's a big problem anymore. ORMs have entered the chat… These sometimes use a lot of dynamic modification, such as adding implicit ID fields or adding properties to navigate a relationship with another type that is defined in code only from the other side. It can also be awkward to deal with “not null” database fields if the way the ORM model classes are defined means fields are nullable as far as the Python type hints are concerned, yet the results of an actual database query should never have a null value there. Guarding against None every time you refer to one of them is tedious. I’m not exactly the world’s loudest advocate for ORMs anyway, but on projects that also try to take type safety seriously, they do seem to be a bit of a dark corner within the Python ecosystem.
  - never_inline 2y ago
    The approach I have found to work is isolate the logic which deals with ORM models, and convert them to/from typed models (eg pydantic) at the function boundary. with sqlalchemy mapped_column, its less of an issue. django, otoh, seems too much magic for static type. (happy to be proven wrong).
lmm 2y ago
Optional typing is always a castle built on sand. I don't see Python typing ever becoming reliable, because there's no way you can retrofit the entire ecosystem that thoroughly.

kristianp 2y ago

Isn't the rust type system fairly off-topic here? Python is a dynamic language, Rust is on the other end of the scale. The Rust Evangelism Strike Force used to be more subtle! (joke)

baranul 2y ago

For the Python people, it seems a matter of habit and culture. When a person has gone down a certain direction for so long, it can be really hard to change. Think that's why it is a good idea to be exposed to other languages earlier on, where the person would have seen other type systems and other ways of doing things. There wouldn't be so much trauma and drama, when confronted with types or differences.