6 ms·

The thing that finally got me on board with optional type hints in Python was realizing that they're mainly valuable as documentation. But it's really valuable

by simonw 1y ago

The thing that finally got me on board with optional type hints in Python was realizing that they're mainly valuable as documentation. But it's really valuable documentation! Knowing what types are expected and returned just by looking at a function signature is super useful.

lysace 1y ago
Decent argument in principle. It still sucks for non-obvious types though: https://old.reddit.com/r/Python/comments/10zdidm/why_type_hinting_sucks/ https://old.reddit.com/r/Python/comments/10zdidm/why_type_hi... Edit: Yes, one can sometimes go with Any, depending on the linter setup, but that's missing the point, isn't it?
- mjr00 1y ago
  As the top comment says, if you don't know or want to define the type just use Any. That's what it's there for. That entire Reddit post is a clueless expert beginner rant about something they don't really understand, unfortunate that it's survived as long as it has or that anyone is taking it as any sort of authoritative argument just because it's long.
  - pdonis 1y ago
    > if you don't know or want to define the type That's not the issue the reddit post is raising. The reddit post is pointing out that what a "type" is is not as simple as it looks. Particularly in a language like Python where user-defined types proliferate, and can add dunder methods that affect statements that involve built-in operations. "Just use Any" doesn't solve any of those problems. > just use Any. All the above said: not putting a type in at all is even easier than using Any, and is semantically equivalent.
    - mjr00 1y ago
      The Reddit post falls under the case of "don't know" the type. If you want to allow users to pass in any objects, try to add and fail at runtime... that's exactly what Any is for. But the entire post is built upon the premise that accepting all types is good API design. Which it isn't, at all.
      - zahlman 1y ago
        > But the entire post is built upon the premise that accepting all types is good API design. Which it isn't, at all. Was Tim Peters also wrong way back in the day when he counseled Guido van Rossum to allow floats to be added to integers without a cast, like other popular languages?
        mjr00 1y ago
        How is `float | int` anywhere close to equivalent to `Any`?
        MrJohz 1y ago
        How is "responds to the `__add__` method" anywhere close to equivalent to `Any`?
        mjr00 1y ago
        If your implication is that "implementing __add__ means you can use the + operator", you are incorrect. This is a common Python beginner mistake, but it isn't really a Python type checking issue, this is complexity with Python built-ins and how they interact with magic methods. My suggestion -- don't rely on magic methods.
        jampekka 1y ago
        > My suggestion -- don't rely on magic methods. So no e.g. numpy or torch then?
        MrJohz 1y ago
- drekipus 1y ago
  The correct response to this is to figure what is the use case for your function: IE: add two numbers. Set the input and output as decimal and call it a day
  - baobabKoodaa 1y ago
    Sure. Let me just quickly refactor Pytorch.
- kstenerud 1y ago
  Actually, it's not missing the point. Sometimes you really do want duck typing, in which case you allow Any. It's not all-or-nothing. What the reddit post is demonstrating is that the Python type system is still too naive in many respects (and that there are implementation divergences in behavior). In other languages, this is a solved problem - and very ergonomic and safe.
smilliken 1y ago
The best kind of documentation is the kind you can trust is accurate. Type defs wouldn't be close to as useful if you didn't really trust them. Similarly, doctests are some of the most useful documentation because you can be sure they are accurate.
- zbentley 1y ago
  The best docs are the ones you can trust are accurate. The second best docs are ones that you can programmatically validate. The worst docs are the ones that can’t be validated without lots of specialized effort. Python’s type hints are in the second category.
  - seniorsassycat 1y ago
    Do you have an example of the first?
    - ItsHarper 1y ago
      Languages with strong static type systems
      - LudwigNagasena 1y ago
        Is there a mainstream language where you can’t arbitrarily cast a variable to any other type?
    - zbentley 1y ago
      When I wrote that, I was thinking about typed, compiled languages' documentation generated by the compiler at build time. Assuming that version drift ("D'oh, I was reading the docs for v1.2.3 but running v4.5.6") is user error and not a docs-trustworthiness issue, that'd qualify. But now that I'm coming back to it, I think that this might be a larger category than I first envisioned, including projects whose build/release processes very reliably include the generation+validation+publication of updated docs. That doesn't imply a specific language or release automation, just a strong track record of doc-accuracy linked to releases. In other words, if a user can validate/regenerate the docs for a project, that gets it 9/10 points. The remaining point is the squishier "the first party docs are always available and well-validated for accuracy" stuff.
ninetyninenine 1y ago
This is a naive realization. When type checking is used to the maximum extent they become as just as important as unit testing. It is an actual safety contribution to the code. Many old school python developers don't realize how important typing actually is. It's not just documentation. It can actually roughly reduce dev time by 50% and increase safety by roughly 2x.
- MangoToupe 1y ago
  Static type checking (which is what I assume you mean by "typing") can also be a massive pain in the ass that stands in the way of incremental development, even if the end-goal is to ship an api with clear type signatures. There are developers who design apis by trying to figure out readable invocations. These developers discover, rather than design, type hierarchies and library interfaces. > Many old school python developers don't realize how important typing actually is. I don't think this is true. There's simply a communication breakdown where type-first developers don't see the benefits of disabling static checking to design interfaces, and interface-first developers don't see why they should put static checking ahead of interface iteration speed.
  - ninetyninenine 1y ago
    >There are developers who design apis by trying to figure out readable invocations. These developers discover, rather than design, type hierarchies and library interfaces. No, you're one of the old school python developers. Types don't hinder creativity, they augment it. The downside is the slight annoyance of updating a type definition and the run time definition vs. just updating the runtime definition. Let me give you an example of how it hinders creativity. Let's say you have a interface that is immensely complex. Many nested structures thousands of keys, and let's say you want to change the design by shifting 3 or 4 things around. Let's also say this interface is utilized by hundreds of other methods and functions. When you move 3 or 4 things around in a complex interface you're going to break a subset of those hundreds of other methods or functions. You're not going to know where they break if you don't have type checking enabled. You're only going to know if you tediously check every single method/function OR if it crashes during runtime. With a statically typed definition you can do that change and the type checker will identify EVERY single place where an additional change to the methods that use that type needs to be changed as well. This allows you to be creative and make any willy nilly changes you want because you are confident that ANY change will be caught by the type checker. This Speeds up creativity, while without it, you will be slowed down, and even afraid to make the breaking change. You are basically the stereotype I described. An old school python developer. Likely one who got used to programming without types and now hasn't utilized types extensively enough to see the benefit. >I don't think this is true. There's simply a communication breakdown where type-first developers don't see the benefits of disabling static checking to design interfaces, and interface-first developers don't see why they should put static checking ahead of interface iteration speed. This is true. You're it. You just don't know it. When I say these developers don't know I'm literally saying they think like you and believe the same things you believe BECAUSE they lack knowledge and have bad habits. The habit thing is what causes the warped knowledge. You're actually slowed down by types because you're not used to it as you spent years coding in python without types so it's ingrained for you to test and think without types. Adding additional types becomes a bit of a initial overhead for these types of people because their programming style is so entrenched. Once you get used to it and once you see that it's really just a slight additional effort, than you will get it. But it takes a bit of discipline and practice to get there.
lelanthran 1y ago
> The thing that finally got me on board with optional type hints in Python was realizing that they're mainly valuable as documentation. > But it's really valuable documentation! Knowing what types are expected and returned just by looking at a function signature is super useful. So ... you didn't have this realisation prior to using Python type hints? Not from any other language you used prior to Python?
- simonw 1y ago
  I didn't. I've been mainly a Python, PHP and JavaScript programmer for ~25 years and my experience with typed languages was mostly pre-type-inference Java which felt wildly less productive than my main languages.
  - chpatrick 1y ago
    That was more because of Java than static typing.
  - lelanthran 1y ago
    > I didn't. I've been mainly a Python, PHP and JavaScript programmer for ~25 years Maybe its time you expanded your horizons, then. Try a few statically typed languages. Even plain C gives you a level of confidence in deployed code that you will not get in Python, PHP or Javascript.
    - sevensor 1y ago
      Maybe if your C has aggressive test coverage and you’re using Valgrind religiously and always checking errno when you’re supposed to and you’re checking the return value of everything. Otherwise lol. C as it’s written by middling teams is a soup of macros, three-star variables, and questionable data structure implementations, where everybody fiddles with everybody else’s data. I’ll take good C over bad Python, but good C is rare.
      - lelanthran 1y ago
__MatrixMan__ 1y ago
Type hints as documentation are a gateway drug to type hints for bug finding. Keep at it :)
Syntaf 1y ago
This resonates with me this so much. I feel like half the comments in this thread are missing the value typing, but maybe they've never had the misfortune of working with hundreds of other developers on a project with no defined contracts on aggregates / value objects outside of code comments and wishful thinking. I've worked on large python codebases for large companies for the last ~6 years of my career; types have been the single biggest boon to developer productivity and error reduction on these codebases. Just having to THINK about types eliminates so many opportunities for errors, and if your type is too complex to express it's _usually_ a code smell; most often these situations can be re-written in a more sane albeit slightly verbose fashion, rather than using the more "custom" typing features. No one gets points for writing "magical" code in large organizations, and typing makes sure of this. There's absolutely nothing wrong with writing "boring" python. Could we have accomplished this by simply having used a different language from the beginning? Absolutely, but often times that's not a option for a company with a mature stack. TL;DR -- Typing in python is an exception tool to scale your engineering organization on a code-base.

by simonw 1y ago

lysace 1y ago

Decent argument in principle. It still sucks for non-obvious types though: https://old.reddit.com/r/Python/comments/10zdidm/why_type_hinting_sucks/ https://old.reddit.com/r/Python/comments/10zdidm/why_type_hi... Edit: Yes, one can sometimes go with Any, depending on the linter setup, but that's missing the point, isn't it?

mjr00 1y ago
As the top comment says, if you don't know or want to define the type just use Any. That's what it's there for. That entire Reddit post is a clueless expert beginner rant about something they don't really understand, unfortunate that it's survived as long as it has or that anyone is taking it as any sort of authoritative argument just because it's long.
- pdonis 1y ago
  > if you don't know or want to define the type That's not the issue the reddit post is raising. The reddit post is pointing out that what a "type" is is not as simple as it looks. Particularly in a language like Python where user-defined types proliferate, and can add dunder methods that affect statements that involve built-in operations. "Just use Any" doesn't solve any of those problems. > just use Any. All the above said: not putting a type in at all is even easier than using Any, and is semantically equivalent.
  - mjr00 1y ago
    The Reddit post falls under the case of "don't know" the type. If you want to allow users to pass in any objects, try to add and fail at runtime... that's exactly what Any is for. But the entire post is built upon the premise that accepting all types is good API design. Which it isn't, at all.
    - zahlman 1y ago
      > But the entire post is built upon the premise that accepting all types is good API design. Which it isn't, at all. Was Tim Peters also wrong way back in the day when he counseled Guido van Rossum to allow floats to be added to integers without a cast, like other popular languages?
      - mjr00 1y ago
        How is `float | int` anywhere close to equivalent to `Any`?
        MrJohz 1y ago
        How is "responds to the `__add__` method" anywhere close to equivalent to `Any`?
        mjr00 1y ago
        If your implication is that "implementing __add__ means you can use the + operator", you are incorrect. This is a common Python beginner mistake, but it isn't really a Python type checking issue, this is complexity with Python built-ins and how they interact with magic methods. My suggestion -- don't rely on magic methods.
        jampekka 1y ago
        > My suggestion -- don't rely on magic methods. So no e.g. numpy or torch then?
        MrJohz 1y ago
drekipus 1y ago
The correct response to this is to figure what is the use case for your function: IE: add two numbers. Set the input and output as decimal and call it a day
- baobabKoodaa 1y ago
  Sure. Let me just quickly refactor Pytorch.
kstenerud 1y ago
Actually, it's not missing the point. Sometimes you really do want duck typing, in which case you allow Any. It's not all-or-nothing. What the reddit post is demonstrating is that the Python type system is still too naive in many respects (and that there are implementation divergences in behavior). In other languages, this is a solved problem - and very ergonomic and safe.

smilliken 1y ago

The best kind of documentation is the kind you can trust is accurate. Type defs wouldn't be close to as useful if you didn't really trust them. Similarly, doctests are some of the most useful documentation because you can be sure they are accurate.

zbentley 1y ago
The best docs are the ones you can trust are accurate. The second best docs are ones that you can programmatically validate. The worst docs are the ones that can’t be validated without lots of specialized effort. Python’s type hints are in the second category.
- seniorsassycat 1y ago
  Do you have an example of the first?
  - ItsHarper 1y ago
    Languages with strong static type systems
    - LudwigNagasena 1y ago
      Is there a mainstream language where you can’t arbitrarily cast a variable to any other type?
  - zbentley 1y ago
    When I wrote that, I was thinking about typed, compiled languages' documentation generated by the compiler at build time. Assuming that version drift ("D'oh, I was reading the docs for v1.2.3 but running v4.5.6") is user error and not a docs-trustworthiness issue, that'd qualify. But now that I'm coming back to it, I think that this might be a larger category than I first envisioned, including projects whose build/release processes very reliably include the generation+validation+publication of updated docs. That doesn't imply a specific language or release automation, just a strong track record of doc-accuracy linked to releases. In other words, if a user can validate/regenerate the docs for a project, that gets it 9/10 points. The remaining point is the squishier "the first party docs are always available and well-validated for accuracy" stuff.

ninetyninenine 1y ago

This is a naive realization. When type checking is used to the maximum extent they become as just as important as unit testing. It is an actual safety contribution to the code. Many old school python developers don't realize how important typing actually is. It's not just documentation. It can actually roughly reduce dev time by 50% and increase safety by roughly 2x.

MangoToupe 1y ago
Static type checking (which is what I assume you mean by "typing") can also be a massive pain in the ass that stands in the way of incremental development, even if the end-goal is to ship an api with clear type signatures. There are developers who design apis by trying to figure out readable invocations. These developers discover, rather than design, type hierarchies and library interfaces. > Many old school python developers don't realize how important typing actually is. I don't think this is true. There's simply a communication breakdown where type-first developers don't see the benefits of disabling static checking to design interfaces, and interface-first developers don't see why they should put static checking ahead of interface iteration speed.
- ninetyninenine 1y ago
  >There are developers who design apis by trying to figure out readable invocations. These developers discover, rather than design, type hierarchies and library interfaces. No, you're one of the old school python developers. Types don't hinder creativity, they augment it. The downside is the slight annoyance of updating a type definition and the run time definition vs. just updating the runtime definition. Let me give you an example of how it hinders creativity. Let's say you have a interface that is immensely complex. Many nested structures thousands of keys, and let's say you want to change the design by shifting 3 or 4 things around. Let's also say this interface is utilized by hundreds of other methods and functions. When you move 3 or 4 things around in a complex interface you're going to break a subset of those hundreds of other methods or functions. You're not going to know where they break if you don't have type checking enabled. You're only going to know if you tediously check every single method/function OR if it crashes during runtime. With a statically typed definition you can do that change and the type checker will identify EVERY single place where an additional change to the methods that use that type needs to be changed as well. This allows you to be creative and make any willy nilly changes you want because you are confident that ANY change will be caught by the type checker. This Speeds up creativity, while without it, you will be slowed down, and even afraid to make the breaking change. You are basically the stereotype I described. An old school python developer. Likely one who got used to programming without types and now hasn't utilized types extensively enough to see the benefit. >I don't think this is true. There's simply a communication breakdown where type-first developers don't see the benefits of disabling static checking to design interfaces, and interface-first developers don't see why they should put static checking ahead of interface iteration speed. This is true. You're it. You just don't know it. When I say these developers don't know I'm literally saying they think like you and believe the same things you believe BECAUSE they lack knowledge and have bad habits. The habit thing is what causes the warped knowledge. You're actually slowed down by types because you're not used to it as you spent years coding in python without types so it's ingrained for you to test and think without types. Adding additional types becomes a bit of a initial overhead for these types of people because their programming style is so entrenched. Once you get used to it and once you see that it's really just a slight additional effort, than you will get it. But it takes a bit of discipline and practice to get there.

lelanthran 1y ago

> The thing that finally got me on board with optional type hints in Python was realizing that they're mainly valuable as documentation. > But it's really valuable documentation! Knowing what types are expected and returned just by looking at a function signature is super useful. So ... you didn't have this realisation prior to using Python type hints? Not from any other language you used prior to Python?

simonw 1y ago
I didn't. I've been mainly a Python, PHP and JavaScript programmer for ~25 years and my experience with typed languages was mostly pre-type-inference Java which felt wildly less productive than my main languages.
- chpatrick 1y ago
  That was more because of Java than static typing.
- lelanthran 1y ago
  > I didn't. I've been mainly a Python, PHP and JavaScript programmer for ~25 years Maybe its time you expanded your horizons, then. Try a few statically typed languages. Even plain C gives you a level of confidence in deployed code that you will not get in Python, PHP or Javascript.
  - sevensor 1y ago
    Maybe if your C has aggressive test coverage and you’re using Valgrind religiously and always checking errno when you’re supposed to and you’re checking the return value of everything. Otherwise lol. C as it’s written by middling teams is a soup of macros, three-star variables, and questionable data structure implementations, where everybody fiddles with everybody else’s data. I’ll take good C over bad Python, but good C is rare.
    - lelanthran 1y ago

__MatrixMan__ 1y ago

Type hints as documentation are a gateway drug to type hints for bug finding. Keep at it :)

Syntaf 1y ago

This resonates with me this so much. I feel like half the comments in this thread are missing the value typing, but maybe they've never had the misfortune of working with hundreds of other developers on a project with no defined contracts on aggregates / value objects outside of code comments and wishful thinking. I've worked on large python codebases for large companies for the last ~6 years of my career; types have been the single biggest boon to developer productivity and error reduction on these codebases. Just having to THINK about types eliminates so many opportunities for errors, and if your type is too complex to express it's _usually_ a code smell; most often these situations can be re-written in a more sane albeit slightly verbose fashion, rather than using the more "custom" typing features. No one gets points for writing "magical" code in large organizations, and typing makes sure of this. There's absolutely nothing wrong with writing "boring" python. Could we have accomplished this by simply having used a different language from the beginning? Absolutely, but often times that's not a option for a company with a mature stack. TL;DR -- Typing in python is an exception tool to scale your engineering organization on a code-base.