6 ms·

I am not well versed in python programming, this is just my opinion as an outsider. For anyone interested in using these tools, I suggest reading the following

I am not well versed in python programming, this is just my opinion as an outsider. For anyone interested in using these tools, I suggest reading the following: https://www.reddit.com/r/Python/comments/10zdidm/why_type_hinting_sucks/ https://www.reddit.com/r/Python/comments/10zdidm/why_type_hi... That post should probably be taken lightly, but I think that the goal there is to understand that even with the best typing tools, you will have troubles, unless you start by establishing good practices. For example, Django is large code base, and if you look at it, you will observe that the code is consistent in which features of python are used and how; this project passes the stricter type checking test without troubles. Likewise, Meta certainly has a very large code base (why develop a type checker otherwise?), and they must have figured out that they cannot let their programmers write code however they like; I guess their type checker is the stricter one for that reason. Python, AFAIK, has many features, a very permissive runtime, and perhaps (not unlike C++) only some limited subset should be used at any time to ensure that the code is manageable. Unfortunately, that subset is probably different depending on who you ask, and what you aim to do. (Interestingly, the Reddit post somehow reminded me of the hurdles Rust people have getting the Linux kernel guys to accept their practice: C has a much simpler and carefree type system, but Rust being much more strict rubs those C guys the wrong way).

mhh__ 1y ago
At this point I'm fairly convinced that the effort one would spend trying to typecheck a python program is better spent migrating away from python into a language that has a proper type system, then using interop so you can still have the bits/people that need python be in python. Obviously that isn't always possible but you can spend far too long trying to make python work.
- davedx 1y ago
  Unfortunately with us being in the middle of the AI hype cycle, everyone and their dog is currently busy migrating to python.
  - hu3 1y ago
    I don't see why AI hype means more Python code. State of the art AI models are all closed source and accessible through an API anyways. APIs that any language can easily access. AAa for AI model development in itself, yes it's mostly Pyython, but niche.
    - davedx 1y ago
      Because everyone and their dog thinks they need to be ready to develop their own models, in python. This is honestly a thing, at least in the startup world.
    - mzl 1y ago
      I think you are underestimating the massive amounts of Python code that is built around these things. Also, a lot of businesses are not really interested in using an API for an LLM, instead they will modify and fine-tune their own models and deploy in their own data-centers (virtual or physical), and that means even more Python code. Sure, a system that only relies on token factory LLM APIs can be written in any language, but that is not the full width and breadth of the AI hype.
      - hu3 1y ago
        > Also, a lot of businesses are not really interested in using an API for an LLM, instead they will modify and fine-tune their own models and deploy in their own data-centers You realize model training cost millions right? "a lot of businesses" doesn't pass sniff test here. I'm not even counting the large swaths of data required to train. And the expensive specialists. And then you'll have to retrain outdated models every so often. There's a reason that AI has only a handful of players delivering SoTA models and these players are all worth $5B+.
        mzl 1y ago
        SOTA LLM model training costs a lot, yes. But fine-tuning and training of smaller models is a lot cheaper. I've trained useful vision-models that delivered business value for industrial applications on a MacBook overnight.
- J_Shelby_J 1y ago
  Six month into learning to build a modern python app, with linters, type systems, tests, venvs, package managers, etc… I realized that the supposed difficulty of rust is drastically less than coming to speed and then keeping up with the python “at scale” ecosystem.
  - mdaniel 1y ago
    My strong suspicion is that such a story depends a great deal upon the personalities of the developers much more than any {tool chaos + type chaos} --- {new syntax + lifespan annotations} spectrum
- bjackman 1y ago
  I think you're forgetting how easy type annotation is. I occasionally spend like 2h working on some old python code. I will spend say 15 minutes of that time adding type annotations (sometimes requires some trivial refactoring). This has an enormous ROI, the cost is so low and the benefit is so immediate. In these cases migrating code to a proper language and figuring out interop is not on my radar, it would be insane. So having the option to get some best-effort type safety is absolutely fantastic. I can definitely see your point, it's a useful analysis for projects under heavy development. But if you have a big Python codebase that basically just works and only sees incremental changes, adding type annotations is a great strategy.
- guappa 1y ago
  If you do that you need to compile, which means you can't just distribute a text file with your python program. You need a build infrastructure for every python version, every architecture and every OS. Have fun with that!
- rtpg 1y ago
  I don't understand this point at all. I've worked on Django codebases which have a huge set of typing problems... and while it's not 100% I get a lot of value out of type checking. You annotate enough functions and you get a really good linter out of it!
- ramraj07 1y ago
  If you're supposedly good at software and you spent too long trying to make python work consider the possibility that you're not good at software? Python has flaws and big ones at that, but there's a reason it's popular. Especially with tools like pydantic and fastapi and uv (and streamlit) you can do insane things in hours what would take weeks and months before. Not to mention how good AI is at generating code in these frameworks. I especially like typing using pydantic, any method is now able to dump and load data from files and dbs and you get extremely terse validated code. Modern IDEs also make quick work of extracting value even from partially typed code. I'd suggest you just open your mind up to imperfect things and give them a shot.
kccqzy 1y ago
> Python, AFAIK, has many features, a very permissive runtime, and perhaps (not unlike C++) only some limited subset should be used at any time to ensure that the code is manageable. Unfortunately, that subset is probably different depending on who you ask, and what you aim to do. I'll get started on the subset of Python that I personally do not wish to use in my own codebase: meta classes, descriptors, callable objects using __call__, object.__new__(cls), names trigger the name mangling rules, self.__dict__. In my opinion, all of the above features involve too much magic and hinder code comprehension.
- kstrauser 1y ago
  There's a time and a place for each of them: * Meta classes: You're writing Pydantic or an ORM. * Descriptors: You're writing Pydantic or an ORM. * Callable objects: I've used these for things like making validators you initialize with their parameters in one place, then pass them around so other functions can call them. I'd probably just use closures if at all possible now. * object.__new__: You're writing Pydantic or an ORM. * Name mangling: I'm fine with using _foo and __bar where appropriate. Those are nice. Don't ever, ever try to de-mangle them or I'll throw a stick at you. * self.__dict__: You're writing Pydantic or an ORM, although if you use this as shorthand for "doing things that need introspection", that's a useful skill and not deep wizardry. Basically, you won't need those things 99.99% of the time. If you think you do, you probably don't. If you're absolutely certain you do, you might. It's still good and important to understand what they are, though. Even if you never write them yourself, at some point you're going to want to figure out why some dependency isn't working the way you expected, and you'll need to read and know what it's doing.
  - kccqzy 1y ago
    > Basically, you won't need those things 99.99% of the time That's kind of my point. If you don't need a language feature 99.99% of the time perhaps it is better to cut it out from your language altogether. Well unless your language is striving to have the same reputation as C++. In Python's case here's a compromise: such features can only be used in a Python extension in C code, signifying their magic nature.
mjr00 1y ago
The top comment in that post shuts down the whole nonsense pretty quickly and firmly: > If you have a super-generic function like that and type hinting enforced, you just use Any and don't care about it. It's a stupid example, but even within the context of a `slow_add` function in a library: maybe the author originally never even thought people would pass in non-numeric values, so in the next version update instead of a hardcoded `time.sleep(0.1)` they decide to `time.sleep(a / b)`. Oops, now it crashes for users who passed in strings or tuples! If only there were a way to declare that the function is only intended to work with numeric values, instead of forcing yourself to provide backwards compatibility for users who used that function in unexpected ways that happened to work. IMO: for Python meant to run non-interactively with any sort of uptime guarantees, type checking is a no-brainer. You're actively making a mistake if you choose to not add type checking.
- notatallshaw 1y ago
  As the author of that post, I'd like to point out the example was meant to be stupid. The purpose was to show different ideologies and expectations on the same code don't work, such as strict backwards compatibilities, duck typing, and strictly following linting or type hinting rules (due to some arbitrary enforcement). Although re-reading it now I wish I'd spent more than an evening working on it, it's full of issues and not very polished. > If you have a super-generic function like that and type hinting enforced, you just use Any and don't care about it. Following the general stupidness of the post: they are now unable to do that because a security consultant said they have to enable and can not break RUFF rule ANN401: https://docs.astral.sh/ruff/rules/any-type/ https://docs.astral.sh/ruff/rules/any-type/
  - mjr00 1y ago
    > Following the general stupidness of the post: they are now unable to do that because a security consultant said they have to enable and can not break RUFF rule ANN401: https://docs.astral.sh/ruff/rules/any-type/ https://docs.astral.sh/ruff/rules/any-type/ Okay, then your function which is extremely generic and needs to support 25 different use cases needs to have an insane type definition which covers all 25 use cases. This isn't an indictment of the type system, this is an indictment of bad code. Don't write functions that support hundreds of input data types, most of which are unintended. Type systems help you avoid this, by the way.
fastasucan 1y ago
Can you share a little bit about what makes you form opinions when you are not even using the language? I think its fascinating how especially discussions about typing makes people shake their fists against a language they don't even use - and like your post make up some contrived example. >I think that the goal there is to understand that even with the best typing tools, you will have troubles, unless you start by establishing good practices. Like - what makes you think that python developers doesn't understand stuff about Python, when they are actively using the language as opposed to you?
- suspended_state 1y ago
  Indeed, I'm not a regular Python practitioner. I had to use it from time to time because it's the language chosen by the tools I happened to use at that time, like Blender, or Django. In the former case, it wasn't very enjoyable (which says a lot about my skills in that area, or rather lack thereof), while in the latter case I found it quite likeable. So that's my background as far as python goes. I must admit that I largely prefer static typing, which is why I got interested in that article. It's true that trying to shoehorn this feature in the Python ecosystem is an uphill battle: there's a lot of good engineering skill spent on this. Perhaps there's a connection to make between this situation and an old theorem about incompleteness? https://copilot.microsoft.com/shares/2LpT2HFBa3m6jYxUhk9fW https://copilot.microsoft.com/shares/2LpT2HFBa3m6jYxUhk9fW (was generated in quick mode, so you might want to double check).
Groxx 1y ago
To try to tl;dr that rather long post: > When you add type hints to your library's arguments, you're going to be bitten by Hyrum's Law and you are not prepared to accurately type your full universe of users That's understandable. But they're making breaking changes, and those are just breaking change pains - it's almost exactly the same if they had instead done this: def slow_add(a, b): throw TypeError if !isinstance(a, int) ... but anyone looking at that would say "well yeah, that's a breaking change, of course people are going to complain". The only real difference here is that it's a developer-breaking change, not a runtime-breaking one, because Python does not enforce type hints at runtime. Existing code will run, but existing tools looking at the code will fail. That offers an easier workaround (just ignore it), but is otherwise just as interruptive to developers because the same code needs to change in the same ways. --- In contrast! Libraries can very frequently add types to their return values and it's immediately useful to their users. You're restricting your output to only the values that you already output - essentially by definition, only incorrect code will fail when you do this.
flanked-evergl 1y ago
As someone who has been writing python for years the worst mistake I have ever seen people make is not add type hints and not using a type checker.
- stavros 1y ago
  Also not creating custom, expressive Pydantic types and using nested dicts in places. Nested dicts suck, you never know what you're getting, and it's well worth the time converting them to classes.
  - mejutoco 1y ago
    TypedDicts or data classes are both a good idea.

mhh__ 1y ago

At this point I'm fairly convinced that the effort one would spend trying to typecheck a python program is better spent migrating away from python into a language that has a proper type system, then using interop so you can still have the bits/people that need python be in python. Obviously that isn't always possible but you can spend far too long trying to make python work.

davedx 1y ago
Unfortunately with us being in the middle of the AI hype cycle, everyone and their dog is currently busy migrating to python.
- hu3 1y ago
  I don't see why AI hype means more Python code. State of the art AI models are all closed source and accessible through an API anyways. APIs that any language can easily access. AAa for AI model development in itself, yes it's mostly Pyython, but niche.
  - davedx 1y ago
    Because everyone and their dog thinks they need to be ready to develop their own models, in python. This is honestly a thing, at least in the startup world.
  - mzl 1y ago
    I think you are underestimating the massive amounts of Python code that is built around these things. Also, a lot of businesses are not really interested in using an API for an LLM, instead they will modify and fine-tune their own models and deploy in their own data-centers (virtual or physical), and that means even more Python code. Sure, a system that only relies on token factory LLM APIs can be written in any language, but that is not the full width and breadth of the AI hype.
    - hu3 1y ago
      > Also, a lot of businesses are not really interested in using an API for an LLM, instead they will modify and fine-tune their own models and deploy in their own data-centers You realize model training cost millions right? "a lot of businesses" doesn't pass sniff test here. I'm not even counting the large swaths of data required to train. And the expensive specialists. And then you'll have to retrain outdated models every so often. There's a reason that AI has only a handful of players delivering SoTA models and these players are all worth $5B+.
      - mzl 1y ago
        SOTA LLM model training costs a lot, yes. But fine-tuning and training of smaller models is a lot cheaper. I've trained useful vision-models that delivered business value for industrial applications on a MacBook overnight.
J_Shelby_J 1y ago
Six month into learning to build a modern python app, with linters, type systems, tests, venvs, package managers, etc… I realized that the supposed difficulty of rust is drastically less than coming to speed and then keeping up with the python “at scale” ecosystem.
- mdaniel 1y ago
  My strong suspicion is that such a story depends a great deal upon the personalities of the developers much more than any {tool chaos + type chaos} --- {new syntax + lifespan annotations} spectrum
bjackman 1y ago
I think you're forgetting how easy type annotation is. I occasionally spend like 2h working on some old python code. I will spend say 15 minutes of that time adding type annotations (sometimes requires some trivial refactoring). This has an enormous ROI, the cost is so low and the benefit is so immediate. In these cases migrating code to a proper language and figuring out interop is not on my radar, it would be insane. So having the option to get some best-effort type safety is absolutely fantastic. I can definitely see your point, it's a useful analysis for projects under heavy development. But if you have a big Python codebase that basically just works and only sees incremental changes, adding type annotations is a great strategy.
guappa 1y ago
If you do that you need to compile, which means you can't just distribute a text file with your python program. You need a build infrastructure for every python version, every architecture and every OS. Have fun with that!
rtpg 1y ago
I don't understand this point at all. I've worked on Django codebases which have a huge set of typing problems... and while it's not 100% I get a lot of value out of type checking. You annotate enough functions and you get a really good linter out of it!
ramraj07 1y ago
If you're supposedly good at software and you spent too long trying to make python work consider the possibility that you're not good at software? Python has flaws and big ones at that, but there's a reason it's popular. Especially with tools like pydantic and fastapi and uv (and streamlit) you can do insane things in hours what would take weeks and months before. Not to mention how good AI is at generating code in these frameworks. I especially like typing using pydantic, any method is now able to dump and load data from files and dbs and you get extremely terse validated code. Modern IDEs also make quick work of extracting value even from partially typed code. I'd suggest you just open your mind up to imperfect things and give them a shot.

kccqzy 1y ago

> Python, AFAIK, has many features, a very permissive runtime, and perhaps (not unlike C++) only some limited subset should be used at any time to ensure that the code is manageable. Unfortunately, that subset is probably different depending on who you ask, and what you aim to do. I'll get started on the subset of Python that I personally do not wish to use in my own codebase: meta classes, descriptors, callable objects using __call__, object.__new__(cls), names trigger the name mangling rules, self.__dict__. In my opinion, all of the above features involve too much magic and hinder code comprehension.

kstrauser 1y ago
There's a time and a place for each of them: * Meta classes: You're writing Pydantic or an ORM. * Descriptors: You're writing Pydantic or an ORM. * Callable objects: I've used these for things like making validators you initialize with their parameters in one place, then pass them around so other functions can call them. I'd probably just use closures if at all possible now. * object.__new__: You're writing Pydantic or an ORM. * Name mangling: I'm fine with using _foo and __bar where appropriate. Those are nice. Don't ever, ever try to de-mangle them or I'll throw a stick at you. * self.__dict__: You're writing Pydantic or an ORM, although if you use this as shorthand for "doing things that need introspection", that's a useful skill and not deep wizardry. Basically, you won't need those things 99.99% of the time. If you think you do, you probably don't. If you're absolutely certain you do, you might. It's still good and important to understand what they are, though. Even if you never write them yourself, at some point you're going to want to figure out why some dependency isn't working the way you expected, and you'll need to read and know what it's doing.
- kccqzy 1y ago
  > Basically, you won't need those things 99.99% of the time That's kind of my point. If you don't need a language feature 99.99% of the time perhaps it is better to cut it out from your language altogether. Well unless your language is striving to have the same reputation as C++. In Python's case here's a compromise: such features can only be used in a Python extension in C code, signifying their magic nature.

mjr00 1y ago

The top comment in that post shuts down the whole nonsense pretty quickly and firmly: > If you have a super-generic function like that and type hinting enforced, you just use Any and don't care about it. It's a stupid example, but even within the context of a `slow_add` function in a library: maybe the author originally never even thought people would pass in non-numeric values, so in the next version update instead of a hardcoded `time.sleep(0.1)` they decide to `time.sleep(a / b)`. Oops, now it crashes for users who passed in strings or tuples! If only there were a way to declare that the function is only intended to work with numeric values, instead of forcing yourself to provide backwards compatibility for users who used that function in unexpected ways that happened to work. IMO: for Python meant to run non-interactively with any sort of uptime guarantees, type checking is a no-brainer. You're actively making a mistake if you choose to not add type checking.

notatallshaw 1y ago
As the author of that post, I'd like to point out the example was meant to be stupid. The purpose was to show different ideologies and expectations on the same code don't work, such as strict backwards compatibilities, duck typing, and strictly following linting or type hinting rules (due to some arbitrary enforcement). Although re-reading it now I wish I'd spent more than an evening working on it, it's full of issues and not very polished. > If you have a super-generic function like that and type hinting enforced, you just use Any and don't care about it. Following the general stupidness of the post: they are now unable to do that because a security consultant said they have to enable and can not break RUFF rule ANN401: https://docs.astral.sh/ruff/rules/any-type/ https://docs.astral.sh/ruff/rules/any-type/
- mjr00 1y ago
  > Following the general stupidness of the post: they are now unable to do that because a security consultant said they have to enable and can not break RUFF rule ANN401: https://docs.astral.sh/ruff/rules/any-type/ https://docs.astral.sh/ruff/rules/any-type/ Okay, then your function which is extremely generic and needs to support 25 different use cases needs to have an insane type definition which covers all 25 use cases. This isn't an indictment of the type system, this is an indictment of bad code. Don't write functions that support hundreds of input data types, most of which are unintended. Type systems help you avoid this, by the way.

fastasucan 1y ago

Can you share a little bit about what makes you form opinions when you are not even using the language? I think its fascinating how especially discussions about typing makes people shake their fists against a language they don't even use - and like your post make up some contrived example. >I think that the goal there is to understand that even with the best typing tools, you will have troubles, unless you start by establishing good practices. Like - what makes you think that python developers doesn't understand stuff about Python, when they are actively using the language as opposed to you?

suspended_state 1y ago
Indeed, I'm not a regular Python practitioner. I had to use it from time to time because it's the language chosen by the tools I happened to use at that time, like Blender, or Django. In the former case, it wasn't very enjoyable (which says a lot about my skills in that area, or rather lack thereof), while in the latter case I found it quite likeable. So that's my background as far as python goes. I must admit that I largely prefer static typing, which is why I got interested in that article. It's true that trying to shoehorn this feature in the Python ecosystem is an uphill battle: there's a lot of good engineering skill spent on this. Perhaps there's a connection to make between this situation and an old theorem about incompleteness? https://copilot.microsoft.com/shares/2LpT2HFBa3m6jYxUhk9fW https://copilot.microsoft.com/shares/2LpT2HFBa3m6jYxUhk9fW (was generated in quick mode, so you might want to double check).

Groxx 1y ago

To try to tl;dr that rather long post: > When you add type hints to your library's arguments, you're going to be bitten by Hyrum's Law and you are not prepared to accurately type your full universe of users That's understandable. But they're making breaking changes, and those are just breaking change pains - it's almost exactly the same if they had instead done this: def slow_add(a, b): throw TypeError if !isinstance(a, int) ... but anyone looking at that would say "well yeah, that's a breaking change, of course people are going to complain". The only real difference here is that it's a developer-breaking change, not a runtime-breaking one, because Python does not enforce type hints at runtime. Existing code will run, but existing tools looking at the code will fail. That offers an easier workaround (just ignore it), but is otherwise just as interruptive to developers because the same code needs to change in the same ways. --- In contrast! Libraries can very frequently add types to their return values and it's immediately useful to their users. You're restricting your output to only the values that you already output - essentially by definition, only incorrect code will fail when you do this.

flanked-evergl 1y ago

As someone who has been writing python for years the worst mistake I have ever seen people make is not add type hints and not using a type checker.

stavros 1y ago
Also not creating custom, expressive Pydantic types and using nested dicts in places. Nested dicts suck, you never know what you're getting, and it's well worth the time converting them to classes.
- mejutoco 1y ago
  TypedDicts or data classes are both a good idea.