6 ms·

I maintain Python code bases for a living, and feel that the language has simply been pushed too far. Static typing in Python doesn't give you the advantage of

I maintain Python code bases for a living, and feel that the language has simply been pushed too far. Static typing in Python doesn't give you the advantage of actually static typing and even IDE support is -- well it's not terrible, just not great. The thing is, once we go through all this static typing exercise in Python, we get no performance advantages, and the whole thing seems bolted on, with worse semantics than most modern typed languages. And yes, it can stifle undeniable advantages a dynamic language like Python can provide. Python remains a fantastic prototyping and scripting language, but my feeling is that it now handles more than it should.

yummypaint 4y ago
I routinely use cython to compile python for heavy workloads. A big part of the ~10x speedup i usually see comes from strategically assigning true static types to certain variables that get frequently iterated or compared. Most variables are left as standard python as it isn't necessary to change them for performance. My experience has been that cython "just works" even with lots of external libraries etc, and the code ends up performing as well as any c++ code i could realistically write while being much more understandable.
- nextaccountic 4y ago
  > I routinely use cython to compile python for heavy workloads. an alternative is rust + pyo3 https://pyo3.rs https://pyo3.rs here's a web framework written with it, https://robyn.tech https://robyn.tech the other poster child for pyo3 is polars, https://www.pola.rs https://www.pola.rs it's simply, amazing.
  - samstave 4y ago
    which of these links should I start with with zero xp?
    - nextaccountic 4y ago
      If you don't know Rust, but know Python, you can install Python libraries written in Rust with pip. Like, pip install polars or pip install robyn. In this case you follow the two bottom links. But then you don't write your own libraries and stuff so.. I guess that's not what you want. If creating your own Python libraries in Rust is what you want, you would check out the first link I sent only, the one for pyo3. But, if you want to learn Rust, you probably wouldn't start out with pyo3. You first install Rust with https://rustup.rs/ https://rustup.rs/ and then check out the official book, and the book rust by example, that you can find here https://www.rust-lang.org/learn https://www.rust-lang.org/learn - and maybe write some code on the Rust playground https://play.rust-lang.org/ https://play.rust-lang.org/ - then, you use pyo3 to build Python libraries in Rust, and then use maturin https://www.maturin.rs/ https://www.maturin.rs/ to build and publish them to Pypi. But if you still prefer to begin with Rust by writing Python libraries (it's a valid strategy if you are very comfortable with working with multiple stacks), the Maturin link has a tutorial that setups a program that is half written in python, half written in Rust, https://www.maturin.rs/tutorial.html https://www.maturin.rs/tutorial.html (well the pyo3 link I sent also has one too. You should refer to the documentation of both, because you will use the two together) After learning Rust and building some stuff with pyo3, the next step is looking for libraries that you could leverage to make Python programs ultra fast. Here https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon is an obvious choice, see some examples from the Rust cookbook https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html https://rust-lang-nursery.github.io/rust-cookbook/concurrenc... - when you create a parallel iterator, it will distribute the processing to many threads (by default, one per core). The rust cookbook, by the way, is a nice reference to see some of the most used crates (Rust libraries) in the Rust ecosystem. If you are doing async stuff in Python, you probably need to look up https://github.com/awestlake87/pyo3-asyncio https://github.com/awestlake87/pyo3-asyncio and https://tokio.rs/ https://tokio.rs/ - that's two libraries that Robyn uses (see Robyn dependencies here https://github.com/sansyrox/robyn/blob/main/Cargo.toml https://github.com/sansyrox/robyn/blob/main/Cargo.toml) Anyway there are some posts about pyo3 on the web, like this blog post https://boring-guy.sh/posts/river-rust/ https://boring-guy.sh/posts/river-rust/ (note: it uses an outdated version of pyo3, and doesn't seem to use maturin which is a newer tool). This post was written by the developers of https://github.com/online-ml/river https://github.com/online-ml/river - another Python library written in Rust
  - dereg 4y ago
    I'm so glad I found Polars. I replaced the most time-consuming parts of my Pandas code with Polars and it has reduced data manipulation times by literally 90-95%. Amazing is not an overstatement.
cameroncairns 4y ago
There have been some efforts to utilize the type hints to give performance boosts. There's a project called mypyc that apparently has been used by black (python formatting library) that will compile type hinted python into c extensions. Unfortunately I think development has stalled, but as more people start using type hints I think there will be more motivation for similar projects.
SAI_Peregrinus 4y ago
The problem with separate prototyping languages is that the most permanent fix is always a temporary one. The prototype usually becomes the final product. If your language isn't suitable for use as a final product, then you're going to have a problem. If it's not suitable for writing a prototype, then you already have a problem.
- dunefox 4y ago
  That's why I love the idea of a language like Common Lisp. Performance, 'debugability', and flexibility in one package.
wefarrell 4y ago
That's also true of Typescript, which has been wildly successful and taken over the javascript ecosystem. The main advantage of types is that it makes your code more maintainable by adding guardrails. You'll still possible run into issues at runtime and they're not perfect, but they're better than not having types at all.
- scrollaway 4y ago
  Typescript really rarely feels bolted on to JavaScript. That also has to do with typescript introducing new syntax to JS.
  - goodoldneon 4y ago
    There's very little new syntax added by TS
- IshKebab 4y ago
  Yeah the difference is that Typescript has a single good implementation (Python has two and the worst one is the most popular), and people actually use Typescript, so your dependencies have types. I've also found that Python developers tend to write highly dynamic code that is difficult to statically type, whereas JavaScript developers - even ones that don't use Typescript - have mostly realised that that is a bad idea (with some exceptions cough Vue).
abdullahkhalids 4y ago
> the whole thing seems bolted on, with worse semantics than most modern typed languages One of the biggest sources of ugliness is the "None". The standard way of declaring variables ahead of time is setting them to None. But then, the type hints just become these ugly unreadable Optional[ActualTypeofVar] everywhere.
- mjr00 4y ago
  That's because your type is Optional[TypeOfVar] though; if you try using that variable in a place that expects just TypeOfVar before it's set, you should get an error.
  - joshuamorton 4y ago
    Right, in langs with c-style scoping, this is an issue, but in python if foo: x = a else: x = b Works, so there's no need to predeclare.
    - abdullahkhalids 4y ago
      A very common pattern in python is to declare all instance variables of a class in the __init__ method by setting them to None. This is done for * code readability; all relevant variables are in one place * Python is a prototyping language, and a very common usage pattern is doing dir(object) to determine the names of all variables that could be set to something.
      - joshuamorton 4y ago
        This is no longer needed with modern python, you can use class Foo: my_int: int To handle this. That said, for the second case, a variable that may be set to something but may not be is optional! That's correct behavior!
  - abdullahkhalids 4y ago
    Let me restate. To prevent ugliness in typing, either * Python should have a way of declaring variables without setting them to None, so we can type hint them to their actual types. Pre-declaration is very important for a prototyping language, and needs to be done in a way that the variable is visible to dir(). * Come up with a cleaner syntax than the wordy Optional[]. Thankfully, they have recently moved on from the ugly Union[a,b] to the more readable a | b. Something more readable could be done for Optional.
lostmsu 4y ago
> Python remains a fantastic prototyping and scripting language I think it sucks for prototyping too if your prototype is constantly evolving. I have a WIP with about 100 files in it that I've been aggressively evolving over the past year, and I already want to rewrite everything in something statically typed, because any kind of change became pain in the ass. Too bad the project requires Python-first libraries.
- ReflectedImage 4y ago
  Large Python projects are done with microservices typically.
  - lostmsu 4y ago
    1. I would not call 100 short files 1/3 of which are simple tests "large". I've handled 10 times larger C# projects without any hiccups. 2. This is a deep learning research project. Files are layers, pure functions, and model builders. Can't really turn it into microservices.
KRAKRISMOTT 4y ago
I think the issue with Python is that it is relatively underfunded compared to other languages of its size. The entire Python ecosystem lacks leadership. Packaging is a clusterfuck, Pypa today still doesn't contain package metadata (which massively slows down downstream resolvers like poetry and wastes terabytes of bandwidth every year). Non-security issues take forever to resolve. Nothing encourages parties to contribute, unlike Go or Rust. You end up with companies having their own subtly-imcompatible internal forks. The benevolent dictator never cared about performance until recently. Most of Python's changes have been simple syntactic sugar. Python as a compiler engineering project is an exercise in mediocrity compared to V8 or LuaJIT.
phlakaton 4y ago
I think the mistake is to assume that Python's type hints form a complete static typing system. Of course they fall short if you think that's what they're supposed to be. I think of it instead as a sometimes-useful subset of checks that you might get from a static typing system, and I use it in those places in my projects where I think it will be the most useful. The very fact that I can think of it as an option I can enable puts it in a totally different space from traditional static typing systems.
- arcturus17 4y ago
  Yea this is exactly how it feels to me. Like “reinforced dynamic”.
deleted 4y ago
[deleted]
ImprobableTruth 4y ago
Honestly, static typing encouraging data shape documentation alone makes it worth it to me. Yes, they're pretty hacky in Python and don't feel great, but I'm sure it'll continue to improve. I'm just so over having to guess what a function might accept or return. Life is too short to spend it constantly reverse engineering code because people can't be arsed to write proper documentation.
- mountainriver 4y ago
  This would sum up my sentiment as well, just for documentation reasons alone it is wildly beneficial. People who love dynamic typing seem to think their code is just “understandable” and it almost always isn’t
zerodensity 4y ago
I also maintain a large python codebase at work, typing makes things simpler. For example using pydantic 90% of serialization code just goes away. Mypy eliminates a large number of bugs. Also when working with multiple people in the same codebase typing makes intent so much clearer. The type system itself is not perfect but it's a clear improvement of not having it at all.
- arcturus17 4y ago
  I could not agree more. I’m fairly language agnostic but typed Python with pydantic is a joy to work with.
- ReflectedImage 4y ago
  I disagree, it makes things more complicated by bringing in concepts such as generics. Using MyPy will on average double your bug count per software feature. You might not think it does that. But if you actually go and measure it, the bugs go through the roof. There are studies on this. It's for subtle reasons revolving around where bugs come from in code.
- lordgroff 4y ago
  It's a net benefit. I just think that we've stretched Python into all kinds of directions which should be served by actually statically typed languages.
codexb 4y ago
Rust and Typescript are some of the most well-loved languages, and the typing semantics of python are almost identical. Have you used VSCode? or PyCharm? Their typehinting support is pretty great. At least as good as anything for typescript.

yummypaint 4y ago

I routinely use cython to compile python for heavy workloads. A big part of the ~10x speedup i usually see comes from strategically assigning true static types to certain variables that get frequently iterated or compared. Most variables are left as standard python as it isn't necessary to change them for performance. My experience has been that cython "just works" even with lots of external libraries etc, and the code ends up performing as well as any c++ code i could realistically write while being much more understandable.

nextaccountic 4y ago
> I routinely use cython to compile python for heavy workloads. an alternative is rust + pyo3 https://pyo3.rs https://pyo3.rs here's a web framework written with it, https://robyn.tech https://robyn.tech the other poster child for pyo3 is polars, https://www.pola.rs https://www.pola.rs it's simply, amazing.
- samstave 4y ago
  which of these links should I start with with zero xp?
  - nextaccountic 4y ago
    If you don't know Rust, but know Python, you can install Python libraries written in Rust with pip. Like, pip install polars or pip install robyn. In this case you follow the two bottom links. But then you don't write your own libraries and stuff so.. I guess that's not what you want. If creating your own Python libraries in Rust is what you want, you would check out the first link I sent only, the one for pyo3. But, if you want to learn Rust, you probably wouldn't start out with pyo3. You first install Rust with https://rustup.rs/ https://rustup.rs/ and then check out the official book, and the book rust by example, that you can find here https://www.rust-lang.org/learn https://www.rust-lang.org/learn - and maybe write some code on the Rust playground https://play.rust-lang.org/ https://play.rust-lang.org/ - then, you use pyo3 to build Python libraries in Rust, and then use maturin https://www.maturin.rs/ https://www.maturin.rs/ to build and publish them to Pypi. But if you still prefer to begin with Rust by writing Python libraries (it's a valid strategy if you are very comfortable with working with multiple stacks), the Maturin link has a tutorial that setups a program that is half written in python, half written in Rust, https://www.maturin.rs/tutorial.html https://www.maturin.rs/tutorial.html (well the pyo3 link I sent also has one too. You should refer to the documentation of both, because you will use the two together) After learning Rust and building some stuff with pyo3, the next step is looking for libraries that you could leverage to make Python programs ultra fast. Here https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon is an obvious choice, see some examples from the Rust cookbook https://rust-lang-nursery.github.io/rust-cookbook/concurrency/parallel.html https://rust-lang-nursery.github.io/rust-cookbook/concurrenc... - when you create a parallel iterator, it will distribute the processing to many threads (by default, one per core). The rust cookbook, by the way, is a nice reference to see some of the most used crates (Rust libraries) in the Rust ecosystem. If you are doing async stuff in Python, you probably need to look up https://github.com/awestlake87/pyo3-asyncio https://github.com/awestlake87/pyo3-asyncio and https://tokio.rs/ https://tokio.rs/ - that's two libraries that Robyn uses (see Robyn dependencies here https://github.com/sansyrox/robyn/blob/main/Cargo.toml https://github.com/sansyrox/robyn/blob/main/Cargo.toml) Anyway there are some posts about pyo3 on the web, like this blog post https://boring-guy.sh/posts/river-rust/ https://boring-guy.sh/posts/river-rust/ (note: it uses an outdated version of pyo3, and doesn't seem to use maturin which is a newer tool). This post was written by the developers of https://github.com/online-ml/river https://github.com/online-ml/river - another Python library written in Rust
- dereg 4y ago
  I'm so glad I found Polars. I replaced the most time-consuming parts of my Pandas code with Polars and it has reduced data manipulation times by literally 90-95%. Amazing is not an overstatement.

cameroncairns 4y ago

There have been some efforts to utilize the type hints to give performance boosts. There's a project called mypyc that apparently has been used by black (python formatting library) that will compile type hinted python into c extensions. Unfortunately I think development has stalled, but as more people start using type hints I think there will be more motivation for similar projects.

SAI_Peregrinus 4y ago

The problem with separate prototyping languages is that the most permanent fix is always a temporary one. The prototype usually becomes the final product. If your language isn't suitable for use as a final product, then you're going to have a problem. If it's not suitable for writing a prototype, then you already have a problem.

dunefox 4y ago
That's why I love the idea of a language like Common Lisp. Performance, 'debugability', and flexibility in one package.

wefarrell 4y ago

That's also true of Typescript, which has been wildly successful and taken over the javascript ecosystem. The main advantage of types is that it makes your code more maintainable by adding guardrails. You'll still possible run into issues at runtime and they're not perfect, but they're better than not having types at all.

scrollaway 4y ago
Typescript really rarely feels bolted on to JavaScript. That also has to do with typescript introducing new syntax to JS.
- goodoldneon 4y ago
  There's very little new syntax added by TS
IshKebab 4y ago
Yeah the difference is that Typescript has a single good implementation (Python has two and the worst one is the most popular), and people actually use Typescript, so your dependencies have types. I've also found that Python developers tend to write highly dynamic code that is difficult to statically type, whereas JavaScript developers - even ones that don't use Typescript - have mostly realised that that is a bad idea (with some exceptions cough Vue).

abdullahkhalids 4y ago

> the whole thing seems bolted on, with worse semantics than most modern typed languages One of the biggest sources of ugliness is the "None". The standard way of declaring variables ahead of time is setting them to None. But then, the type hints just become these ugly unreadable Optional[ActualTypeofVar] everywhere.

mjr00 4y ago
That's because your type is Optional[TypeOfVar] though; if you try using that variable in a place that expects just TypeOfVar before it's set, you should get an error.
- joshuamorton 4y ago
  Right, in langs with c-style scoping, this is an issue, but in python if foo: x = a else: x = b Works, so there's no need to predeclare.
  - abdullahkhalids 4y ago
    A very common pattern in python is to declare all instance variables of a class in the __init__ method by setting them to None. This is done for * code readability; all relevant variables are in one place * Python is a prototyping language, and a very common usage pattern is doing dir(object) to determine the names of all variables that could be set to something.
    - joshuamorton 4y ago
      This is no longer needed with modern python, you can use class Foo: my_int: int To handle this. That said, for the second case, a variable that may be set to something but may not be is optional! That's correct behavior!
- abdullahkhalids 4y ago
  Let me restate. To prevent ugliness in typing, either * Python should have a way of declaring variables without setting them to None, so we can type hint them to their actual types. Pre-declaration is very important for a prototyping language, and needs to be done in a way that the variable is visible to dir(). * Come up with a cleaner syntax than the wordy Optional[]. Thankfully, they have recently moved on from the ugly Union[a,b] to the more readable a | b. Something more readable could be done for Optional.

lostmsu 4y ago

> Python remains a fantastic prototyping and scripting language I think it sucks for prototyping too if your prototype is constantly evolving. I have a WIP with about 100 files in it that I've been aggressively evolving over the past year, and I already want to rewrite everything in something statically typed, because any kind of change became pain in the ass. Too bad the project requires Python-first libraries.

ReflectedImage 4y ago
Large Python projects are done with microservices typically.
- lostmsu 4y ago
  1. I would not call 100 short files 1/3 of which are simple tests "large". I've handled 10 times larger C# projects without any hiccups. 2. This is a deep learning research project. Files are layers, pure functions, and model builders. Can't really turn it into microservices.

KRAKRISMOTT 4y ago

I think the issue with Python is that it is relatively underfunded compared to other languages of its size. The entire Python ecosystem lacks leadership. Packaging is a clusterfuck, Pypa today still doesn't contain package metadata (which massively slows down downstream resolvers like poetry and wastes terabytes of bandwidth every year). Non-security issues take forever to resolve. Nothing encourages parties to contribute, unlike Go or Rust. You end up with companies having their own subtly-imcompatible internal forks. The benevolent dictator never cared about performance until recently. Most of Python's changes have been simple syntactic sugar. Python as a compiler engineering project is an exercise in mediocrity compared to V8 or LuaJIT.

phlakaton 4y ago

I think the mistake is to assume that Python's type hints form a complete static typing system. Of course they fall short if you think that's what they're supposed to be. I think of it instead as a sometimes-useful subset of checks that you might get from a static typing system, and I use it in those places in my projects where I think it will be the most useful. The very fact that I can think of it as an option I can enable puts it in a totally different space from traditional static typing systems.

arcturus17 4y ago
Yea this is exactly how it feels to me. Like “reinforced dynamic”.

deleted 4y ago

[deleted]

ImprobableTruth 4y ago

Honestly, static typing encouraging data shape documentation alone makes it worth it to me. Yes, they're pretty hacky in Python and don't feel great, but I'm sure it'll continue to improve. I'm just so over having to guess what a function might accept or return. Life is too short to spend it constantly reverse engineering code because people can't be arsed to write proper documentation.

mountainriver 4y ago
This would sum up my sentiment as well, just for documentation reasons alone it is wildly beneficial. People who love dynamic typing seem to think their code is just “understandable” and it almost always isn’t

zerodensity 4y ago

I also maintain a large python codebase at work, typing makes things simpler. For example using pydantic 90% of serialization code just goes away. Mypy eliminates a large number of bugs. Also when working with multiple people in the same codebase typing makes intent so much clearer. The type system itself is not perfect but it's a clear improvement of not having it at all.

arcturus17 4y ago
I could not agree more. I’m fairly language agnostic but typed Python with pydantic is a joy to work with.
ReflectedImage 4y ago
I disagree, it makes things more complicated by bringing in concepts such as generics. Using MyPy will on average double your bug count per software feature. You might not think it does that. But if you actually go and measure it, the bugs go through the roof. There are studies on this. It's for subtle reasons revolving around where bugs come from in code.
lordgroff 4y ago
It's a net benefit. I just think that we've stretched Python into all kinds of directions which should be served by actually statically typed languages.

codexb 4y ago

Rust and Typescript are some of the most well-loved languages, and the typing semantics of python are almost identical. Have you used VSCode? or PyCharm? Their typehinting support is pretty great. At least as good as anything for typescript.