4 ms·

Lots of very smart people have worked very hard on Python tools written in Python, yet the rust rewrites of those tools are so much faster. Sometimes it really

Lots of very smart people have worked very hard on Python tools written in Python, yet the rust rewrites of those tools are so much faster. Sometimes it really is the programming language.

anyfoo 2y ago
Choosing the right algorithm effectively means optimizing runtime complexity. Then, once runtime complexity is fixed with the right algorithm, you're still left with a lot of constant factors that O-notation deliberately ignores (it's only about growth of the runtime). Sometimes, optimizing those constant factors can be significant, and then the choice of language matters. And even some details about the CPU you are targeting, and overall system architecture.
- o11c 2y ago
  Often languages like Javascript and Python don't allow optimal runtime complexity, because the types baked in to external interfaces fundamentally disallow the desired operation. And these languages are too slow to rewrite the core logic in the language itself. (but of course, the vast majority of the code, even in widely used tools, isn't properly designed for optimization in the first place) I only dabble in javascript, but `tsc` is abominable.
charrondev 2y ago
In the JavaScript world a lot of speed up comes from 3 major things as far as I can tell: - easier concurrency. - the fact that things are actually getting rewritten with the purpose of speeding them up. - a lot of the JS tooling getting speedups deals with heavily with string parsing, tokenizing, generating and manipulation of ASTs. Being able to have shared references to slices of strings, carefully manage when strings are copied, and have strict typing of the AST nodes you enable things to be much faster than JavaScript.
jvanderbot 2y ago
This is a very nice counterexample, but it's not actually a counter example without an example. Also, this was a thing before Rust. I've rewritten several things in C or Cpp for python back ends, and most pytbon performance-critical code is already an API to a shared library. You'd be surprised to run OR tools and find Fortran libraries loaded by your python code.
- runesoerensen 2y ago
  Ruff is one example https://astral.sh/ruff https://astral.sh/ruff
  - aragilar 2y ago
    But can I write plugins for it? My understanding it is only implements a subset of the common plugins (and does not do any of the linting that pylint is useful for), so it avoids scanning the filesystem for plugins?
jampekka 2y ago
Python is really really slow compared to JS though.
- gaganyaan 2y ago
  CPython is (though it's slowly getting better). Pypy is amazingly fast
- zeroonetwothree 2y ago
  I once worked on a Python system that had 50 machines dedicated to it. We were able to rewrite it in a more performant language such that it easily ran on one machine. This also allowed us to avoid all the issues distributed systems have. So yeah, Python is not great for systems programming
- kukkamario 2y ago
  Node is so slow to start that python script can complete before Javascript even begins to execute.
  - jampekka 2y ago
    For extremely simple scripts maybe. I get around 70 ms difference in startup time. $ time python3 -c "print('Hello world')" Hello world real 0m0.017s $ time node -e "console.log('Hello world')" Hello world real 0m0.084s
worik 2y ago
> Lots of very smart people have worked very hard on Python tools written in Python Yes, I agree that is very sad Python is achingly slow. I know the Python people want to address this, I do not understand. Python makes sense as a scripting/job control language, and execution speed does not matter. As an application development language it is diabolical. For a lot of reasons, not just speed
bsder 2y ago
> Lots of very smart people have worked very hard on Python tools written in Python, yet the rust rewrites of those tools are so much faster. So? Some tool got written and did its job sufficiently well that it became a bottleneck worth optimizing. That's a win. "Finishing the task" is, by far, the most difficult thing in programming. And the two biggest contributors to that are 1) simplicity of programming language and 2) convenience of ecosystem. Python and Javascript are so popular because they tick both boxes.
- tyree731 2y ago
  Don’t disagree about finishing the task, but personally I don’t find more performant languages any less productive for the sort of programming I tend to do.
  - bsder 2y ago
    Congratulations on being a programming god. This discussion isn't for you. From my point of view, I'm happy if I can convince my juniors to learn a scripting language. Okay? I don't care which one--any one. I'd prefer that they learn one of the portable ones but even PowerShell is fine. I have seen sooooo many junior folks struggle for days to do something that is 10 lines in any scripting language. Those folks who program but don't know a scripting language far outnumber the rest of us.
    - fredrikholm 2y ago
      > I have seen sooooo many junior folks struggle for days to do something that is 10 lines in any scripting language. > Those folks who program but don't know a scripting language far outnumber the rest of us. What domain are you in? This sounds like the complete inverse of every company I've ever worked at. Entire products are built on Python, Node ect, and the time after the initial honeymoon phase (if it exists) is spent retrofitting types on top in order to get a handle, any handle, on the complexity that arises without static analysis and compile time errors. At around the same time, services start OOM'ming left and right, parallellism=1 becomes a giant bottleneck, JIT fails in one path bringing the service performance down an order of magnitude every now and then etc... > Congratulations on being a programming god. This discussion isn't for you. On the behalf of mediocre developers everywhere, a lot of us prefer statically typed languages because we are mediocre; I cannot hold thousands of implicit types and heuristics in my head at the same time. Luckily, the type system can.

Lots of very smart people have worked very hard on Python tools written in Python, yet the rust rewrites of those tools are so much faster. Sometimes it really is the programming language.

anyfoo 2y ago

Choosing the right algorithm effectively means optimizing runtime complexity. Then, once runtime complexity is fixed with the right algorithm, you're still left with a lot of constant factors that O-notation deliberately ignores (it's only about growth of the runtime). Sometimes, optimizing those constant factors can be significant, and then the choice of language matters. And even some details about the CPU you are targeting, and overall system architecture.

o11c 2y ago
Often languages like Javascript and Python don't allow optimal runtime complexity, because the types baked in to external interfaces fundamentally disallow the desired operation. And these languages are too slow to rewrite the core logic in the language itself. (but of course, the vast majority of the code, even in widely used tools, isn't properly designed for optimization in the first place) I only dabble in javascript, but `tsc` is abominable.

charrondev 2y ago

In the JavaScript world a lot of speed up comes from 3 major things as far as I can tell: - easier concurrency. - the fact that things are actually getting rewritten with the purpose of speeding them up. - a lot of the JS tooling getting speedups deals with heavily with string parsing, tokenizing, generating and manipulation of ASTs. Being able to have shared references to slices of strings, carefully manage when strings are copied, and have strict typing of the AST nodes you enable things to be much faster than JavaScript.

jvanderbot 2y ago

This is a very nice counterexample, but it's not actually a counter example without an example. Also, this was a thing before Rust. I've rewritten several things in C or Cpp for python back ends, and most pytbon performance-critical code is already an API to a shared library. You'd be surprised to run OR tools and find Fortran libraries loaded by your python code.

runesoerensen 2y ago
Ruff is one example https://astral.sh/ruff https://astral.sh/ruff
- aragilar 2y ago
  But can I write plugins for it? My understanding it is only implements a subset of the common plugins (and does not do any of the linting that pylint is useful for), so it avoids scanning the filesystem for plugins?

jampekka 2y ago

Python is really really slow compared to JS though.

gaganyaan 2y ago
CPython is (though it's slowly getting better). Pypy is amazingly fast
zeroonetwothree 2y ago
I once worked on a Python system that had 50 machines dedicated to it. We were able to rewrite it in a more performant language such that it easily ran on one machine. This also allowed us to avoid all the issues distributed systems have. So yeah, Python is not great for systems programming
kukkamario 2y ago
Node is so slow to start that python script can complete before Javascript even begins to execute.
- jampekka 2y ago
  For extremely simple scripts maybe. I get around 70 ms difference in startup time. $ time python3 -c "print('Hello world')" Hello world real 0m0.017s $ time node -e "console.log('Hello world')" Hello world real 0m0.084s

worik 2y ago

> Lots of very smart people have worked very hard on Python tools written in Python Yes, I agree that is very sad Python is achingly slow. I know the Python people want to address this, I do not understand. Python makes sense as a scripting/job control language, and execution speed does not matter. As an application development language it is diabolical. For a lot of reasons, not just speed

bsder 2y ago

> Lots of very smart people have worked very hard on Python tools written in Python, yet the rust rewrites of those tools are so much faster. So? Some tool got written and did its job sufficiently well that it became a bottleneck worth optimizing. That's a win. "Finishing the task" is, by far, the most difficult thing in programming. And the two biggest contributors to that are 1) simplicity of programming language and 2) convenience of ecosystem. Python and Javascript are so popular because they tick both boxes.

tyree731 2y ago
Don’t disagree about finishing the task, but personally I don’t find more performant languages any less productive for the sort of programming I tend to do.
- bsder 2y ago
  Congratulations on being a programming god. This discussion isn't for you. From my point of view, I'm happy if I can convince my juniors to learn a scripting language. Okay? I don't care which one--any one. I'd prefer that they learn one of the portable ones but even PowerShell is fine. I have seen sooooo many junior folks struggle for days to do something that is 10 lines in any scripting language. Those folks who program but don't know a scripting language far outnumber the rest of us.
  - fredrikholm 2y ago
    > I have seen sooooo many junior folks struggle for days to do something that is 10 lines in any scripting language. > Those folks who program but don't know a scripting language far outnumber the rest of us. What domain are you in? This sounds like the complete inverse of every company I've ever worked at. Entire products are built on Python, Node ect, and the time after the initial honeymoon phase (if it exists) is spent retrofitting types on top in order to get a handle, any handle, on the complexity that arises without static analysis and compile time errors. At around the same time, services start OOM'ming left and right, parallellism=1 becomes a giant bottleneck, JIT fails in one path bringing the service performance down an order of magnitude every now and then etc... > Congratulations on being a programming god. This discussion isn't for you. On the behalf of mediocre developers everywhere, a lot of us prefer statically typed languages because we are mediocre; I cannot hold thousands of implicit types and heuristics in my head at the same time. Luckily, the type system can.