3 ms·

I like Python too but let's face it, it is not performant. Why do we have to pretend otherwise?

wenc 3y ago
I think everybody agrees and is not pretending otherwise. But my thing is, I think the problem could be solved in a different way in Python where we can use performant libraries to get an answer in a reasonable time. The author says the naive Python implementation with a for-loop takes 36 milliseconds per iteration, and the problem requires 2.5 billion iterations (= 2.9 years, which is unreasonable) while optimized Rust takes 8 mins (corrected). I believe we can solve the problem in Python in a reasonable amount of time (not 2.9 years) by expressing it differently. And I believe we can do it in Python without trying to optimize Python operations like the author is doing with Rust. Imagine your boss came up to you and said I need the answer by this week and that you could only use Python, you would need to come up with a way to solve it. I wouldn't start by trying to optimizing Python's for-loop -- I would break out of the loop paradigm altogether and use arrays, database indices, optimized dataframe libraries (probably written in C++ or Rust) to get there. Because Python is not fast -- everyone knows this -- Python programmers will often think of other ways (generally reaching for libraries) to solve the problem.
- bibmbop 3y ago
  8 minutes.
tetsuhamu 3y ago
The industry agreed to standardize on Python for the task of describing compute-graphs that get executed by compute engines implemented in something other than Python. Python is not meant to be used for the computation itself.
- FridgeSeal 3y ago
  And then other people came along, using other languages, and realised we could do the same compute, with a fraction of the hardware, in about the same time, in the languages they’re already using, which really diminishes the appeal of Python, which boils down to: “Setup some infra, and get someone who knows how to operate it, and then write new code in python, and babysit it as it ossifies into a core piece of infrastructure” _or_ optimise it in a language you’re already using, or is semi-common in your stack already, and then move on with things.

wenc 3y ago

I think everybody agrees and is not pretending otherwise. But my thing is, I think the problem could be solved in a different way in Python where we can use performant libraries to get an answer in a reasonable time. The author says the naive Python implementation with a for-loop takes 36 milliseconds per iteration, and the problem requires 2.5 billion iterations (= 2.9 years, which is unreasonable) while optimized Rust takes 8 mins (corrected). I believe we can solve the problem in Python in a reasonable amount of time (not 2.9 years) by expressing it differently. And I believe we can do it in Python without trying to optimize Python operations like the author is doing with Rust. Imagine your boss came up to you and said I need the answer by this week and that you could only use Python, you would need to come up with a way to solve it. I wouldn't start by trying to optimizing Python's for-loop -- I would break out of the loop paradigm altogether and use arrays, database indices, optimized dataframe libraries (probably written in C++ or Rust) to get there. Because Python is not fast -- everyone knows this -- Python programmers will often think of other ways (generally reaching for libraries) to solve the problem.

bibmbop 3y ago
8 minutes.

tetsuhamu 3y ago

The industry agreed to standardize on Python for the task of describing compute-graphs that get executed by compute engines implemented in something other than Python. Python is not meant to be used for the computation itself.

FridgeSeal 3y ago
And then other people came along, using other languages, and realised we could do the same compute, with a fraction of the hardware, in about the same time, in the languages they’re already using, which really diminishes the appeal of Python, which boils down to: “Setup some infra, and get someone who knows how to operate it, and then write new code in python, and babysit it as it ossifies into a core piece of infrastructure” _or_ optimise it in a language you’re already using, or is semi-common in your stack already, and then move on with things.