6 ms·

Yes, Python Is Slow, and I Don’t Care

31 points by zerogvt 8y ago

melling 8y ago
“Python is more than 2x as productive as Java in the above study.“ How about compared to a functional language like Haskell, OCaml, or F#? I agree that we should choose productivity over performance when possible. However, Java, for example, is extremely verbose. My impression is that people who use functional languages are quite productive. Plus, they need fewer unit tests.
- heavenlyblue 8y ago
  "Median hours to solve a problem" The same researcher must write a study of the median temperature at the hospital.
- dual_basis 8y ago
  I'm not sure if people using functional languages are more productive. One of the points Uncle Bob makes in his talk supporting functional programming is that it is harder and slower to write: https://youtu.be/7Zlp9rKHGD4 https://youtu.be/7Zlp9rKHGD4 That said, I have to wonder if these dynamic scripting languages are truly enabling productivity or if it just feels like that because you can write it and try many things quickly. It reminds me of being such in traffic on the highway - sometimes I get off and take back roads even if my GPS tells me it will take longer, because it is more pleasant to be moving than feeling stuck. That said, I think the biggest benefit of dynamic languages is that they are so easy to refactor. You don't have as rigid a structure, so you can try easily hack together some idea that, done "properly", would require a big refactor. Despite being functional, Haskell's strong type system makes refactoring a huge pain. If course the same things enabling this quick refactoring in the dynamic languages works against you when it comes to speed and reliability.
  - ggm 8y ago
    I sit with an FP expert (he might deny it, but he is). What I find, is that he takes a lot longer to get to the solution that I hack out in a day in Python3 (with his assistance most of the time) but his solution has three interesting properties. 1) its usually founded on a very good understanding of the problem, and the abstractions inherent in the problem. If he can re-use the abstractions he has used before he does, to great effect in speed-to-solution 2) it is very usually correct. it is remarkably bug free because the Haskell typing removes a huge amount of ambiguity about the problem going in 3) he has absolutely no problem explaining what it does 3 months or more later. I value these behaviours immensely. My own Python is usually buggy, often re-produces techniques I did a week ago, badly, and I can't explain why I used that lambda/map function.
    - 2wrist 8y ago
      It is really cool when you can find people like this. He sounds class!
      - ggm 8y ago
        He is!
    - gugagore 8y ago
      I'm very curious about what kinds of problems these are. Any chance you can give us an idea?
      - ggm 8y ago
        Typically processing data from logs, analyses of SQL state changes in the absence of good knowledge of the prior states. Some stuff which is aggregation over disparate inputs where a DSL can be useful. A lot of the time radix trees and other prefix/length matching logic is being used. Sometimes we're cross correlating different sources against disparate common keys. Some stuff connecting type: value record structures into JSON. We work a lot on ill-formed requests to refine data previously worked on with new questions. We work a lot on small distributed systems exploring eventual consistency and event sourcing. We work with quite large logfiles collected asynchronously, with poor clock alignment. So data buffering and sorting is not uncommon.
  - michaelcampbell 8y ago
    Faster to write -> faster to market Slower to write (and get right) -> fewer bugs to deal with later Pick your poison
    - coldtea 8y ago
      That's really a no brainer. Having your company survive (because you got something faster to market), and having bugs to deal with later, is a so-called "nice problem to have". The bug-less software of a dead company does nobody good.
      - dTal 8y ago
        Quick 'n' dirty might be fine for a move-fast-and-break-things web startup, but the world is much larger than that. I sincerely hope the people writing the software in, let's say an airliner to choose a topical example, choose option 2. Dealing with bugs that have killed people is not a "nice problem to have". (It depresses me that I have to write a defense of good engineering on Hacker news. The value of doing things correctly should be a tacit core principle of any engineer, to be broken only under circumstances of exceptional duress.)
        coldtea 8y ago
        Most domains are not of the "killing people" type. In those places, they also have other kinds of compliance requirements for the code -- which makes sense: I wouldn't leave the airline software PMs to chose between the above two options.
        dTal 8y ago
        Most domains are also not the "finish the job by next Thursday or the company folds" type either. You're going to have to pay back that technical debt someday, with interest - why not save time and do it right the first time? (Typical SV answer: you're only thinking of the short term because you're trying to get investment and you plan to exit stage left with a fat packet long before the technical debt rears its ugly head, leaving somebody else to clean up the mess. But this is not how most engineering is done in the real world.)
  - nikofeyn 8y ago
    i don't think that's true for multiparadigm functional languages like f#, racket, and others. these languages allow you to solve problems in the best way and don't necessarily bind you to some draconian perspective like haskell. functional doesn't necessarily mean static typing. scheme and racket are functional languages.
    - dual_basis 8y ago
      I have found that much of the benefit of functional programming is in the consistent guarantees it makes, not only about your own code but also others' that you depend on. Languages that try to do it all often leave me feeling disjointed. You end up with some libraries written in a strong OOP style, but then others are written with a functional flavor, and then your own code necessarily has to take on a strange amalgamation of the two.
- pjmlp 8y ago
  Plus all of them provide JIT/AOT compilers.
- deleted 8y ago
  [deleted]
vardump 8y ago
Whether slow code matters or not depends on the scale, because executing code requires power. If it's a one off or otherwise rarely executed, yeah, it doesn't matter much. But it starts to matter when it's constantly running on a significant number of systems. Widely distributed frequently running slow code can waste a lot of energy. Cell phones and laptops heat up and have a shorter battery life. Even desktops require more power. On the global scale, a large number of such systems can waste several nuclear reactors worth of electricity. 10 times slower code might require 10 times the number of servers. Maybe it doesn't matter much when it's just the difference between one and 10 servers. Or perhaps you don't even consume 10% of a single server. But there's a point when the hardware and power costs start to dominate. The scale matters.
mruts 8y ago
But python is both slow and inexpressive. No types, no match statements, no easy FP. Also the language is poorly designed and inconsistent. I’m practically forced to use python for quant finance at my job because of the network effect: numpy, pandas, cvxpy, etc. But the language itself is total garbage compared to, say, Scala or OCaml. And both of those languages are also much faster.
- yvdriess 8y ago
  It is interesting to note that the listed libraries are not written in Python (Cython at best). The moment what you want to do with your data is not covered by the library, you either hack it or you're screwed.
- deogeo 8y ago
  That is my experience as well - the only redeeming feature of the language is the vast selection of easy-to-use libraries. The speed that the language sacrificed isn't some inherent trade-off, but completely orthogonal to what makes the language useful.
  - pjmlp 8y ago
    Exactly, Lisp, Scheme, Dylan, Smalltalk, JavaScript, Prolog are all examples of languages as dynamic as Python, while having good JIT/AOT toolchains available by default.
    - mruts 8y ago
      Python has to be one of the worst languages used today. It literally provides no value except for an abundance of libraries. Maybe you could argue it’s easy to learn, but so is Racket or Scheme. It’s dog slow, doesn’t support multi-threading, and lacks any sort of cohesive design. Lambdas suck, collection types are totally inconsistent (list, dict, etc), too many magic keywords and concepts. Trying to program using a FP style is almost impossible. Also no macros, no traits, no real classes (duck typing is garbage and lazy). You know a language is garbage when the most popular libraries aren’t even written in the language!
bobjordan 8y ago
Great observation, 100% on point. "Microservices are inherently slow. The very concept of a microservice is to break up a boundary by a network call. This means you are taking what was a function call (a couple cpu cycles) and turning it into a network call. There isn’t much you could do that is worse in terms of performance. Network calls are really slow compared to the CPU. But these big companies still choose to use microservices. There really isn’t an architecture slower than microservices that I know of. Microservices’ biggest con is performance, but greatest pro is time-to-market. By building teams around smaller projects and code bases, a company is able to iterate and innovate at a much faster pace. This just goes to show that very large companies also care about time-to-market, not just startups."
- cdoxsey 8y ago
  Except it doesn't scale. Amazon, Google and Netflix (the companies referenced before this quote) can't run their applications on a single machine. Distributed programming is unavoidable at scale. Also there's a big difference between throughput and latency. Networking might introduce a longer delay than a CPU call, but if each one of those calls does significantly more work, the overall performance of the system can increase.
pjmlp 8y ago
Thing is, there are many languages expressive as Python, while providing compilation to native code without having to search for additional tooling. No need to sacrifice performance for productivity.
- craftyguy 8y ago
  Care to name some examples?
  - pjmlp 8y ago
    Lisp, Scheme, Dylan[1], Smalltalk, JavaScript, Prolog. [1] - Yeah, I know it is dead now, just wanted to mention it anyway.
    - purple_ducks 8y ago
      Is Prolog known for it's performance?
      - pjmlp 8y ago
        Well, it beats Python, given that most well known implementations do have JIT compilers.
vp8989 8y ago
The idea that a certain programming language could be proven to be more productive than another is laughable. There are so many factors involved in that, I can pretty much guarantee you it will never be studied to an acceptable level of rigor. The author presents some flimsy "study" about coding string processing algorithms with a straight face, and basically loses all credibility. What about lifetime productivity of a codebase? Speed of refactoring, onboarding, testing etc.. Why don't you just be honest and say you don't care Python is slow because you like using it? Performance doesn't matter? Of course it does. On the frontend, bad performance has been proven by many eCommerce companies to lead to lost business. On backend data pipelines, bad performance kills your flexibility. The longer it takes you to process N, the less times you can do it. The more times you can do it, the more times you can get it wrong. Also who says that time-to-market is everything? Tech history has shown that product quality/brand marketing win out in the long term. There are tons of forgotten shitty products that were first. Being first is over-rated.
- reallydude 8y ago
  > The idea that a certain programming language could be proven to be more productive than another is laughable. The idea that one chemical could be proven to be more effective than another is an equivalent problem. History, application, duration, there's a permutation of issues under that umbrella, as well. The problem is that nobody wants to argue about it very much. The Quorum project tried, to a small degree. > Why don't you just be honest and say you don't care Python is slow because you like using it? It's that and that you have an existing toolchain/codebase you don't want to rewrite in a potentially SLOWER way, because you don't know another language well enough. Python devs rather "solve real problems" than worry about a class of problems that their tool is inadequate for. There are always new trivial problems, where performance doesn't matter.
- pjmlp 8y ago
  Adlib and 3Dfx are two good examples how first to market doesn't mean success.
- Someone 8y ago
  ”The idea that a certain programming language could be proven to be more productive than another is laughable.” I disagree. In the extreme, take python on the one hand and a variant of brainfuck that requires the hash (could be as simple as CRC-32 or as hard as SHA-256) of every source file to be zero on the other. I don’t think you have to go that far to get provable differences, though. Some features that, I think, can fairly easily be proven to make programmers more productive in large parts of the space of programming tasks: - assembler macros - variable names that can have more than two characters - direct support for floating point - support for complex numbers - local variables - structures - support for serialization and deserialization. Compare the way one can make simple structures serializable using Swift’s Codable protocol to whatever is ‘best’ for C (probably something macro-based) - generics - support for Unicode strings - iterators Of course, “python vs Java” or “python vs Perl” are much closer races where I expect measuring differences is way harder and results will depend on problem domain, experience of the subjects of the study, number of good and bad answers on StackOverflow, etc, but even for ‘normal’ languages, I expect some conclusions to be possible, given enough effort is spent. Python 1.0 didn’t have complex numbers, for example, so if your program needed them, python 1.4 was a huge improvement. I doubt evaluating productivity of languages is worthwhile, though.
geebee 8y ago
One big factor in performance is whether a process can be parallelized. It can be an interesting computational problem, and these techniques has really gained relevance lately now the the barrier to clusters has dropped (cost and technical complexity). A lot of algorithms (say for solving a massive optimization problem) that were 10x slower in total operations but can be split up and solved in arbitrarily small pieces are suddenly immensely valuable. And in this context, the actual speed of the programming language probably isn’t terribly important.
truth_seeker 8y ago
Long story in short description: Python isn't slow. Programming languages aren't slow or fast but interpreters are. cpython is slow. PyPy interpreter is reasonably faster, upto 8 times. Recently pypy 7.0 released which brought more hope. http://speed.pypy.org http://speed.pypy.org

31 points by zerogvt 8y ago

“Python is more than 2x as productive as Java in the above study.“ How about compared to a functional language like Haskell, OCaml, or F#? I agree that we should choose productivity over performance when possible. However, Java, for example, is extremely verbose. My impression is that people who use functional languages are quite productive. Plus, they need fewer unit tests.

heavenlyblue 8y ago
"Median hours to solve a problem" The same researcher must write a study of the median temperature at the hospital.
dual_basis 8y ago
I'm not sure if people using functional languages are more productive. One of the points Uncle Bob makes in his talk supporting functional programming is that it is harder and slower to write: https://youtu.be/7Zlp9rKHGD4 https://youtu.be/7Zlp9rKHGD4 That said, I have to wonder if these dynamic scripting languages are truly enabling productivity or if it just feels like that because you can write it and try many things quickly. It reminds me of being such in traffic on the highway - sometimes I get off and take back roads even if my GPS tells me it will take longer, because it is more pleasant to be moving than feeling stuck. That said, I think the biggest benefit of dynamic languages is that they are so easy to refactor. You don't have as rigid a structure, so you can try easily hack together some idea that, done "properly", would require a big refactor. Despite being functional, Haskell's strong type system makes refactoring a huge pain. If course the same things enabling this quick refactoring in the dynamic languages works against you when it comes to speed and reliability.
- ggm 8y ago
  I sit with an FP expert (he might deny it, but he is). What I find, is that he takes a lot longer to get to the solution that I hack out in a day in Python3 (with his assistance most of the time) but his solution has three interesting properties. 1) its usually founded on a very good understanding of the problem, and the abstractions inherent in the problem. If he can re-use the abstractions he has used before he does, to great effect in speed-to-solution 2) it is very usually correct. it is remarkably bug free because the Haskell typing removes a huge amount of ambiguity about the problem going in 3) he has absolutely no problem explaining what it does 3 months or more later. I value these behaviours immensely. My own Python is usually buggy, often re-produces techniques I did a week ago, badly, and I can't explain why I used that lambda/map function.
  - 2wrist 8y ago
    It is really cool when you can find people like this. He sounds class!
    - ggm 8y ago
      He is!
  - gugagore 8y ago
    I'm very curious about what kinds of problems these are. Any chance you can give us an idea?
    - ggm 8y ago
      Typically processing data from logs, analyses of SQL state changes in the absence of good knowledge of the prior states. Some stuff which is aggregation over disparate inputs where a DSL can be useful. A lot of the time radix trees and other prefix/length matching logic is being used. Sometimes we're cross correlating different sources against disparate common keys. Some stuff connecting type: value record structures into JSON. We work a lot on ill-formed requests to refine data previously worked on with new questions. We work a lot on small distributed systems exploring eventual consistency and event sourcing. We work with quite large logfiles collected asynchronously, with poor clock alignment. So data buffering and sorting is not uncommon.
- michaelcampbell 8y ago
  Faster to write -> faster to market Slower to write (and get right) -> fewer bugs to deal with later Pick your poison
  - coldtea 8y ago
    That's really a no brainer. Having your company survive (because you got something faster to market), and having bugs to deal with later, is a so-called "nice problem to have". The bug-less software of a dead company does nobody good.
    - dTal 8y ago
      Quick 'n' dirty might be fine for a move-fast-and-break-things web startup, but the world is much larger than that. I sincerely hope the people writing the software in, let's say an airliner to choose a topical example, choose option 2. Dealing with bugs that have killed people is not a "nice problem to have". (It depresses me that I have to write a defense of good engineering on Hacker news. The value of doing things correctly should be a tacit core principle of any engineer, to be broken only under circumstances of exceptional duress.)
      - coldtea 8y ago
        Most domains are not of the "killing people" type. In those places, they also have other kinds of compliance requirements for the code -- which makes sense: I wouldn't leave the airline software PMs to chose between the above two options.
        dTal 8y ago
        Most domains are also not the "finish the job by next Thursday or the company folds" type either. You're going to have to pay back that technical debt someday, with interest - why not save time and do it right the first time? (Typical SV answer: you're only thinking of the short term because you're trying to get investment and you plan to exit stage left with a fat packet long before the technical debt rears its ugly head, leaving somebody else to clean up the mess. But this is not how most engineering is done in the real world.)
- nikofeyn 8y ago
  i don't think that's true for multiparadigm functional languages like f#, racket, and others. these languages allow you to solve problems in the best way and don't necessarily bind you to some draconian perspective like haskell. functional doesn't necessarily mean static typing. scheme and racket are functional languages.
  - dual_basis 8y ago
    I have found that much of the benefit of functional programming is in the consistent guarantees it makes, not only about your own code but also others' that you depend on. Languages that try to do it all often leave me feeling disjointed. You end up with some libraries written in a strong OOP style, but then others are written with a functional flavor, and then your own code necessarily has to take on a strange amalgamation of the two.
pjmlp 8y ago
Plus all of them provide JIT/AOT compilers.
deleted 8y ago
[deleted]

vardump 8y ago

Whether slow code matters or not depends on the scale, because executing code requires power. If it's a one off or otherwise rarely executed, yeah, it doesn't matter much. But it starts to matter when it's constantly running on a significant number of systems. Widely distributed frequently running slow code can waste a lot of energy. Cell phones and laptops heat up and have a shorter battery life. Even desktops require more power. On the global scale, a large number of such systems can waste several nuclear reactors worth of electricity. 10 times slower code might require 10 times the number of servers. Maybe it doesn't matter much when it's just the difference between one and 10 servers. Or perhaps you don't even consume 10% of a single server. But there's a point when the hardware and power costs start to dominate. The scale matters.

mruts 8y ago

But python is both slow and inexpressive. No types, no match statements, no easy FP. Also the language is poorly designed and inconsistent. I’m practically forced to use python for quant finance at my job because of the network effect: numpy, pandas, cvxpy, etc. But the language itself is total garbage compared to, say, Scala or OCaml. And both of those languages are also much faster.

yvdriess 8y ago
It is interesting to note that the listed libraries are not written in Python (Cython at best). The moment what you want to do with your data is not covered by the library, you either hack it or you're screwed.
deogeo 8y ago
That is my experience as well - the only redeeming feature of the language is the vast selection of easy-to-use libraries. The speed that the language sacrificed isn't some inherent trade-off, but completely orthogonal to what makes the language useful.
- pjmlp 8y ago
  Exactly, Lisp, Scheme, Dylan, Smalltalk, JavaScript, Prolog are all examples of languages as dynamic as Python, while having good JIT/AOT toolchains available by default.
  - mruts 8y ago
    Python has to be one of the worst languages used today. It literally provides no value except for an abundance of libraries. Maybe you could argue it’s easy to learn, but so is Racket or Scheme. It’s dog slow, doesn’t support multi-threading, and lacks any sort of cohesive design. Lambdas suck, collection types are totally inconsistent (list, dict, etc), too many magic keywords and concepts. Trying to program using a FP style is almost impossible. Also no macros, no traits, no real classes (duck typing is garbage and lazy). You know a language is garbage when the most popular libraries aren’t even written in the language!

bobjordan 8y ago

Great observation, 100% on point. "Microservices are inherently slow. The very concept of a microservice is to break up a boundary by a network call. This means you are taking what was a function call (a couple cpu cycles) and turning it into a network call. There isn’t much you could do that is worse in terms of performance. Network calls are really slow compared to the CPU. But these big companies still choose to use microservices. There really isn’t an architecture slower than microservices that I know of. Microservices’ biggest con is performance, but greatest pro is time-to-market. By building teams around smaller projects and code bases, a company is able to iterate and innovate at a much faster pace. This just goes to show that very large companies also care about time-to-market, not just startups."

cdoxsey 8y ago
Except it doesn't scale. Amazon, Google and Netflix (the companies referenced before this quote) can't run their applications on a single machine. Distributed programming is unavoidable at scale. Also there's a big difference between throughput and latency. Networking might introduce a longer delay than a CPU call, but if each one of those calls does significantly more work, the overall performance of the system can increase.

pjmlp 8y ago

Thing is, there are many languages expressive as Python, while providing compilation to native code without having to search for additional tooling. No need to sacrifice performance for productivity.

craftyguy 8y ago
Care to name some examples?
- pjmlp 8y ago
  Lisp, Scheme, Dylan[1], Smalltalk, JavaScript, Prolog. [1] - Yeah, I know it is dead now, just wanted to mention it anyway.
  - purple_ducks 8y ago
    Is Prolog known for it's performance?
    - pjmlp 8y ago
      Well, it beats Python, given that most well known implementations do have JIT compilers.

vp8989 8y ago

The idea that a certain programming language could be proven to be more productive than another is laughable. There are so many factors involved in that, I can pretty much guarantee you it will never be studied to an acceptable level of rigor. The author presents some flimsy "study" about coding string processing algorithms with a straight face, and basically loses all credibility. What about lifetime productivity of a codebase? Speed of refactoring, onboarding, testing etc.. Why don't you just be honest and say you don't care Python is slow because you like using it? Performance doesn't matter? Of course it does. On the frontend, bad performance has been proven by many eCommerce companies to lead to lost business. On backend data pipelines, bad performance kills your flexibility. The longer it takes you to process N, the less times you can do it. The more times you can do it, the more times you can get it wrong. Also who says that time-to-market is everything? Tech history has shown that product quality/brand marketing win out in the long term. There are tons of forgotten shitty products that were first. Being first is over-rated.

reallydude 8y ago
> The idea that a certain programming language could be proven to be more productive than another is laughable. The idea that one chemical could be proven to be more effective than another is an equivalent problem. History, application, duration, there's a permutation of issues under that umbrella, as well. The problem is that nobody wants to argue about it very much. The Quorum project tried, to a small degree. > Why don't you just be honest and say you don't care Python is slow because you like using it? It's that and that you have an existing toolchain/codebase you don't want to rewrite in a potentially SLOWER way, because you don't know another language well enough. Python devs rather "solve real problems" than worry about a class of problems that their tool is inadequate for. There are always new trivial problems, where performance doesn't matter.
pjmlp 8y ago
Adlib and 3Dfx are two good examples how first to market doesn't mean success.
Someone 8y ago
”The idea that a certain programming language could be proven to be more productive than another is laughable.” I disagree. In the extreme, take python on the one hand and a variant of brainfuck that requires the hash (could be as simple as CRC-32 or as hard as SHA-256) of every source file to be zero on the other. I don’t think you have to go that far to get provable differences, though. Some features that, I think, can fairly easily be proven to make programmers more productive in large parts of the space of programming tasks: - assembler macros - variable names that can have more than two characters - direct support for floating point - support for complex numbers - local variables - structures - support for serialization and deserialization. Compare the way one can make simple structures serializable using Swift’s Codable protocol to whatever is ‘best’ for C (probably something macro-based) - generics - support for Unicode strings - iterators Of course, “python vs Java” or “python vs Perl” are much closer races where I expect measuring differences is way harder and results will depend on problem domain, experience of the subjects of the study, number of good and bad answers on StackOverflow, etc, but even for ‘normal’ languages, I expect some conclusions to be possible, given enough effort is spent. Python 1.0 didn’t have complex numbers, for example, so if your program needed them, python 1.4 was a huge improvement. I doubt evaluating productivity of languages is worthwhile, though.

geebee 8y ago

One big factor in performance is whether a process can be parallelized. It can be an interesting computational problem, and these techniques has really gained relevance lately now the the barrier to clusters has dropped (cost and technical complexity). A lot of algorithms (say for solving a massive optimization problem) that were 10x slower in total operations but can be split up and solved in arbitrarily small pieces are suddenly immensely valuable. And in this context, the actual speed of the programming language probably isn’t terribly important.

truth_seeker 8y ago

Long story in short description: Python isn't slow. Programming languages aren't slow or fast but interpreters are. cpython is slow. PyPy interpreter is reasonably faster, upto 8 times. Recently pypy 7.0 released which brought more hope. http://speed.pypy.org http://speed.pypy.org