8 ms·

A few people tried to excuse the slow python, but as far as I know the story, excuses are not necessary. Truth is that python was not meant to be fast, its sour

A few people tried to excuse the slow python, but as far as I know the story, excuses are not necessary. Truth is that python was not meant to be fast, its source code was not meant to be fast, and its design was not optimized with the idea of being fast. Python was meant as a scrypting language that was easy to learn and work with on all levels and the issue of its slowness became important when it outgrew its role and became an application language powering large parts of the internet and the bigger part of the very expensive ML industry. I know that speed is a virtue, but it becomes a fundamental virtue when you have to scale and python was not meant to scale. So, yes, it is easy for people to be righteous and furious over the issue, but being righteous in hindsight is much easier than useful.

atwood22 4y ago
> Python was meant as a scrypting language that was easy to learn and work with on all levels. Being fast isn't contradictory with this goal. If anything, this is a lesson that so many developers forget. Things should be fast by default.
- BeetleB 4y ago
  > Things should be fast by default. In over 90% of my work in the SW industry, being fast(er) was of no benefit to anyone. So no, it should not be fast by default.
  - vorticalbox 4y ago
    Being first to market is usually more important than being faster then those who got there first.
  - uoaei 4y ago
    Ha! Good luck convincing end-users that that's true.
    - BeetleB 4y ago
      I don't need to. Less than 10% of end users have put in requests for performance improvements.
    - zo1 4y ago
      Client doesn't care. As long as their advertising spend leads to conversion into your "slow" app, they're happy.
  - dahart 4y ago
    > no, it should not be fast by default. Maybe better to elaborate on what it should be, if not fast? Surely you aren’t advocating things should be intentionally slow by default, or carelessly inefficient? There’s a valid tradeoff between perf and developer time, and it’s fair to want to prioritize developer time. There’s a valid reason to not care about fast if the process is fast enough that a human doesn’t notice. That said, depends on what your work is, but defaulting to writing faster, more efficient code might benefit a lot of people indirectly. Lower power is valuable for server code and for electricity bills and at some level for air quality in places where power isn’t renewable. Faster benefits parallel processing, it leaves more room for other processes than yours. Faster means companies and users can buy cheaper hardware.
    - BeetleB 4y ago
      > Maybe better to elaborate on what it should be, if not fast? It should satisfy the needs of the customer and it should be reasonably secure. Everything else is a luxury. My point was that in most of my time in the industry being faster would not have benefited my customers in any manner worth measuring. I'm not anti-performance. I fantasize about how to make my software faster to maintain my geek credentials. But neither my bosses nor my customers pay for it. If a customer says they want it faster we'll oblige.
      - dolmen 4y ago
        Clean code, easy to maintain is not luxury when talking about a programming language implementation.
- soperj 4y ago
  When you only have so many hours to go around, you concentrate on the main goals.
  - atwood22 4y ago
    My point is that you can write fast code just as easily as you can write slow code. So engineers should write fast code when possible. Obviously you can spend a lot of time making things faster, but that doesn't mean you can't be fast by default.
    - jjnoakes 4y ago
      > you can write fast code just as easily as you can write slow code I think some people can do this and some can't. For some, writing slow code is much easier, and their contributions are still valuable. Once the bottlenecks are problems, someone with more performance-oriented skills can help speed up the critical path, and slow code outside of the critical path is just fine to leave as-is. If you somehow limited contributions only to those who write fast code, I think you'd be leaving way too much on the table.
    - zo1 4y ago
      Being fast requires effort. It's not always about raw performance of the language yo use, it's about using the right structures, algorithms, tradeoffs, solving the right problems, etc. It's not trivial and I've seen so many bad implementations in "fast" compiled languages.
    - otherme123 4y ago
      Not true. Premature optimization is the root of all evil. You first write clean code, and then you profile and optimize. I refer you to the underlyings of dicts through the years (https://www.youtube.com/watch?v=npw4s1QTmPg https://www.youtube.com/watch?v=npw4s1QTmPg) as an example of that optimization taking years of incremental changes. Once you see the current version it's easy to claim that you would have get to the current and best version in the first place, as obvious as it looks in hindsight.
- amelius 4y ago
  "Premature optimization is the root of all evil." -- Donald Knuth
  - GrayShade 4y ago
    You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: https://softwareengineering.stackexchange.com/a/80092 https://softwareengineering.stackexchange.com/a/80092 > Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.
    - winstonewert 4y ago
      > You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: I'm confused. What do you think the context changes? At least as I read it, both the short form and full context convey the same idea.
      - GrayShade 4y ago
        That quote has been thrown around every time in order to justify writing inefficient code and never optimizing it. Python is 10-1000x slower than C, but sure, let's keep using it because premature optimization is the root of all evil, as Knuth said. People really love to ignore the "premature" word in that quote. Instead, what he meant is that you should profile what part of the code is slow and focus on it first. Knuth didn't say you should be fine with 10-1000x slower code overall.
        bshipp 4y ago
        You certainly can accept that slowdown if the total program run-time remains within acceptable limits and the use of a rapid prototyping language reduces development time. There are times when doing computationally heavy, long-running processes where speed is important, but if the 1000x speedup is not noticeable to the user than is it really a good use of development time to convert that to a more optimized language? As was said, profile, find user-impacting bottlenecks, and then optimize.
- pydry 4y ago
  Fast typically comes with trade offs. Languages that tried to be all things to all people really havent done so well.
- _the_inflator 4y ago
  Fast in which regard? Fast coding? Fast results after hitting "run"? ;)
- pizza234 4y ago
  > Being fast isn't contradictory with this goal. If anything, this is a lesson that so many developers forget. Things should be fast by default. It absolutely is contradictory. If you look at the development of programming languages interpreters/VMs, after a certain point, improvements in speed become a matter of more complex algorithms and data structures. Check out garbage collectors - it's true that Golang keeps a simple one, but other languages progressively increase its sophistication - think about Java or Ruby. Or JITs, for example, which are the latest and greatest in terms of programming languages optimization; they are complicated beasts.
  - atwood22 4y ago
    Yes, you can spend a large amount of time making things faster. But note that Go's GC is fast, even though it is simple. It's not the fastest, but it is acceptably fast.
    - YesThatTom2 4y ago
      Funny you should pick that example in a sub thread that you started with an assertion that code should be fast from by default. Go’s GC was intentionally slow at first. They wanted to get it right THEN make it fast. No offense but you’re not making a strong case. You’re sounding like an inexperienced coder that hasn’t yet learned that premature optimization is bad.
      - atwood22 4y ago
        Just FYI, Go's GC mantra is "GC latency is an existential threat to Go."
      - gpderetta 4y ago
        Far from it, Go was designed to be optimizable form the start. The GC was obviously not optimal, but the language semantics were such that the GC can be replaced with a better one with relatively minimal disruption. Of course one can't release optimal code from version one, that would be absurd. Also your last sentence is extremely condescending.
- gpderetta 4y ago
  Or at the very least they should be designed in such a way that it optimizing them later is still possible.
zitterbewegung 4y ago
Meta has an active port to optimize instagram to make it faster and they just open sourced it to have optimizations merged back into CPython When you have so many large companies with a vested interest in optimization I believe that Python can become faster by doing realistic and targeted optimizations . The other strategies to optimize didn’t work at all or just served internal problems at large companies .
- gostsamo 4y ago
  O, I agree. As I said, when python and its uses scaled, it became quite necessary to make it fast. I like that it will be fast as well and I am not happy that it is slow at the moment. My point is that there are reasons why it was not optimized in the beginning and why this process of optimizations has started now.
  - zitterbewegung 4y ago
    Back when Python was started there was really C or C++ for optimized programs and scripting languages like Python and Perl. But since Python had the ability for C Extensions it allowed it to bypass those problems. Since Python was easy to learn both web developers and scientists to learn. Then financial organizations started to get interested and that’s really how Python cemented itself. What exactly do you do with Python that slows you down ?
    - lanstin 4y ago
      I worked on Python almost exclusively for maybe five years. Then I tried go. Each time I wrote a go program, I am giddy with excitement at how fast my first attempt is, scaling so smoothly with the number of cores. I also wrote a lot of fairly performant C in the 90s, so I know what computers can do in a second. I still use Python for cases when dev time is more important than execution time (which is rarer now that I'm working adjacent to "big data") or when I'm doing things like writing python to close gaps in the various arrays of web apps provided for navigating the corporate work flow, and if I went as fast as a 64 core box let me, we'd have some outages in corp github or artifactory or the like, so I just do it one slow thing at a time on 1 core and wait for the results. Maybe multiprocessing with 10 process worker pool once I'm somewhat confident in the back end system I am talking to. (edit: removed my normal email signoff)
ihaveajob 4y ago
Indeed, and that's why I love it. When I need extra performance, which is very rarely, I don't mind spending extra effort outsourcing it to a binary module. More often, the problem is an inefficient algorithm or data arrangement. I took a graduate level data structures class and the professor used Python among other things "because it's about 80 times slower than C, so you have to think hard about your algorithms". At scale, that matters.
- thesuperbigfrog 4y ago
  Global Interpreter Lock (GIL) is also an issue affecting Python execution speed: https://en.wikipedia.org/wiki/Global_interpreter_lock https://en.wikipedia.org/wiki/Global_interpreter_lock
  - dralley 4y ago
    Not really, no. It prevents you from taking advantage of multiple cores. Doesn't really impact straight-line execution speed. A data structures course is primarily not going to be concerned with multithreading.
    - thesuperbigfrog 4y ago
      >> It prevents you from taking advantage of multiple cores. Doesn't really impact straight-line execution speed. Most computers running Python today will have multiple cores. If your program can only use a fraction of its available computing power, it affects the program's execution speed. Python is too widely used in compute-intensive applications (e.g. machine learning) for GIL-related issues to be ignored in the long term. Ideally, Python should have automatic load scaling and thread-safe data structures in the standard library to take advantage of all available CPU cores. Java has had concurrent data structures in its standard library for years and is adding support for "easy" multi-threaded execution with Project Loom. Python needs to add its own Pythonic versions of thread-safe data structures and compute load scaling to take advantage of the multi-core CPUs that it runs on. >> A data structures course is primarily not going to be concerned with multithreading. A graduate-level data structures course might include concurrent and parallel data structures.
gary_0 4y ago
> python was not meant to scale So many successful projects/technologies started out that way. The Web, JavaScript, e-mail. DNS started out as a HOSTS.TXT file that people copied around. Linus Torvalds announced Linux as "just a hobby, won't be big and professional like gnu". Minecraft rendered huge worlds with unoptimized Java and fixed-function OpenGL.

atwood22 4y ago

> Python was meant as a scrypting language that was easy to learn and work with on all levels. Being fast isn't contradictory with this goal. If anything, this is a lesson that so many developers forget. Things should be fast by default.

BeetleB 4y ago
> Things should be fast by default. In over 90% of my work in the SW industry, being fast(er) was of no benefit to anyone. So no, it should not be fast by default.
- vorticalbox 4y ago
  Being first to market is usually more important than being faster then those who got there first.
- uoaei 4y ago
  Ha! Good luck convincing end-users that that's true.
  - BeetleB 4y ago
    I don't need to. Less than 10% of end users have put in requests for performance improvements.
  - zo1 4y ago
    Client doesn't care. As long as their advertising spend leads to conversion into your "slow" app, they're happy.
- dahart 4y ago
  > no, it should not be fast by default. Maybe better to elaborate on what it should be, if not fast? Surely you aren’t advocating things should be intentionally slow by default, or carelessly inefficient? There’s a valid tradeoff between perf and developer time, and it’s fair to want to prioritize developer time. There’s a valid reason to not care about fast if the process is fast enough that a human doesn’t notice. That said, depends on what your work is, but defaulting to writing faster, more efficient code might benefit a lot of people indirectly. Lower power is valuable for server code and for electricity bills and at some level for air quality in places where power isn’t renewable. Faster benefits parallel processing, it leaves more room for other processes than yours. Faster means companies and users can buy cheaper hardware.
  - BeetleB 4y ago
    > Maybe better to elaborate on what it should be, if not fast? It should satisfy the needs of the customer and it should be reasonably secure. Everything else is a luxury. My point was that in most of my time in the industry being faster would not have benefited my customers in any manner worth measuring. I'm not anti-performance. I fantasize about how to make my software faster to maintain my geek credentials. But neither my bosses nor my customers pay for it. If a customer says they want it faster we'll oblige.
    - dolmen 4y ago
      Clean code, easy to maintain is not luxury when talking about a programming language implementation.
soperj 4y ago
When you only have so many hours to go around, you concentrate on the main goals.
- atwood22 4y ago
  My point is that you can write fast code just as easily as you can write slow code. So engineers should write fast code when possible. Obviously you can spend a lot of time making things faster, but that doesn't mean you can't be fast by default.
  - jjnoakes 4y ago
    > you can write fast code just as easily as you can write slow code I think some people can do this and some can't. For some, writing slow code is much easier, and their contributions are still valuable. Once the bottlenecks are problems, someone with more performance-oriented skills can help speed up the critical path, and slow code outside of the critical path is just fine to leave as-is. If you somehow limited contributions only to those who write fast code, I think you'd be leaving way too much on the table.
  - zo1 4y ago
    Being fast requires effort. It's not always about raw performance of the language yo use, it's about using the right structures, algorithms, tradeoffs, solving the right problems, etc. It's not trivial and I've seen so many bad implementations in "fast" compiled languages.
  - otherme123 4y ago
    Not true. Premature optimization is the root of all evil. You first write clean code, and then you profile and optimize. I refer you to the underlyings of dicts through the years (https://www.youtube.com/watch?v=npw4s1QTmPg https://www.youtube.com/watch?v=npw4s1QTmPg) as an example of that optimization taking years of incremental changes. Once you see the current version it's easy to claim that you would have get to the current and best version in the first place, as obvious as it looks in hindsight.
amelius 4y ago
"Premature optimization is the root of all evil." -- Donald Knuth
- GrayShade 4y ago
  You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: https://softwareengineering.stackexchange.com/a/80092 https://softwareengineering.stackexchange.com/a/80092 > Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.
  - winstonewert 4y ago
    > You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: I'm confused. What do you think the context changes? At least as I read it, both the short form and full context convey the same idea.
    - GrayShade 4y ago
      That quote has been thrown around every time in order to justify writing inefficient code and never optimizing it. Python is 10-1000x slower than C, but sure, let's keep using it because premature optimization is the root of all evil, as Knuth said. People really love to ignore the "premature" word in that quote. Instead, what he meant is that you should profile what part of the code is slow and focus on it first. Knuth didn't say you should be fine with 10-1000x slower code overall.
      - bshipp 4y ago
        You certainly can accept that slowdown if the total program run-time remains within acceptable limits and the use of a rapid prototyping language reduces development time. There are times when doing computationally heavy, long-running processes where speed is important, but if the 1000x speedup is not noticeable to the user than is it really a good use of development time to convert that to a more optimized language? As was said, profile, find user-impacting bottlenecks, and then optimize.
pydry 4y ago
Fast typically comes with trade offs. Languages that tried to be all things to all people really havent done so well.
_the_inflator 4y ago
Fast in which regard? Fast coding? Fast results after hitting "run"? ;)
pizza234 4y ago
> Being fast isn't contradictory with this goal. If anything, this is a lesson that so many developers forget. Things should be fast by default. It absolutely is contradictory. If you look at the development of programming languages interpreters/VMs, after a certain point, improvements in speed become a matter of more complex algorithms and data structures. Check out garbage collectors - it's true that Golang keeps a simple one, but other languages progressively increase its sophistication - think about Java or Ruby. Or JITs, for example, which are the latest and greatest in terms of programming languages optimization; they are complicated beasts.
- atwood22 4y ago
  Yes, you can spend a large amount of time making things faster. But note that Go's GC is fast, even though it is simple. It's not the fastest, but it is acceptably fast.
  - YesThatTom2 4y ago
    Funny you should pick that example in a sub thread that you started with an assertion that code should be fast from by default. Go’s GC was intentionally slow at first. They wanted to get it right THEN make it fast. No offense but you’re not making a strong case. You’re sounding like an inexperienced coder that hasn’t yet learned that premature optimization is bad.
    - atwood22 4y ago
      Just FYI, Go's GC mantra is "GC latency is an existential threat to Go."
    - gpderetta 4y ago
      Far from it, Go was designed to be optimizable form the start. The GC was obviously not optimal, but the language semantics were such that the GC can be replaced with a better one with relatively minimal disruption. Of course one can't release optimal code from version one, that would be absurd. Also your last sentence is extremely condescending.
gpderetta 4y ago
Or at the very least they should be designed in such a way that it optimizing them later is still possible.

zitterbewegung 4y ago

Meta has an active port to optimize instagram to make it faster and they just open sourced it to have optimizations merged back into CPython When you have so many large companies with a vested interest in optimization I believe that Python can become faster by doing realistic and targeted optimizations . The other strategies to optimize didn’t work at all or just served internal problems at large companies .

gostsamo 4y ago
O, I agree. As I said, when python and its uses scaled, it became quite necessary to make it fast. I like that it will be fast as well and I am not happy that it is slow at the moment. My point is that there are reasons why it was not optimized in the beginning and why this process of optimizations has started now.
- zitterbewegung 4y ago
  Back when Python was started there was really C or C++ for optimized programs and scripting languages like Python and Perl. But since Python had the ability for C Extensions it allowed it to bypass those problems. Since Python was easy to learn both web developers and scientists to learn. Then financial organizations started to get interested and that’s really how Python cemented itself. What exactly do you do with Python that slows you down ?
  - lanstin 4y ago
    I worked on Python almost exclusively for maybe five years. Then I tried go. Each time I wrote a go program, I am giddy with excitement at how fast my first attempt is, scaling so smoothly with the number of cores. I also wrote a lot of fairly performant C in the 90s, so I know what computers can do in a second. I still use Python for cases when dev time is more important than execution time (which is rarer now that I'm working adjacent to "big data") or when I'm doing things like writing python to close gaps in the various arrays of web apps provided for navigating the corporate work flow, and if I went as fast as a 64 core box let me, we'd have some outages in corp github or artifactory or the like, so I just do it one slow thing at a time on 1 core and wait for the results. Maybe multiprocessing with 10 process worker pool once I'm somewhat confident in the back end system I am talking to. (edit: removed my normal email signoff)

ihaveajob 4y ago

Indeed, and that's why I love it. When I need extra performance, which is very rarely, I don't mind spending extra effort outsourcing it to a binary module. More often, the problem is an inefficient algorithm or data arrangement. I took a graduate level data structures class and the professor used Python among other things "because it's about 80 times slower than C, so you have to think hard about your algorithms". At scale, that matters.

thesuperbigfrog 4y ago
Global Interpreter Lock (GIL) is also an issue affecting Python execution speed: https://en.wikipedia.org/wiki/Global_interpreter_lock https://en.wikipedia.org/wiki/Global_interpreter_lock
- dralley 4y ago
  Not really, no. It prevents you from taking advantage of multiple cores. Doesn't really impact straight-line execution speed. A data structures course is primarily not going to be concerned with multithreading.
  - thesuperbigfrog 4y ago
    >> It prevents you from taking advantage of multiple cores. Doesn't really impact straight-line execution speed. Most computers running Python today will have multiple cores. If your program can only use a fraction of its available computing power, it affects the program's execution speed. Python is too widely used in compute-intensive applications (e.g. machine learning) for GIL-related issues to be ignored in the long term. Ideally, Python should have automatic load scaling and thread-safe data structures in the standard library to take advantage of all available CPU cores. Java has had concurrent data structures in its standard library for years and is adding support for "easy" multi-threaded execution with Project Loom. Python needs to add its own Pythonic versions of thread-safe data structures and compute load scaling to take advantage of the multi-core CPUs that it runs on. >> A data structures course is primarily not going to be concerned with multithreading. A graduate-level data structures course might include concurrent and parallel data structures.

gary_0 4y ago

> python was not meant to scale So many successful projects/technologies started out that way. The Web, JavaScript, e-mail. DNS started out as a HOSTS.TXT file that people copied around. Linus Torvalds announced Linux as "just a hobby, won't be big and professional like gnu". Minecraft rendered huge worlds with unoptimized Java and fixed-function OpenGL.