7 ms·

For use cases where performance is important, using an interpreted (implementation of a) language is a bad idea. There are many great reasons to use Python, bu

For use cases where performance is important, using an interpreted (implementation of a) language is a bad idea. There are many great reasons to use Python, but execution speed is not one of them.

std_throwaway 8y ago
But what else do you switch to? Imagine you import tons of modules which often are only available in Python. This gets you going really quickly with your project and it runs very smoothly. Transferring this to C++ would probably take so long you won't even finish to find out before you run out of funding. I have hopes that Rust or some descendant of Rust will get us there in maybe 10 years but in the meantime it would be better to get Python up to speed as good as possible.
- julvo 8y ago
  For some use cases, Go might be a good alternative to Python. It's performant, yet simple and readable and it has a great ecosystem.
  - std_throwaway 8y ago
    Do we have numpy, matplotlib, scipy, pickle etc. for Go?
    - jerf 8y ago
      Do the build processes mentioned in this email make heavy use of numpy, matplotlib, scipy, pickle, etc.? NumPy is very exciting and all, but it's a subcommunity of the Python community, not 90% of its usage. NumPy users are probably not, in general, trying to spawn twenty five thousand processes in sequence to accomplish some task. The people who are complaining about fractions of a millisecond of startup time are not inverting massive matrices.
      - std_throwaway 8y ago
        People who are importing numpy usually are also not inverting massive matrices. These libraries provide tons of functionality that you would like to use in a short script. For people accustomed to these libraries the code is quick to write in a reliable fashion, easy to read but not quick to run. Everything is great except the startup time (and speed when you have more data).
        jerf 8y ago
        This group of people is so precise and exclusive that it is irrelevant. "People using NumPy for system scripting who are intensely sensitive to python startup time" is not a large enough group to be trying to argue Python-global policy with. "People using Python for system scripting who are intensely sensitive to python startup time" is at least large enough to be worth talking about (since speeding up startup time will mostly only help, modulo any possible resources spent to accomplish it), though I'd notice that it hasn't prevented Python from becoming very popular. And plenty of them will find that Go could meet their needs, in a hypothetical universe in which switching languages was free. (That is, I'm not particularly advocating it. It's a last resort for sure.) (Also this argument is predicated on the false assumption that Go has nothing like those things. They aren't as mature by any means, of course, and I generally consider them a bad idea [2], but they do exist.) [2]: https://news.ycombinator.com/item?id=16959022 https://news.ycombinator.com/item?id=16959022
        mratsim 8y ago
        So precise and exclusive that it's irrelevant? Google, Facebook, Netflix, Uber, Amazon, Microsoft are all using Numpy in their data science pipelines, spinning and taking down dockers for ML-as-a-service. I'm pretty sure they care about startup time of both Go and Python.
        jerf 8y ago
        Again, read the article for what the topic of conversation is. If you're "spinning up an entire Docker container", a Python startup time is going to disappear into the multiple seconds that already takes. You are not spinning up several hundred docker containers per second, on a sustained basis for hours at a time, on a single piece of hardware, constrained only by Python startup time. That's going to be a vanishing fraction of the problem, even if you are spinning up that many containers that quickly, and the optimization for that is already obvious (don't do that, do more per container). You are conflating what systems scripting is, which would be what would be managing the docker containers themselves, with what the docker containers would be doing, which would be very likely starting up just one Python instance to "do the thing". I don't imagine there are very many systems scripts out there in the world being started dozens of times per second that use NumPy. Anything that did, again, the obvious optimization would be "don't do that".
  - sametmax 8y ago
    Only if you are an experienced programmer. But a lot of great python tools are created by mathematicians, geographers, biologists, students, sys admins, etc.
    - julvo 8y ago
      I agree, but in this case I meant especially command line tools, such as mercurial. Of course, for scripting experiments etc. Python is the better choice, but these applications are normally not too worried about startup time.
  - TylerE 8y ago
    Personally I found Go almost uniquely UNsuited to Python-style explorative programming. The lack of generics, and the high-ceremony error handling are pretty much 180 degrees from python. While of course it doesn't have backing from a giant multi-nation, I think Nim (https://nim-lang.org/ https://nim-lang.org/) is a much better fit for "python, but fast".
  - no_wizard 8y ago
    I find laying out a go project with dependencies to be miserable though. I write a lot of go and python code and why on God’s green earth did google decide to handle dependencies by having you effectively git clone a library and then have these huge tree deps subdirectories and so on I don’t know. It’s mess. I vastly prefer the approach of either having one canonically designated folder to install dependencies to where each dependency is a top level directory that can be scanned or having them all stored in a project folder relative to root of the workspace similar to node_modules then the current mess. Drives me nuts. Look at this layout if you don’t know what I’m referencing: https://golang.org/doc/code.html#remote https://golang.org/doc/code.html#remote Yes I have used dep And yes I have all kinds of shortcut commands for navigating my Go workspace But look at this canonical example From the docs and it’s easy to see this is a giant mess that is utterly unnecessary. No other language I’ve used has had such a gross problem with dependency layout. It also leads to gross import strings. It’s one of my biggest criticisms of Go to be honest. Thst and it’s reliance on environmental variables that have to be set perfectly in order to actually do anything (thank god for direnv https://direnv.net/ https://direnv.net/)
    - julvo 8y ago
      Isn't what a dependency folder like node_modules pretty much what the vendor folder is to go. Have you tried using a dep mangement tool like glide before?
      - no_wizard 8y ago
        Thst is one I haven’t tried I tried a few others whose names I forgot because most alt managers seemingly stopped development but this looks active. Thank you! I now would shift my argument to the fact that Google should really just adopt this as their standard if it works as advertised. Link for those who haven’t seen it: https://glide.sh https://glide.sh
      - deleted 8y ago
        [deleted]
- sametmax 8y ago
  If Python had a 10th of the funding JS has, we would have start up time, packaging, gui and mobile apps solved by now.
TylerE 8y ago
Performance and startup performance are really seperate things. For instance, for many many CLI tasks a python script will be many times faster than a Java tool, just due to the JVM startup. It doesn't really matter if the Java would even run INSTANTLY... the JVM startup time just kills speed for small CLI invocations.
- merb 8y ago
  > It doesn't really matter if the Java would even run INSTANTLY... the JVM startup time just kills speed for small CLI invocations luckily these problems actually do get addressed slowly via AoT/Graal and Substrate VM. Here comparing a simple hello world program one is written in java and uses Substrate VM to create a binary and compare it with python: $ time ./hello.py hello world! real 0m0.041s user 0m0.017s sys 0m0.023s $ time ./hello.main hello world! real 0m0.019s user 0m0.008s sys 0m0.010s of course the comparsion is unfair
w0m 8y ago
performance != startup time for lots of applications. I use a C curses application (dokia) to store some oft-used commands, but anything that won't be run 500x/day or runs longer than ~1/10th of a second I'll write in python for easier/more powerful development
- rhizome 8y ago
  dokia Google has apparently never heard of this?
  - w0m 8y ago
    https://github.com/skamsie/dokia https://github.com/skamsie/dokia Random github repo I found, I modified it to default to paste and not prepend 'cd' and find it convenient for complex bash strings I use once or twice a day every day. edit Whups, the link 404s now (i put link in a personal wiki once I realized I liked it). I could put it on github to share if anyone wanted, but I would first ping original author (skamsie) about why he made it private (or deleted) out of respect.
typomatic 8y ago
Maybe you didn't read the post, but in the use cases specified, performance is not important--the author is mostly speaking about deploy and test scripts. Whether these take 1 minute or 10 is not particularly interesting, but you would of course prefer faster if possible. That's the point here--of course a faster Python interpreter is better, and the Python maintainers should place a higher priority on it than the very low priority that they currently do.
pfranz 8y ago
Most of this email thread is comparing Mercurial (Python) to Git. I'm not familiar enough with Git's internals to know why and where the languages are split, but it uses a significant amount of Shell scripting and Perl it its code base. You can put 'git-foo' anywhere on PATH and it'll get picked up. So in the comparison they're making startup time doesn't seem to be an issue for a combination of those languages, but it is for Python. It doesn't sound like their problem is that Python is an interpreted language.
- indygreg2 8y ago
  Git is actively rewriting many of their shell and Perl code in C. Performance and portability are given as reasons (having shell and Perl as a dependency on Windows is a bit of a kludge). And shell scripts are much slower on Windows because new process overhead on Windows is ~10x what it is on POSIX platforms. (New threads, however, are faster to create on Windows than on POSIX.)
fsloth 8y ago
"There are many great reasons to use Python, but execution speed is not one of them." Um, I think there are lots of examples where using pythons internal data structures as they were designed results in code that is fast enough. Even though the language implementation is interpreted, lots of common things can be optimized under the hood using data structure and data type specific execution paths and so on.
geofft 8y ago
Shell is an interpreted language and its startup time is quite fast (5-7 ms on my machine, which is not a particularly fast machine). In fact, large parts of git were written in shell until they realized that shell is only fast on UNIX because of co-evolution (you can fork without exec, and fork is quite fast), and on other platforms like Windows, existing shell implementations are much slower and there isn't a well-tuned production-ready shell that does things completely differently. Then they started rewriting everything in C.
quotemstr 8y ago
Why shouldn't the interpreted program start faster? Bytecode is usually at least 2x denser than machine code, so all things being equal, when starting an interpreted program, you should be doing less IO, take fewer page faults, and so run faster, at least if you defer computationally-intensive work to specialized AOT-compiled helpers. That interpreted programs frequently start slower than their compiled equivalents reflects badly on interpreter implementations, not the concept of interpretation itself.

For use cases where performance is important, using an interpreted (implementation of a) language is a bad idea. There are many great reasons to use Python, but execution speed is not one of them.

std_throwaway 8y ago

But what else do you switch to? Imagine you import tons of modules which often are only available in Python. This gets you going really quickly with your project and it runs very smoothly. Transferring this to C++ would probably take so long you won't even finish to find out before you run out of funding. I have hopes that Rust or some descendant of Rust will get us there in maybe 10 years but in the meantime it would be better to get Python up to speed as good as possible.

julvo 8y ago
For some use cases, Go might be a good alternative to Python. It's performant, yet simple and readable and it has a great ecosystem.
- std_throwaway 8y ago
  Do we have numpy, matplotlib, scipy, pickle etc. for Go?
  - jerf 8y ago
    Do the build processes mentioned in this email make heavy use of numpy, matplotlib, scipy, pickle, etc.? NumPy is very exciting and all, but it's a subcommunity of the Python community, not 90% of its usage. NumPy users are probably not, in general, trying to spawn twenty five thousand processes in sequence to accomplish some task. The people who are complaining about fractions of a millisecond of startup time are not inverting massive matrices.
    - std_throwaway 8y ago
      People who are importing numpy usually are also not inverting massive matrices. These libraries provide tons of functionality that you would like to use in a short script. For people accustomed to these libraries the code is quick to write in a reliable fashion, easy to read but not quick to run. Everything is great except the startup time (and speed when you have more data).
      - jerf 8y ago
        This group of people is so precise and exclusive that it is irrelevant. "People using NumPy for system scripting who are intensely sensitive to python startup time" is not a large enough group to be trying to argue Python-global policy with. "People using Python for system scripting who are intensely sensitive to python startup time" is at least large enough to be worth talking about (since speeding up startup time will mostly only help, modulo any possible resources spent to accomplish it), though I'd notice that it hasn't prevented Python from becoming very popular. And plenty of them will find that Go could meet their needs, in a hypothetical universe in which switching languages was free. (That is, I'm not particularly advocating it. It's a last resort for sure.) (Also this argument is predicated on the false assumption that Go has nothing like those things. They aren't as mature by any means, of course, and I generally consider them a bad idea [2], but they do exist.) [2]: https://news.ycombinator.com/item?id=16959022 https://news.ycombinator.com/item?id=16959022
        mratsim 8y ago
        So precise and exclusive that it's irrelevant? Google, Facebook, Netflix, Uber, Amazon, Microsoft are all using Numpy in their data science pipelines, spinning and taking down dockers for ML-as-a-service. I'm pretty sure they care about startup time of both Go and Python.
        jerf 8y ago
        Again, read the article for what the topic of conversation is. If you're "spinning up an entire Docker container", a Python startup time is going to disappear into the multiple seconds that already takes. You are not spinning up several hundred docker containers per second, on a sustained basis for hours at a time, on a single piece of hardware, constrained only by Python startup time. That's going to be a vanishing fraction of the problem, even if you are spinning up that many containers that quickly, and the optimization for that is already obvious (don't do that, do more per container). You are conflating what systems scripting is, which would be what would be managing the docker containers themselves, with what the docker containers would be doing, which would be very likely starting up just one Python instance to "do the thing". I don't imagine there are very many systems scripts out there in the world being started dozens of times per second that use NumPy. Anything that did, again, the obvious optimization would be "don't do that".
- sametmax 8y ago
  Only if you are an experienced programmer. But a lot of great python tools are created by mathematicians, geographers, biologists, students, sys admins, etc.
  - julvo 8y ago
    I agree, but in this case I meant especially command line tools, such as mercurial. Of course, for scripting experiments etc. Python is the better choice, but these applications are normally not too worried about startup time.
- TylerE 8y ago
  Personally I found Go almost uniquely UNsuited to Python-style explorative programming. The lack of generics, and the high-ceremony error handling are pretty much 180 degrees from python. While of course it doesn't have backing from a giant multi-nation, I think Nim (https://nim-lang.org/ https://nim-lang.org/) is a much better fit for "python, but fast".
- no_wizard 8y ago
  I find laying out a go project with dependencies to be miserable though. I write a lot of go and python code and why on God’s green earth did google decide to handle dependencies by having you effectively git clone a library and then have these huge tree deps subdirectories and so on I don’t know. It’s mess. I vastly prefer the approach of either having one canonically designated folder to install dependencies to where each dependency is a top level directory that can be scanned or having them all stored in a project folder relative to root of the workspace similar to node_modules then the current mess. Drives me nuts. Look at this layout if you don’t know what I’m referencing: https://golang.org/doc/code.html#remote https://golang.org/doc/code.html#remote Yes I have used dep And yes I have all kinds of shortcut commands for navigating my Go workspace But look at this canonical example From the docs and it’s easy to see this is a giant mess that is utterly unnecessary. No other language I’ve used has had such a gross problem with dependency layout. It also leads to gross import strings. It’s one of my biggest criticisms of Go to be honest. Thst and it’s reliance on environmental variables that have to be set perfectly in order to actually do anything (thank god for direnv https://direnv.net/ https://direnv.net/)
  - julvo 8y ago
    Isn't what a dependency folder like node_modules pretty much what the vendor folder is to go. Have you tried using a dep mangement tool like glide before?
    - no_wizard 8y ago
      Thst is one I haven’t tried I tried a few others whose names I forgot because most alt managers seemingly stopped development but this looks active. Thank you! I now would shift my argument to the fact that Google should really just adopt this as their standard if it works as advertised. Link for those who haven’t seen it: https://glide.sh https://glide.sh
    - deleted 8y ago
      [deleted]
sametmax 8y ago
If Python had a 10th of the funding JS has, we would have start up time, packaging, gui and mobile apps solved by now.

TylerE 8y ago

Performance and startup performance are really seperate things. For instance, for many many CLI tasks a python script will be many times faster than a Java tool, just due to the JVM startup. It doesn't really matter if the Java would even run INSTANTLY... the JVM startup time just kills speed for small CLI invocations.

merb 8y ago
> It doesn't really matter if the Java would even run INSTANTLY... the JVM startup time just kills speed for small CLI invocations luckily these problems actually do get addressed slowly via AoT/Graal and Substrate VM. Here comparing a simple hello world program one is written in java and uses Substrate VM to create a binary and compare it with python: $ time ./hello.py hello world! real 0m0.041s user 0m0.017s sys 0m0.023s $ time ./hello.main hello world! real 0m0.019s user 0m0.008s sys 0m0.010s of course the comparsion is unfair

w0m 8y ago

performance != startup time for lots of applications. I use a C curses application (dokia) to store some oft-used commands, but anything that won't be run 500x/day or runs longer than ~1/10th of a second I'll write in python for easier/more powerful development

rhizome 8y ago
dokia Google has apparently never heard of this?
- w0m 8y ago
  https://github.com/skamsie/dokia https://github.com/skamsie/dokia Random github repo I found, I modified it to default to paste and not prepend 'cd' and find it convenient for complex bash strings I use once or twice a day every day. edit Whups, the link 404s now (i put link in a personal wiki once I realized I liked it). I could put it on github to share if anyone wanted, but I would first ping original author (skamsie) about why he made it private (or deleted) out of respect.

typomatic 8y ago

Maybe you didn't read the post, but in the use cases specified, performance is not important--the author is mostly speaking about deploy and test scripts. Whether these take 1 minute or 10 is not particularly interesting, but you would of course prefer faster if possible. That's the point here--of course a faster Python interpreter is better, and the Python maintainers should place a higher priority on it than the very low priority that they currently do.

pfranz 8y ago

Most of this email thread is comparing Mercurial (Python) to Git. I'm not familiar enough with Git's internals to know why and where the languages are split, but it uses a significant amount of Shell scripting and Perl it its code base. You can put 'git-foo' anywhere on PATH and it'll get picked up. So in the comparison they're making startup time doesn't seem to be an issue for a combination of those languages, but it is for Python. It doesn't sound like their problem is that Python is an interpreted language.

indygreg2 8y ago
Git is actively rewriting many of their shell and Perl code in C. Performance and portability are given as reasons (having shell and Perl as a dependency on Windows is a bit of a kludge). And shell scripts are much slower on Windows because new process overhead on Windows is ~10x what it is on POSIX platforms. (New threads, however, are faster to create on Windows than on POSIX.)

fsloth 8y ago

"There are many great reasons to use Python, but execution speed is not one of them." Um, I think there are lots of examples where using pythons internal data structures as they were designed results in code that is fast enough. Even though the language implementation is interpreted, lots of common things can be optimized under the hood using data structure and data type specific execution paths and so on.

geofft 8y ago

Shell is an interpreted language and its startup time is quite fast (5-7 ms on my machine, which is not a particularly fast machine). In fact, large parts of git were written in shell until they realized that shell is only fast on UNIX because of co-evolution (you can fork without exec, and fork is quite fast), and on other platforms like Windows, existing shell implementations are much slower and there isn't a well-tuned production-ready shell that does things completely differently. Then they started rewriting everything in C.

quotemstr 8y ago

Why shouldn't the interpreted program start faster? Bytecode is usually at least 2x denser than machine code, so all things being equal, when starting an interpreted program, you should be doing less IO, take fewer page faults, and so run faster, at least if you defer computationally-intensive work to specialized AOT-compiled helpers. That interpreted programs frequently start slower than their compiled equivalents reflects badly on interpreter implementations, not the concept of interpretation itself.